| 1 | plugin_name: netflow-plugin |
| 2 | modules: |
| 3 | - meta: |
| 4 | plugin_name: netflow-plugin |
| 5 | module_name: netflow |
| 6 | monitored_instance: |
| 7 | name: NetFlow |
| 8 | link: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_pi/configuration/15-s/nf-15-s-book/ios-netflow-ov.html |
| 9 | categories: |
| 10 | - flows.sources |
| 11 | icon_filename: network-wired.svg |
| 12 | keywords: |
| 13 | - netflow |
| 14 | - netflow v5 |
| 15 | - netflow v7 |
| 16 | - netflow v9 |
| 17 | - cisco |
| 18 | - flows |
| 19 | - network flows |
| 20 | - flow collector |
| 21 | related_resources: |
| 22 | integrations: |
| 23 | list: |
| 24 | - plugin_name: netflow-plugin |
| 25 | module_name: ipfix |
| 26 | - plugin_name: netflow-plugin |
| 27 | module_name: sflow |
| 28 | info_provided_to_referring_integrations: |
| 29 | description: "" |
| 30 | overview: |
| 31 | data_collection: |
| 32 | metrics_description: | |
| 33 | Collect network flow records from NetFlow v5, v7, and v9 exporters such as |
| 34 | routers, switches, and firewalls. Each record exposes source and destination IP, |
| 35 | ports, protocol, bytes, packets, ToS, TCP flags, and ingress/egress interface indices. |
| 36 | Enrichment adds GeoIP country/city/ASN, static metadata, and classifier tags. |
| 37 | |
| 38 | For full documentation including vendor configuration examples, sampling caveats, template |
| 39 | handling and verification steps, see the [Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/). |
| 40 | method_description: | |
| 41 | The plugin listens on a configurable UDP socket for NetFlow datagrams. |
| 42 | NetFlow v5 and v7 records are decoded directly. NetFlow v9 records are decoded using |
| 43 | dynamic templates cached from the exporter. Decoded records are enriched in-memory |
| 44 | and appended to disk-backed journal tiers (raw, 1-minute, 5-minute, 1-hour rollups). |
| 45 | supported_platforms: |
| 46 | include: |
| 47 | - Linux |
| 48 | exclude: [] |
| 49 | multi_instance: false |
| 50 | additional_permissions: |
| 51 | description: "" |
| 52 | default_behavior: |
| 53 | auto_detection: |
| 54 | description: "The plugin starts when enabled in netflow.yaml and listens on the configured UDP port." |
| 55 | limits: |
| 56 | description: "Operational limits are driven by sustained flows/s, cardinality, retention, storage speed, and enrichment. Plan around 25k sustained flows/s per well-provisioned agent for the full raw + rollup pipeline; use distributed agents for larger deployments." |
| 57 | performance_impact: |
| 58 | description: "Disabled until exporters send traffic. Once active, CPU and disk I/O scale with flow rate and cardinality; size retention and storage from observed flows/s." |
| 59 | setup: |
| 60 | prerequisites: |
| 61 | list: |
| 62 | - title: NetFlow-capable exporter |
| 63 | description: | |
| 64 | A router, switch, or firewall configured to export NetFlow v5, v7, or v9 datagrams to the |
| 65 | Netdata agent's UDP listener. |
| 66 | configuration: |
| 67 | file: |
| 68 | name: netflow.yaml |
| 69 | options: |
| 70 | description: | |
| 71 | The plugin is configured via `netflow.yaml` in the Netdata configuration directory. |
| 72 | folding: |
| 73 | title: Config options |
| 74 | enabled: true |
| 75 | list: |
| 76 | - name: listener.listen |
| 77 | description: UDP endpoint for NetFlow datagrams. |
| 78 | default_value: "0.0.0.0:2055" |
| 79 | required: false |
| 80 | - name: protocols.v5 |
| 81 | description: Enable NetFlow v5 decoding. |
| 82 | default_value: true |
| 83 | required: false |
| 84 | - name: protocols.v7 |
| 85 | description: Enable NetFlow v7 decoding. |
| 86 | default_value: true |
| 87 | required: false |
| 88 | - name: protocols.v9 |
| 89 | description: Enable NetFlow v9 decoding. |
| 90 | default_value: true |
| 91 | required: false |
| 92 | - name: journal.journal_dir |
| 93 | description: Directory for journal files (relative to NETDATA_CACHE_DIR). |
| 94 | default_value: flows |
| 95 | required: false |
| 96 | - name: journal.tiers.<tier>.size_of_journal_files |
| 97 | description: Per-tier hard size cap. Replace `<tier>` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for time-only retention. |
| 98 | default_value: "10GB" |
| 99 | required: false |
| 100 | - name: journal.tiers.<tier>.duration_of_journal_files |
| 101 | description: Per-tier maximum age. Replace `<tier>` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for size-only retention. |
| 102 | default_value: "7d" |
| 103 | required: false |
| 104 | examples: |
| 105 | folding: |
| 106 | title: Config |
| 107 | enabled: true |
| 108 | list: |
| 109 | - name: Basic NetFlow v5/v9 collection |
| 110 | folding: |
| 111 | enabled: false |
| 112 | description: Listen on Netdata's default flow listener port for v5 and v9 records. |
| 113 | config: | |
| 114 | enabled: true |
| 115 | listener: |
| 116 | listen: "0.0.0.0:2055" |
| 117 | protocols: |
| 118 | v5: true |
| 119 | v9: true |
| 120 | - name: NetFlow v9 only with extended retention |
| 121 | folding: |
| 122 | enabled: true |
| 123 | description: Accept only v9 records, keep one day of raw data, and keep longer rollups. |
| 124 | config: | |
| 125 | enabled: true |
| 126 | listener: |
| 127 | listen: "0.0.0.0:2055" |
| 128 | protocols: |
| 129 | v5: false |
| 130 | v7: false |
| 131 | v9: true |
| 132 | journal: |
| 133 | journal_dir: flows |
| 134 | tiers: |
| 135 | raw: { size_of_journal_files: 50GB, duration_of_journal_files: 24h } |
| 136 | minute_1: { size_of_journal_files: 10GB, duration_of_journal_files: 14d } |
| 137 | minute_5: { size_of_journal_files: 10GB, duration_of_journal_files: 30d } |
| 138 | hour_1: { size_of_journal_files: 10GB, duration_of_journal_files: 365d } |
| 139 | troubleshooting: |
| 140 | problems: |
| 141 | list: |
| 142 | - name: Verifying flow data is arriving and diagnosing failures |
| 143 | description: | |
| 144 | See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for |
| 145 | the full diagnostic recipe -- including UDP path checks, template-error analysis, |
| 146 | and the "looks like a bug but isn't" section (doubling, mirroring, internal-IP geolocation). |
| 147 | See also [Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation-and-data-quality) |
| 148 | and [Anti-patterns](https://learn.netdata.cloud/docs/network-flows/anti-patterns). |
| 149 | alerts: [] |
| 150 | metrics: |
| 151 | folding: |
| 152 | title: Metrics |
| 153 | enabled: false |
| 154 | description: | |
| 155 | Network flow data is stored as journal records, not as traditional metrics. |
| 156 | Use the Network Flows view in the Netdata dashboard to query and visualize flow data. |
| 157 | For the full list of fields and the per-protocol availability matrix, see the |
| 158 | [Field Reference](https://learn.netdata.cloud/docs/network-flows/field-reference). |
| 159 | For visualisation guidance see [Sankey and Table](https://learn.netdata.cloud/docs/network-flows/visualization/sankey-and-table), |
| 160 | [Time-Series](https://learn.netdata.cloud/docs/network-flows/visualization/time-series), |
| 161 | and [Maps and Globe](https://learn.netdata.cloud/docs/network-flows/visualization/maps-and-globe). |
| 162 | availability: [] |
| 163 | scopes: [] |
| 164 | |
| 165 | - meta: |
| 166 | plugin_name: netflow-plugin |
| 167 | module_name: ipfix |
| 168 | monitored_instance: |
| 169 | name: IPFIX |
| 170 | link: https://datatracker.ietf.org/doc/html/rfc7011 |
| 171 | categories: |
| 172 | - flows.sources |
| 173 | icon_filename: network-wired.svg |
| 174 | keywords: |
| 175 | - ipfix |
| 176 | - netflow v10 |
| 177 | - flows |
| 178 | - network flows |
| 179 | - flow collector |
| 180 | - rfc 7011 |
| 181 | related_resources: |
| 182 | integrations: |
| 183 | list: |
| 184 | - plugin_name: netflow-plugin |
| 185 | module_name: netflow |
| 186 | - plugin_name: netflow-plugin |
| 187 | module_name: sflow |
| 188 | info_provided_to_referring_integrations: |
| 189 | description: "" |
| 190 | overview: |
| 191 | data_collection: |
| 192 | metrics_description: | |
| 193 | Collect network flow records from IPFIX (NetFlow v10) exporters. IPFIX extends |
| 194 | NetFlow v9 with variable-length fields, vendor-specific information elements, |
| 195 | and template withdrawal. Each record exposes the same core fields as NetFlow plus |
| 196 | any additional IEs the exporter provides. |
| 197 | |
| 198 | For full documentation including vendor configuration examples (Cisco, Juniper, Arista, |
| 199 | ASA NSEL), biflow handling, sampling caveats, and verification steps, see the |
| 200 | [Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/). |
| 201 | method_description: | |
| 202 | The plugin listens on the same UDP socket as NetFlow. IPFIX messages are identified by |
| 203 | version number 10 and decoded using cached templates. Decoded records are enriched and |
| 204 | appended to disk-backed journal tiers. |
| 205 | supported_platforms: |
| 206 | include: |
| 207 | - Linux |
| 208 | exclude: [] |
| 209 | multi_instance: false |
| 210 | additional_permissions: |
| 211 | description: "" |
| 212 | default_behavior: |
| 213 | auto_detection: |
| 214 | description: "The plugin starts when enabled in netflow.yaml and listens on the configured UDP port." |
| 215 | limits: |
| 216 | description: "Operational limits are driven by sustained flows/s, template churn, cardinality, retention, storage speed, and enrichment. Plan around 25k sustained flows/s per well-provisioned agent for the full raw + rollup pipeline; use distributed agents for larger deployments." |
| 217 | performance_impact: |
| 218 | description: "Disabled until exporters send traffic. Once active, CPU and disk I/O scale with flow rate, template volume, and cardinality; size retention and storage from observed flows/s." |
| 219 | setup: |
| 220 | prerequisites: |
| 221 | list: |
| 222 | - title: IPFIX-capable exporter |
| 223 | description: | |
| 224 | A router, switch, or firewall configured to export IPFIX datagrams to the |
| 225 | Netdata agent's UDP listener. |
| 226 | configuration: |
| 227 | file: |
| 228 | name: netflow.yaml |
| 229 | options: |
| 230 | description: | |
| 231 | IPFIX shares the same `netflow.yaml` configuration file as NetFlow and sFlow. |
| 232 | Enable IPFIX via the `protocols.ipfix` option. |
| 233 | folding: |
| 234 | title: Config options |
| 235 | enabled: true |
| 236 | list: |
| 237 | - name: listener.listen |
| 238 | description: UDP endpoint for IPFIX datagrams. |
| 239 | default_value: "0.0.0.0:2055" |
| 240 | required: false |
| 241 | - name: protocols.ipfix |
| 242 | description: Enable IPFIX decoding. |
| 243 | default_value: true |
| 244 | required: false |
| 245 | - name: journal.journal_dir |
| 246 | description: Directory for journal files (relative to NETDATA_CACHE_DIR). |
| 247 | default_value: flows |
| 248 | required: false |
| 249 | - name: journal.tiers.<tier>.size_of_journal_files |
| 250 | description: Per-tier hard size cap. Replace `<tier>` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for time-only retention. |
| 251 | default_value: "10GB" |
| 252 | required: false |
| 253 | - name: journal.tiers.<tier>.duration_of_journal_files |
| 254 | description: Per-tier maximum age. Replace `<tier>` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for size-only retention. |
| 255 | default_value: "7d" |
| 256 | required: false |
| 257 | examples: |
| 258 | folding: |
| 259 | title: Config |
| 260 | enabled: true |
| 261 | list: |
| 262 | - name: IPFIX collection |
| 263 | folding: |
| 264 | enabled: false |
| 265 | description: Listen for IPFIX records on Netdata's default flow listener port. |
| 266 | config: | |
| 267 | enabled: true |
| 268 | listener: |
| 269 | listen: "0.0.0.0:2055" |
| 270 | protocols: |
| 271 | v5: false |
| 272 | v7: false |
| 273 | v9: false |
| 274 | ipfix: true |
| 275 | sflow: false |
| 276 | troubleshooting: |
| 277 | problems: |
| 278 | list: |
| 279 | - name: Verifying flow data is arriving and diagnosing failures |
| 280 | description: | |
| 281 | See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for |
| 282 | the full diagnostic recipe. For IPFIX specifically, watch the `template_errors` dimension |
| 283 | on `netflow.input_packets` -- IPFIX is template-driven and data records arriving before |
| 284 | their templates are dropped. See also |
| 285 | [Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation-and-data-quality). |
| 286 | alerts: [] |
| 287 | metrics: |
| 288 | folding: |
| 289 | title: Metrics |
| 290 | enabled: false |
| 291 | description: | |
| 292 | Network flow data is stored as journal records, not as traditional metrics. |
| 293 | Use the Network Flows view in the Netdata dashboard to query and visualize flow data. |
| 294 | For the full list of fields and the per-protocol availability matrix, see the |
| 295 | [Field Reference](https://learn.netdata.cloud/docs/network-flows/field-reference). |
| 296 | IPFIX biflow records are emitted as two separate flow records (forward + reverse). |
| 297 | availability: [] |
| 298 | scopes: [] |
| 299 | |
| 300 | - meta: |
| 301 | plugin_name: netflow-plugin |
| 302 | module_name: sflow |
| 303 | monitored_instance: |
| 304 | name: sFlow |
| 305 | link: https://sflow.org/ |
| 306 | categories: |
| 307 | - flows.sources |
| 308 | icon_filename: network-wired.svg |
| 309 | keywords: |
| 310 | - sflow |
| 311 | - sflow v5 |
| 312 | - sampled flows |
| 313 | - flows |
| 314 | - network flows |
| 315 | - flow collector |
| 316 | - inmon |
| 317 | related_resources: |
| 318 | integrations: |
| 319 | list: |
| 320 | - plugin_name: netflow-plugin |
| 321 | module_name: netflow |
| 322 | - plugin_name: netflow-plugin |
| 323 | module_name: ipfix |
| 324 | info_provided_to_referring_integrations: |
| 325 | description: "" |
| 326 | overview: |
| 327 | data_collection: |
| 328 | metrics_description: | |
| 329 | Collect sampled network flow records from sFlow v5 agents. sFlow provides |
| 330 | statistically sampled packet headers, interface counters, or extended gateway data. |
| 331 | Each flow record exposes source and destination IP, ports, protocol, bytes, packets, |
| 332 | and sampling rate information. |
| 333 | |
| 334 | For full documentation including how sFlow differs fundamentally from NetFlow (packet |
| 335 | sampling vs aggregated flows), vendor configuration examples (Arista, Juniper, Aruba CX, |
| 336 | Ruckus, hsflowd), and the limits of sampled data, see the |
| 337 | [Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/). |
| 338 | method_description: | |
| 339 | The plugin listens on the same UDP socket as NetFlow. sFlow datagrams are identified by |
| 340 | their distinct header format and decoded per the sFlow v5 specification. Decoded records |
| 341 | are enriched and appended to disk-backed journal tiers. |
| 342 | supported_platforms: |
| 343 | include: |
| 344 | - Linux |
| 345 | exclude: [] |
| 346 | multi_instance: false |
| 347 | additional_permissions: |
| 348 | description: "" |
| 349 | default_behavior: |
| 350 | auto_detection: |
| 351 | description: "The plugin starts when enabled in netflow.yaml and listens on the configured UDP port." |
| 352 | limits: |
| 353 | description: "Operational limits are driven by sustained samples/s, sampling rate, cardinality, retention, storage speed, and enrichment. Plan capacity from the received sample rate and the expanded byte/packet estimates." |
| 354 | performance_impact: |
| 355 | description: "Disabled until sFlow agents send traffic. Once active, CPU and disk I/O scale with received samples/s and cardinality; size retention and storage from observed flow records/s." |
| 356 | setup: |
| 357 | prerequisites: |
| 358 | list: |
| 359 | - title: sFlow-capable agent |
| 360 | description: | |
| 361 | A switch, router, or host-based sFlow agent (such as Host sFlow) configured to send |
| 362 | sFlow v5 datagrams to the Netdata agent's UDP listener. |
| 363 | configuration: |
| 364 | file: |
| 365 | name: netflow.yaml |
| 366 | options: |
| 367 | description: | |
| 368 | sFlow shares the same `netflow.yaml` configuration file as NetFlow and IPFIX. |
| 369 | Enable sFlow via the `protocols.sflow` option. |
| 370 | folding: |
| 371 | title: Config options |
| 372 | enabled: true |
| 373 | list: |
| 374 | - name: listener.listen |
| 375 | description: UDP endpoint for sFlow datagrams. |
| 376 | default_value: "0.0.0.0:2055" |
| 377 | required: false |
| 378 | - name: protocols.sflow |
| 379 | description: Enable sFlow decoding. |
| 380 | default_value: true |
| 381 | required: false |
| 382 | - name: journal.journal_dir |
| 383 | description: Directory for journal files (relative to NETDATA_CACHE_DIR). |
| 384 | default_value: flows |
| 385 | required: false |
| 386 | - name: journal.tiers.<tier>.size_of_journal_files |
| 387 | description: Per-tier hard size cap. Replace `<tier>` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for time-only retention. |
| 388 | default_value: "10GB" |
| 389 | required: false |
| 390 | - name: journal.tiers.<tier>.duration_of_journal_files |
| 391 | description: Per-tier maximum age. Replace `<tier>` with `raw`, `minute_1`, `minute_5`, or `hour_1`. Set to `null` for size-only retention. |
| 392 | default_value: "7d" |
| 393 | required: false |
| 394 | examples: |
| 395 | folding: |
| 396 | title: Config |
| 397 | enabled: true |
| 398 | list: |
| 399 | - name: sFlow collection |
| 400 | folding: |
| 401 | enabled: false |
| 402 | description: Listen for sFlow v5 datagrams on Netdata's default flow listener port. |
| 403 | config: | |
| 404 | enabled: true |
| 405 | listener: |
| 406 | listen: "0.0.0.0:2055" |
| 407 | protocols: |
| 408 | v5: false |
| 409 | v7: false |
| 410 | v9: false |
| 411 | ipfix: false |
| 412 | sflow: true |
| 413 | troubleshooting: |
| 414 | problems: |
| 415 | list: |
| 416 | - name: Verifying sFlow is arriving and diagnosing failures |
| 417 | description: | |
| 418 | See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for |
| 419 | the full diagnostic recipe. sFlow-specific gotchas: counter samples are not surfaced |
| 420 | (only flow samples), bytes/packets are statistical estimates that won't match SNMP |
| 421 | byte-for-byte, and VLAN information comes from `ExtendedSwitch` records only -- not |
| 422 | from 802.1Q tags inside the sampled header. See also |
| 423 | [Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation-and-data-quality) |
| 424 | and the sFlow section of [Anti-patterns](https://learn.netdata.cloud/docs/network-flows/anti-patterns). |
| 425 | alerts: [] |
| 426 | metrics: |
| 427 | folding: |
| 428 | title: Metrics |
| 429 | enabled: false |
| 430 | description: | |
| 431 | Network flow data is stored as journal records, not as traditional metrics. |
| 432 | Use the Network Flows view in the Netdata dashboard to query and visualize flow data. |
| 433 | For the full list of fields and the per-protocol availability matrix, see the |
| 434 | [Field Reference](https://learn.netdata.cloud/docs/network-flows/field-reference). |
| 435 | sFlow byte counts are statistical estimates (sampled packets multiplied by sampling rate); |
| 436 | they will not reconcile byte-for-byte with SNMP interface counters. |
| 437 | availability: [] |
| 438 | scopes: [] |
| 439 | |
| 440 | # ===================================================================== |
| 441 | # IP Intelligence: GeoIP / ASN data sources used to enrich flow records |
| 442 | # ===================================================================== |
| 443 | |
| 444 | - meta: |
| 445 | plugin_name: netflow-plugin |
| 446 | module_name: dbip |
| 447 | monitored_instance: |
| 448 | name: DB-IP IP Intelligence |
| 449 | link: https://db-ip.com/ |
| 450 | categories: |
| 451 | - flows.enrichment-methods |
| 452 | icon_filename: network-wired.svg |
| 453 | keywords: |
| 454 | - geoip |
| 455 | - asn |
| 456 | - dbip |
| 457 | - db-ip |
| 458 | - mmdb |
| 459 | - ip intelligence |
| 460 | - flow enrichment |
| 461 | related_resources: |
| 462 | integrations: |
| 463 | list: |
| 464 | - plugin_name: netflow-plugin |
| 465 | module_name: maxmind |
| 466 | - plugin_name: netflow-plugin |
| 467 | module_name: iptoasn |
| 468 | - plugin_name: netflow-plugin |
| 469 | module_name: caida-prefix2as |
| 470 | - plugin_name: netflow-plugin |
| 471 | module_name: ip2location |
| 472 | - plugin_name: netflow-plugin |
| 473 | module_name: ipdeny |
| 474 | - plugin_name: netflow-plugin |
| 475 | module_name: ipip |
| 476 | - plugin_name: netflow-plugin |
| 477 | module_name: custom-mmdb |
| 478 | info_provided_to_referring_integrations: |
| 479 | description: "" |
| 480 | overview: |
| 481 | data_collection: |
| 482 | metrics_description: | |
| 483 | Enrich network flows with ASN and geographic context from DB-IP Lite MMDB |
| 484 | databases. DB-IP IP Intelligence is the default IP intelligence source for the |
| 485 | Netdata netflow plugin: when neither `enrichment.geoip.asn_database` nor |
| 486 | `enrichment.geoip.geo_database` is configured, the plugin auto-detects the |
| 487 | DB-IP-built MMDB files at startup and uses them. Two MMDB files are involved: |
| 488 | an ASN database (`topology-ip-asn.mmdb`) and a geographic database |
| 489 | (`topology-ip-geo.mmdb`). Native packages (DEB, RPM) ship a stock copy under |
| 490 | `/usr/share/netdata/topology-ip-intel/`; the bundled refresh tool fetches updates |
| 491 | into `/var/cache/netdata/topology-ip-intel/`. |
| 492 | |
| 493 | DB-IP publishes the Lite databases monthly under a Creative Commons Attribution 4.0 |
| 494 | International licence (https://db-ip.com/db/lite.php). The MMDB format is the MaxMind |
| 495 | binary database; lookups are in-process with no per-flow network call. |
| 496 | |
| 497 | Populates these flow record fields when a lookup succeeds: |
| 498 | |
| 499 | - From the **geo MMDB**: `SRC_COUNTRY`, `DST_COUNTRY`, `SRC_GEO_STATE`, `DST_GEO_STATE`, |
| 500 | `SRC_GEO_CITY`, `DST_GEO_CITY`, `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE`, |
| 501 | `SRC_GEO_LONGITUDE`, `DST_GEO_LONGITUDE`. Country and state survive into all four |
| 502 | journal tiers; city, latitude, and longitude are kept only in the raw tier so the |
| 503 | rollups stay cardinality-bounded. |
| 504 | - From the **ASN MMDB**: `SRC_AS_NAME`, `DST_AS_NAME`. The DB-IP-built ASN database also |
| 505 | tags address ranges as private/reserved (RFC 1918, link-local, RFC 6598, etc.); when |
| 506 | the resolved AS number is `0` and that flag is set, the plugin renders the AS name as |
| 507 | `AS0 Private IP Address Space` instead of `AS0 Unknown ASN`. The AS *number* itself |
| 508 | comes from the wider asn_providers chain, not directly from the MMDB. |
| 509 | |
| 510 | For the cross-provider mechanics (lookup pass ordering, the 30-second hot-reload poll, |
| 511 | the auto-detect path order, multi-database composition, IPv4/IPv6 handling, the failure |
| 512 | modes shared by every MMDB provider), see |
| 513 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 514 | method_description: | |
| 515 | Files are read on plugin start and reloaded automatically when their mtime or size |
| 516 | changes. Auto-detection scans `${NETDATA_CACHE_DIR}/topology-ip-intel/` first |
| 517 | (typically `/var/cache/netdata/topology-ip-intel/`), falling back to the stock copy |
| 518 | under `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/` (typically |
| 519 | `/usr/share/netdata/topology-ip-intel/`). When databases are auto-detected they are |
| 520 | treated as optional, so a missing or unreadable file does not abort plugin startup. |
| 521 | supported_platforms: |
| 522 | include: |
| 523 | - Linux |
| 524 | exclude: [] |
| 525 | multi_instance: false |
| 526 | additional_permissions: |
| 527 | description: "" |
| 528 | default_behavior: |
| 529 | auto_detection: |
| 530 | description: "Native packages ship the stock DB-IP MMDB files; the plugin auto-detects them at startup. No configuration required for the default install." |
| 531 | limits: |
| 532 | description: "Lookup coverage and freshness depend on the DB-IP Lite files installed on disk. Native packages provide a stock copy; schedule the downloader if you need monthly refreshes." |
| 533 | performance_impact: |
| 534 | description: "Lookups are local MMDB reads with no per-flow network call. Memory use is mostly the mapped database files and the kernel page cache needed to keep active pages hot." |
| 535 | setup: |
| 536 | prerequisites: |
| 537 | list: |
| 538 | - title: DB-IP MMDB files |
| 539 | description: | |
| 540 | Native DEB / RPM packages ship the stock DB-IP MMDB files under |
| 541 | `/usr/share/netdata/topology-ip-intel/`. For source builds, or to get a fresher copy |
| 542 | than the one bundled with the package, run `topology-ip-intel-downloader` once when |
| 543 | the binary is available (packaged 32-bit installs do not include it) to populate |
| 544 | `/var/cache/netdata/topology-ip-intel/`: |
| 545 | |
| 546 | ```bash |
| 547 | sudo /usr/sbin/topology-ip-intel-downloader |
| 548 | ``` |
| 549 | |
| 550 | See the [Enrichment Intel Downloader](https://learn.netdata.cloud/docs/network-flows/enrichment-intel-downloader) |
| 551 | page for downloader options and how to schedule periodic refreshes. DB-IP Lite |
| 552 | data is published monthly, so a monthly cron of the downloader is the right |
| 553 | cadence -- more frequent runs will not produce newer data. |
| 554 | - title: Licence acknowledgement |
| 555 | description: | |
| 556 | DB-IP Lite databases are distributed under the Creative Commons Attribution 4.0 |
| 557 | International licence (https://db-ip.com/db/lite.php). Attribution is required |
| 558 | when redistributing the data or derivative dashboards. |
| 559 | configuration: |
| 560 | file: |
| 561 | name: netflow.yaml |
| 562 | options: |
| 563 | description: | |
| 564 | Configure DB-IP under `enrichment.geoip` in `netflow.yaml`. Empty `asn_database` |
| 565 | and `geo_database` enable auto-detection. |
| 566 | folding: |
| 567 | title: Config options |
| 568 | enabled: true |
| 569 | list: |
| 570 | - name: enrichment.geoip.asn_database |
| 571 | description: List of MMDB paths providing AS data. Empty = auto-detect under cache/stock dirs. |
| 572 | default_value: "[] (auto-detect)" |
| 573 | required: false |
| 574 | - name: enrichment.geoip.geo_database |
| 575 | description: List of MMDB paths providing geo data. Empty = auto-detect. |
| 576 | default_value: "[] (auto-detect)" |
| 577 | required: false |
| 578 | - name: enrichment.geoip.optional |
| 579 | description: When true, missing or unreadable MMDBs are warnings, not fatal. Auto-detected files default to optional. |
| 580 | default_value: "false (true when auto-detected)" |
| 581 | required: false |
| 582 | examples: |
| 583 | folding: |
| 584 | title: Config |
| 585 | enabled: true |
| 586 | list: |
| 587 | - name: Default (auto-detect stock files) |
| 588 | folding: |
| 589 | enabled: false |
| 590 | description: Native package install. No explicit configuration; the plugin finds the stock or cache copy automatically. |
| 591 | config: | |
| 592 | enrichment: |
| 593 | geoip: |
| 594 | asn_database: [] |
| 595 | geo_database: [] |
| 596 | optional: true |
| 597 | - name: Explicit DB-IP paths |
| 598 | folding: |
| 599 | enabled: true |
| 600 | description: Override auto-detection by pointing to specific DB-IP MMDBs (for example, after running the downloader to a non-standard location). |
| 601 | config: | |
| 602 | enrichment: |
| 603 | geoip: |
| 604 | asn_database: |
| 605 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 606 | geo_database: |
| 607 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 608 | optional: false |
| 609 | troubleshooting: |
| 610 | problems: |
| 611 | list: |
| 612 | - name: Private IPs have empty GeoIP fields |
| 613 | description: | |
| 614 | GeoIP databases normally have no country, city, or coordinate entry for RFC 1918 / |
| 615 | private space. The DB-IP-built ASN database tags private ranges so `*_AS_NAME` |
| 616 | renders as `AS0 Private IP Address Space`, while geographic fields stay empty and |
| 617 | private addresses do not appear on maps. Declare your internal CIDRs under |
| 618 | `enrichment.networks` when you want internal labels -- see |
| 619 | [Static metadata](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 620 | - name: Stale databases |
| 621 | description: | |
| 622 | The plugin does not alert on staleness. Check file mtime: |
| 623 | `ls -la /var/cache/netdata/topology-ip-intel/`. DB-IP Lite is published monthly, |
| 624 | so a monthly cron of `/usr/sbin/topology-ip-intel-downloader` keeps you on the |
| 625 | upstream cadence when the downloader is installed; running it more often will not |
| 626 | produce fresher data. |
| 627 | - name: Map renders empty over a long time window |
| 628 | description: | |
| 629 | `SRC_GEO_CITY`, `DST_GEO_CITY`, `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE`, |
| 630 | `SRC_GEO_LONGITUDE`, and `DST_GEO_LONGITUDE` are stored only in the raw journal |
| 631 | tier; the 1-minute, 5-minute, and 1-hour rollups drop them to keep cardinality |
| 632 | bounded. A query that auto-falls back to a rollup tier therefore renders an empty |
| 633 | city map. Narrow the time range so the query fits the raw tier, or use the |
| 634 | country / state map (those survive into rollups). |
| 635 | alerts: [] |
| 636 | metrics: |
| 637 | folding: |
| 638 | title: Metrics |
| 639 | enabled: false |
| 640 | description: | |
| 641 | IP intelligence enriches existing flow records; it does not produce metrics of its own. |
| 642 | Verify enrichment is working by querying `SRC_COUNTRY` / `DST_COUNTRY` on the |
| 643 | Network Flows view and confirming non-empty values for public IPs. |
| 644 | availability: [] |
| 645 | scopes: [] |
| 646 | - meta: |
| 647 | plugin_name: netflow-plugin |
| 648 | module_name: maxmind |
| 649 | monitored_instance: |
| 650 | name: MaxMind GeoIP / GeoLite2 |
| 651 | link: https://www.maxmind.com/ |
| 652 | categories: |
| 653 | - flows.enrichment-methods |
| 654 | icon_filename: network-wired.svg |
| 655 | keywords: |
| 656 | - maxmind |
| 657 | - geoip2 |
| 658 | - geolite2 |
| 659 | - geoip |
| 660 | - asn |
| 661 | - mmdb |
| 662 | - ip intelligence |
| 663 | - flow enrichment |
| 664 | related_resources: |
| 665 | integrations: |
| 666 | list: |
| 667 | - plugin_name: netflow-plugin |
| 668 | module_name: dbip |
| 669 | - plugin_name: netflow-plugin |
| 670 | module_name: iptoasn |
| 671 | - plugin_name: netflow-plugin |
| 672 | module_name: caida-prefix2as |
| 673 | - plugin_name: netflow-plugin |
| 674 | module_name: ip2location |
| 675 | - plugin_name: netflow-plugin |
| 676 | module_name: ipdeny |
| 677 | - plugin_name: netflow-plugin |
| 678 | module_name: ipip |
| 679 | - plugin_name: netflow-plugin |
| 680 | module_name: custom-mmdb |
| 681 | info_provided_to_referring_integrations: |
| 682 | description: "" |
| 683 | overview: |
| 684 | data_collection: |
| 685 | metrics_description: | |
| 686 | Enrich network flows with ASN and geographic context from MaxMind GeoIP / |
| 687 | GeoLite2 MMDB databases. MaxMind is the originator of the MMDB binary format and |
| 688 | the canonical commercial source for geo and ASN data. The netflow plugin reads |
| 689 | MaxMind MMDB files directly; they are interchangeable with other MMDB providers |
| 690 | (DB-IP, custom builds). |
| 691 | |
| 692 | Two product lines are supported: |
| 693 | |
| 694 | - **GeoLite2** (free): requires a free MaxMind account and a license key; users |
| 695 | must accept the GeoLite2 EULA, which includes an attribution requirement. See |
| 696 | <https://dev.maxmind.com/geoip/geolite2-free-geolocation-data>. |
| 697 | - **GeoIP2** (commercial): paid subscription with the same license-key download |
| 698 | mechanism but higher accuracy and more frequent updates. See |
| 699 | <https://www.maxmind.com/en/geoip2-databases>. |
| 700 | |
| 701 | Each lookup populates the same flow-record fields as DB-IP: `SRC_COUNTRY`, |
| 702 | `DST_COUNTRY`, `SRC_GEO_STATE`, `DST_GEO_STATE`, `SRC_GEO_CITY`, `DST_GEO_CITY`, |
| 703 | `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE`, `SRC_GEO_LONGITUDE`, `DST_GEO_LONGITUDE`, |
| 704 | `SRC_AS_NAME`, and `DST_AS_NAME`. Country and state survive into all rollup tiers; |
| 705 | city and lat/lon are **raw-tier-only**. Rollup tiers keep only the |
| 706 | bounded-cardinality geography columns. |
| 707 | |
| 708 | For the cross-cutting concept (resolver poll loop, multi-database composition, |
| 709 | ASN provider chain, failure-modes table), see |
| 710 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 711 | method_description: | |
| 712 | You can use MaxMind in two ways: |
| 713 | |
| 714 | - Use `geoipupdate` (MaxMind's official downloader, configured with your account |
| 715 | ID + license key) or direct HTTPS download, then point `asn_database` and |
| 716 | `geo_database` at the resulting MaxMind MMDB files. |
| 717 | - Use Netdata's `topology-ip-intel-downloader` with `MAXMIND_LICENSE_KEY` in |
| 718 | the environment to fetch GeoLite2 ASN and GeoLite2 Country CSV sources and |
| 719 | generate the standard Netdata topology MMDB files. |
| 720 | |
| 721 | MaxMind requires authentication for all downloads -- there is no anonymous |
| 722 | access. Commercial GeoIP2 databases and richer GeoLite2 City files should still |
| 723 | be maintained with `geoipupdate` or direct paths when you want city and |
| 724 | coordinate enrichment. |
| 725 | |
| 726 | Once the files are on disk, point `asn_database` and `geo_database` at their |
| 727 | paths in `netflow.yaml`. The plugin reloads on file change automatically -- a |
| 728 | successful `geoipupdate` run causes the plugin to swap readers within ~30s, no |
| 729 | restart needed. |
| 730 | supported_platforms: |
| 731 | include: |
| 732 | - Linux |
| 733 | exclude: [] |
| 734 | multi_instance: false |
| 735 | additional_permissions: |
| 736 | description: "" |
| 737 | default_behavior: |
| 738 | auto_detection: |
| 739 | description: "Arbitrary MaxMind paths are not auto-detected; configure `enrichment.geoip.asn_database` and `enrichment.geoip.geo_database` for those. MaxMind-derived topology MMDB files generated by `topology-ip-intel-downloader` are auto-detected because they use the standard Netdata cache paths." |
| 740 | limits: |
| 741 | description: "Lookup coverage, fields, license terms, and refresh cadence depend on the MaxMind database edition you configure. The plugin itself does no network fetching; the downloader is an operator-run refresh step." |
| 742 | performance_impact: |
| 743 | description: "Lookups are local MMDB reads with no per-flow network call. Memory use is mostly the mapped database files and the kernel page cache needed to keep active pages hot." |
| 744 | setup: |
| 745 | prerequisites: |
| 746 | list: |
| 747 | - title: MaxMind account and license key |
| 748 | description: | |
| 749 | MaxMind does not allow anonymous downloads. Required for both GeoLite2 (free) |
| 750 | and GeoIP2 (paid): |
| 751 | |
| 752 | 1. Sign up at <https://www.maxmind.com/en/geolite2/signup> (GeoLite2) or |
| 753 | purchase a GeoIP2 subscription. |
| 754 | 2. Generate a license key under "Account > Manage License Keys". |
| 755 | 3. Note your account ID (visible on the same account page). |
| 756 | - title: Install and configure geoipupdate |
| 757 | description: | |
| 758 | `geoipupdate` is MaxMind's official downloader. Install it from your distro |
| 759 | (`apt install geoipupdate`, `dnf install geoipupdate`) or download from |
| 760 | <https://github.com/maxmind/geoipupdate/releases>. |
| 761 | |
| 762 | Edit `/etc/GeoIP.conf`: |
| 763 | |
| 764 | ```ini |
| 765 | AccountID YOUR_ACCOUNT_ID |
| 766 | LicenseKey YOUR_LICENSE_KEY |
| 767 | # GeoLite2 (free): |
| 768 | EditionIDs GeoLite2-City GeoLite2-ASN |
| 769 | # ...or for GeoIP2 (paid): |
| 770 | # EditionIDs GeoIP2-City GeoIP2-ISP |
| 771 | DatabaseDirectory /usr/share/GeoIP |
| 772 | ``` |
| 773 | |
| 774 | Run `sudo geoipupdate` to fetch initial files, then schedule it via cron or |
| 775 | the bundled `geoipupdate.timer` systemd unit. Recommended cadences match the |
| 776 | upstream publish frequency: |
| 777 | |
| 778 | - **GeoLite2 City / Country**: published twice weekly (Tuesday and Friday). |
| 779 | - **GeoLite2 ASN**: published every weekday (Mon-Fri). |
| 780 | - **GeoIP2 commercial editions**: published "every weekday, Monday through |
| 781 | Friday" per <https://www.maxmind.com/en/geoip2-databases>. |
| 782 | |
| 783 | Source: <https://support.maxmind.com/hc/en-us/articles/4408216129947>. |
| 784 | - title: Optional Netdata topology downloader path |
| 785 | description: | |
| 786 | To generate Netdata's auto-detected topology MMDB files from GeoLite2 ASN |
| 787 | and GeoLite2 Country, run: |
| 788 | |
| 789 | ```bash |
| 790 | sudo MAXMIND_LICENSE_KEY="YOUR_LICENSE_KEY" \ |
| 791 | /usr/sbin/topology-ip-intel-downloader \ |
| 792 | --asn maxmind:geolite2-asn \ |
| 793 | --geo maxmind:geolite2-country |
| 794 | ``` |
| 795 | |
| 796 | This produces `/var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb` |
| 797 | and `/var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb`. The |
| 798 | generated metadata redacts URL query strings; do not put the license key |
| 799 | directly in committed config files. |
| 800 | - title: Single MMDB per database type (dual-stack) |
| 801 | description: | |
| 802 | A single MaxMind MMDB file covers both IPv4 and IPv6 in one binary tree -- |
| 803 | you do **not** need separate IPv4 and IPv6 files. Point `asn_database` at one |
| 804 | file and `geo_database` at one file. (The CSV exports MaxMind also publishes |
| 805 | are split per family, but the netflow plugin only reads the binary MMDB |
| 806 | format.) |
| 807 | configuration: |
| 808 | file: |
| 809 | name: netflow.yaml |
| 810 | options: |
| 811 | description: | |
| 812 | Override the default DB-IP auto-detection by pointing `asn_database` and |
| 813 | `geo_database` at your MaxMind MMDB files. Both keys take a list -- you can |
| 814 | chain MaxMind alongside other MMDB sources; see the |
| 815 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment) |
| 816 | page for the per-field "last non-empty wins" composition rule. |
| 817 | folding: |
| 818 | title: Config options |
| 819 | enabled: true |
| 820 | list: |
| 821 | - name: enrichment.geoip.asn_database |
| 822 | description: Paths to MaxMind ASN MMDB files (typically `GeoLite2-ASN.mmdb` for the free tier or `GeoIP2-ISP.mmdb` for the paid tier; the dual-stack file covers IPv4 and IPv6). |
| 823 | default_value: "[]" |
| 824 | required: true |
| 825 | - name: enrichment.geoip.geo_database |
| 826 | description: Paths to MaxMind geographic MMDB files (typically `GeoLite2-City.mmdb` for the free tier or `GeoIP2-City.mmdb` for the paid tier; the dual-stack file covers IPv4 and IPv6). |
| 827 | default_value: "[]" |
| 828 | required: true |
| 829 | - name: enrichment.geoip.optional |
| 830 | description: When true, missing or unreadable MMDBs are warnings, not fatal at startup. Useful while `geoipupdate` is being set up for the first time. |
| 831 | default_value: "false" |
| 832 | required: false |
| 833 | examples: |
| 834 | folding: |
| 835 | title: Config |
| 836 | enabled: true |
| 837 | list: |
| 838 | - name: GeoLite2 (free tier) |
| 839 | folding: |
| 840 | enabled: false |
| 841 | description: Standard `geoipupdate` install path. Free tier requires a MaxMind account and license key; usage is subject to the GeoLite2 EULA (attribution required). |
| 842 | config: | |
| 843 | enrichment: |
| 844 | geoip: |
| 845 | asn_database: |
| 846 | - /usr/share/GeoIP/GeoLite2-ASN.mmdb |
| 847 | geo_database: |
| 848 | - /usr/share/GeoIP/GeoLite2-City.mmdb |
| 849 | optional: false |
| 850 | - name: GeoIP2 (paid) |
| 851 | folding: |
| 852 | enabled: true |
| 853 | description: Commercial subscription. Higher accuracy than GeoLite2 and a weekday update cadence on most editions. |
| 854 | config: | |
| 855 | enrichment: |
| 856 | geoip: |
| 857 | asn_database: |
| 858 | - /usr/share/GeoIP/GeoIP2-ISP.mmdb |
| 859 | geo_database: |
| 860 | - /usr/share/GeoIP/GeoIP2-City.mmdb |
| 861 | optional: false |
| 862 | - name: MaxMind ASN + DB-IP geo (mixed) |
| 863 | folding: |
| 864 | enabled: true |
| 865 | description: Use MaxMind for ASN richness and keep the bundled DB-IP geo file. Both lookups run; the multi-DB composition rule is documented on the [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment) page. |
| 866 | config: | |
| 867 | enrichment: |
| 868 | geoip: |
| 869 | asn_database: |
| 870 | - /usr/share/GeoIP/GeoLite2-ASN.mmdb |
| 871 | geo_database: |
| 872 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 873 | optional: false |
| 874 | troubleshooting: |
| 875 | problems: |
| 876 | list: |
| 877 | - name: License key missing or expired |
| 878 | description: | |
| 879 | `geoipupdate` fails (often silently from the plugin's perspective) and the |
| 880 | MMDB files become stale. Check the `geoipupdate` exit status and its log; |
| 881 | regenerate the key under "Account > Manage License Keys" if needed and |
| 882 | update `/etc/GeoIP.conf`. |
| 883 | - name: GeoLite2 attribution requirement |
| 884 | description: | |
| 885 | The GeoLite2 EULA requires attribution to MaxMind. If you redistribute |
| 886 | dashboards or screenshots that include GeoLite2-derived data, follow the |
| 887 | upstream guidance at <https://dev.maxmind.com/geoip/geolite2-free-geolocation-data>. |
| 888 | GeoIP2 (commercial) does not have the same attribution requirement. |
| 889 | - name: Schema differences between GeoLite2 and GeoIP2 |
| 890 | description: | |
| 891 | Both share the standard MMDB structure for the fields the plugin reads |
| 892 | (country, subdivisions, city, location, autonomous_system_number, |
| 893 | autonomous_system_organization). MaxMind's specialty databases (Anonymous IP, |
| 894 | Connection Type, Domain) carry GeoIP2-only fields the plugin does not consume. |
| 895 | Use `City` for geographic enrichment and `ASN` (GeoLite2) or `ISP` (GeoIP2) for |
| 896 | AS data. |
| 897 | - name: Mixing MaxMind with the bundled DB-IP files |
| 898 | description: | |
| 899 | The plugin loads every configured MMDB. If you list both, the per-field |
| 900 | "last non-empty wins" rule applies (see the |
| 901 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment) |
| 902 | concept page). Order matters: list the source whose values you want to win |
| 903 | **last**. |
| 904 | alerts: [] |
| 905 | metrics: |
| 906 | folding: |
| 907 | title: Metrics |
| 908 | enabled: false |
| 909 | description: | |
| 910 | IP intelligence enriches existing flow records; it produces no metrics of its own. |
| 911 | Verify enrichment is working by querying `SRC_COUNTRY` / `DST_COUNTRY` and |
| 912 | `SRC_AS_NAME` / `DST_AS_NAME` on the Network Flows view and confirming non-empty |
| 913 | values for public IPs. |
| 914 | availability: [] |
| 915 | scopes: [] |
| 916 | - meta: |
| 917 | plugin_name: netflow-plugin |
| 918 | module_name: iptoasn |
| 919 | monitored_instance: |
| 920 | name: IPtoASN |
| 921 | link: https://iptoasn.com/ |
| 922 | categories: |
| 923 | - flows.enrichment-methods |
| 924 | icon_filename: network-wired.svg |
| 925 | keywords: |
| 926 | - iptoasn |
| 927 | - asn |
| 928 | - bgp |
| 929 | - public asn |
| 930 | - public domain |
| 931 | - pddl |
| 932 | - ip intelligence |
| 933 | related_resources: |
| 934 | integrations: |
| 935 | list: |
| 936 | - plugin_name: netflow-plugin |
| 937 | module_name: dbip |
| 938 | - plugin_name: netflow-plugin |
| 939 | module_name: maxmind |
| 940 | - plugin_name: netflow-plugin |
| 941 | module_name: caida-prefix2as |
| 942 | - plugin_name: netflow-plugin |
| 943 | module_name: ip2location |
| 944 | - plugin_name: netflow-plugin |
| 945 | module_name: ipdeny |
| 946 | - plugin_name: netflow-plugin |
| 947 | module_name: ipip |
| 948 | - plugin_name: netflow-plugin |
| 949 | module_name: custom-mmdb |
| 950 | info_provided_to_referring_integrations: |
| 951 | description: "" |
| 952 | overview: |
| 953 | data_collection: |
| 954 | metrics_description: | |
| 955 | Enrich network flows with public ASN and country context from IPtoASN |
| 956 | BGP-derived datasets. [IPtoASN](https://iptoasn.com/) is a free, public-domain |
| 957 | feed of IP-to-ASN mappings. The upstream publishes its dataset |
| 958 | under the [Public Domain Dedication and License (PDDL)](https://opendatacommons.org/licenses/pddl/) |
| 959 | (see [iptoasn.com](https://iptoasn.com/) -- "Frequently updated, public domain") |
| 960 | and rebuilds it hourly, making it the highest-cadence and lowest-friction public |
| 961 | ASN source available. Use it as a free, open alternative to MaxMind ASN data when |
| 962 | license cost or terms matter. |
| 963 | |
| 964 | IPtoASN's `ip2asn-combined` artifact contains five tab-separated columns: |
| 965 | `range_start`, `range_end`, `AS_number`, `country_code`, and `AS_description` |
| 966 | (see [iptoasn.com](https://iptoasn.com/) for the format reference). So |
| 967 | although IPtoASN is often described as "ASN-only", the feed also carries an |
| 968 | ISO 3166 country code per range and the Netdata downloader exposes both as |
| 969 | ASN + geo MMDBs. Pair with DB-IP or MaxMind when you also need state, city, |
| 970 | or coordinates. |
| 971 | |
| 972 | For the IP-intelligence mechanism (lookup order, MMDB composition, refresh, |
| 973 | provider chains), see |
| 974 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 975 | method_description: | |
| 976 | The Netdata plugin reads MMDB only. IPtoASN ships gzipped TSV |
| 977 | (`https://iptoasn.com/data/ip2asn-combined.tsv.gz`), so the TSV must be |
| 978 | converted to MMDB before the plugin can use it. |
| 979 | |
| 980 | When available, Netdata provides a converter for this exact purpose: the |
| 981 | `topology-ip-intel-downloader` Go tool fetches the upstream TSV, parses it, |
| 982 | and emits MMDB files the plugin auto-detects. This is |
| 983 | a separate operator step (cron); the plugin itself does no fetching or |
| 984 | conversion. Packaged 32-bit installs ship the stock MMDB payload but do not include |
| 985 | the downloader binary. If you prefer not to use the bundled tool, any third-party |
| 986 | IPtoASN-to-MMDB converter that produces a standard |
| 987 | [MaxMind DB](https://maxmind.github.io/MaxMind-DB/) with `iso_code`, |
| 988 | `autonomous_system_number`, and `autonomous_system_organization` fields |
| 989 | will also work with the Custom MMDB integration. |
| 990 | supported_platforms: |
| 991 | include: |
| 992 | - Linux |
| 993 | exclude: [] |
| 994 | multi_instance: false |
| 995 | additional_permissions: |
| 996 | description: "" |
| 997 | default_behavior: |
| 998 | auto_detection: |
| 999 | description: | |
| 1000 | Not auto-detected as the default ASN source -- the plugin auto-detects |
| 1001 | DB-IP MMDBs in the cache directory at startup |
| 1002 | (see [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment)). |
| 1003 | To use IPtoASN, run `topology-ip-intel-downloader` when available to produce IPtoASN-derived |
| 1004 | MMDBs (the downloader writes to the same cache paths the plugin scans, |
| 1005 | so once present they are picked up automatically on the 30-second |
| 1006 | file-signature check). |
| 1007 | limits: |
| 1008 | description: "IPtoASN provides AS number and country-level data only. It does not provide city, coordinates, or rich organization names unless you layer another MMDB source." |
| 1009 | performance_impact: |
| 1010 | description: "Lookups are local MMDB reads after the downloader converts the TSV feed. Memory use is mostly the mapped database files and the kernel page cache needed to keep active pages hot." |
| 1011 | setup: |
| 1012 | prerequisites: |
| 1013 | list: |
| 1014 | - title: Run the downloader with IPtoASN as ASN source |
| 1015 | description: | |
| 1016 | When available, the downloader knows how to fetch the upstream TSV and convert |
| 1017 | it to MMDB. Packaged 32-bit installs do not include this binary: |
| 1018 | |
| 1019 | ```bash |
| 1020 | sudo /usr/sbin/topology-ip-intel-downloader \ |
| 1021 | --asn iptoasn:combined \ |
| 1022 | --geo iptoasn:combined |
| 1023 | ``` |
| 1024 | |
| 1025 | `iptoasn:combined` is valid for both `--asn` and `--geo` because the |
| 1026 | upstream TSV carries both AS data and country. If you |
| 1027 | want richer geographic data (state, city, coordinates), pair IPtoASN |
| 1028 | ASN with DB-IP or MaxMind geo: |
| 1029 | |
| 1030 | ```bash |
| 1031 | sudo /usr/sbin/topology-ip-intel-downloader \ |
| 1032 | --asn iptoasn:combined \ |
| 1033 | --geo dbip:city-lite |
| 1034 | ``` |
| 1035 | |
| 1036 | IPtoASN rebuilds hourly upstream, but a daily cron is sufficient for |
| 1037 | flow enrichment (ASN ownership rarely changes within a single day). |
| 1038 | Do not exceed hourly -- you only get a fresh dataset once an hour |
| 1039 | regardless. |
| 1040 | configuration: |
| 1041 | file: |
| 1042 | name: netflow.yaml |
| 1043 | options: |
| 1044 | description: | |
| 1045 | Once the downloader has produced IPtoASN-derived MMDB files in the cache |
| 1046 | directory, the plugin auto-detects them. To pin the path explicitly, set |
| 1047 | `enrichment.geoip.asn_database` (and `geo_database` when you also want |
| 1048 | the country data IPtoASN provides). |
| 1049 | folding: |
| 1050 | title: Config options |
| 1051 | enabled: true |
| 1052 | list: |
| 1053 | - name: enrichment.geoip.asn_database |
| 1054 | description: Path to the IPtoASN-derived ASN MMDB. Empty = auto-detect from cache directory. |
| 1055 | default_value: "[]" |
| 1056 | required: false |
| 1057 | - name: enrichment.geoip.geo_database |
| 1058 | description: Path to the IPtoASN-derived geo MMDB (country only). Empty = auto-detect; or point at a richer DB-IP / MaxMind geo MMDB for state/city/coordinates. |
| 1059 | default_value: "[]" |
| 1060 | required: false |
| 1061 | - name: enrichment.geoip.optional |
| 1062 | description: When true, missing or unreadable MMDBs are warnings, not fatal at startup. |
| 1063 | default_value: "false (true when auto-detected)" |
| 1064 | required: false |
| 1065 | examples: |
| 1066 | folding: |
| 1067 | title: Config |
| 1068 | enabled: true |
| 1069 | list: |
| 1070 | - name: After running the downloader with IPtoASN |
| 1071 | folding: |
| 1072 | enabled: false |
| 1073 | description: Auto-detection picks up the cache copy. Both ASN and country come from IPtoASN. |
| 1074 | config: | |
| 1075 | enrichment: |
| 1076 | geoip: |
| 1077 | asn_database: [] |
| 1078 | geo_database: [] |
| 1079 | optional: true |
| 1080 | - name: Explicit IPtoASN paths (ASN + country only) |
| 1081 | folding: |
| 1082 | enabled: true |
| 1083 | description: | |
| 1084 | Pin the IPtoASN-derived MMDBs explicitly. Country only -- no |
| 1085 | state/city. Use the DB-IP or MaxMind integration cards for richer |
| 1086 | geographic enrichment. |
| 1087 | config: | |
| 1088 | enrichment: |
| 1089 | geoip: |
| 1090 | asn_database: |
| 1091 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 1092 | geo_database: |
| 1093 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 1094 | optional: false |
| 1095 | - name: IPtoASN ASN + DB-IP city-lite geo |
| 1096 | folding: |
| 1097 | enabled: true |
| 1098 | description: | |
| 1099 | Free, fully public-domain ASN combined with DB-IP city-lite for |
| 1100 | state/city/coordinates. Run the downloader as |
| 1101 | `topology-ip-intel-downloader --asn iptoasn:combined --geo dbip:city-lite` |
| 1102 | and let auto-detection handle the rest. |
| 1103 | config: | |
| 1104 | enrichment: |
| 1105 | geoip: |
| 1106 | asn_database: [] |
| 1107 | geo_database: [] |
| 1108 | optional: true |
| 1109 | troubleshooting: |
| 1110 | problems: |
| 1111 | list: |
| 1112 | - name: ASN names not appearing |
| 1113 | description: | |
| 1114 | IPtoASN's `AS_description` column is sometimes empty for less-common |
| 1115 | ASNs (the upstream derives names from RIRs and the chain occasionally |
| 1116 | has gaps). When that happens the plugin renders the AS as `AS{n}` with |
| 1117 | no organisation. This is data-source-level, not a plugin issue. If richer AS-name |
| 1118 | coverage matters, layer a MaxMind GeoLite2-ASN MMDB after IPtoASN in |
| 1119 | `asn_database` -- per-field "last database with a non-empty value |
| 1120 | wins" composition (see |
| 1121 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment)) |
| 1122 | means MaxMind names override IPtoASN's empty entries. |
| 1123 | - name: Country empty although IPtoASN was selected |
| 1124 | description: | |
| 1125 | IPtoASN's combined TSV publishes country only when it is known. Public |
| 1126 | IPs that the upstream cannot attribute will have an empty |
| 1127 | `*_COUNTRY`. If you need broader country coverage, point |
| 1128 | `geo_database` at a DB-IP or MaxMind country MMDB instead -- the |
| 1129 | plugin reads any combination. |
| 1130 | - name: Outdated ASN attribution |
| 1131 | description: | |
| 1132 | IPtoASN rebuilds hourly upstream. A daily cron of the downloader is |
| 1133 | the recommended cadence for flow enrichment; weekly is too slow |
| 1134 | because BGP-driven prefix re-assignments will land in the dataset |
| 1135 | within hours but not in your cache until the next download. The |
| 1136 | plugin reloads MMDB files in place every 30 seconds when the file |
| 1137 | signature changes, so a fresh file lands without restart. |
| 1138 | alerts: [] |
| 1139 | metrics: |
| 1140 | folding: |
| 1141 | title: Metrics |
| 1142 | enabled: false |
| 1143 | description: | |
| 1144 | Enriches flow records with AS numbers, AS names, and country codes; produces |
| 1145 | no metrics of its own. Verify on the Network Flows view via `*_AS_NAME` and |
| 1146 | `*_COUNTRY` columns. |
| 1147 | availability: [] |
| 1148 | scopes: [] |
| 1149 | - meta: |
| 1150 | plugin_name: netflow-plugin |
| 1151 | module_name: caida-prefix2as |
| 1152 | monitored_instance: |
| 1153 | name: CAIDA Routeviews Prefix-to-AS |
| 1154 | link: https://www.caida.org/catalog/datasets/routeviews-prefix2as/ |
| 1155 | categories: |
| 1156 | - flows.enrichment-methods |
| 1157 | icon_filename: network-wired.svg |
| 1158 | keywords: |
| 1159 | - caida |
| 1160 | - routeviews |
| 1161 | - prefix2as |
| 1162 | - pfx2as |
| 1163 | - asn |
| 1164 | - bgp |
| 1165 | - ip intelligence |
| 1166 | - flow enrichment |
| 1167 | related_resources: |
| 1168 | integrations: |
| 1169 | list: |
| 1170 | - plugin_name: netflow-plugin |
| 1171 | module_name: dbip |
| 1172 | - plugin_name: netflow-plugin |
| 1173 | module_name: maxmind |
| 1174 | - plugin_name: netflow-plugin |
| 1175 | module_name: iptoasn |
| 1176 | - plugin_name: netflow-plugin |
| 1177 | module_name: custom-mmdb |
| 1178 | info_provided_to_referring_integrations: |
| 1179 | description: "" |
| 1180 | overview: |
| 1181 | data_collection: |
| 1182 | metrics_description: | |
| 1183 | Enrich network flows with AS numbers from CAIDA's Routeviews Prefix-to-AS |
| 1184 | mapping dataset. CAIDA derives this public dataset from RouteViews BGP data and |
| 1185 | publishes one prefix-to-origin-AS mapping per line. The official format is |
| 1186 | tab-separated `IP prefix`, `prefix length`, and `AS number`; multi-origin AS |
| 1187 | entries can contain multiple ASNs, and the Netdata downloader keeps the first |
| 1188 | listed origin AS as the simplified per-prefix value. |
| 1189 | |
| 1190 | CAIDA Prefix-to-AS is ASN-only. It does not provide AS organization names or |
| 1191 | geographic fields. Pair it with DB-IP, MaxMind, IP2Location, IPDeny, IPIP, or |
| 1192 | another geo provider when you also need `*_COUNTRY`, city, state, or |
| 1193 | coordinates. |
| 1194 | |
| 1195 | For the IP-intelligence mechanism (lookup order, MMDB composition, refresh, |
| 1196 | provider chains), see |
| 1197 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 1198 | method_description: | |
| 1199 | The netflow plugin reads MMDB files only. CAIDA publishes compressed text data, |
| 1200 | so use Netdata's `topology-ip-intel-downloader` to fetch the latest Routeviews |
| 1201 | Prefix-to-AS file, convert it to `topology-ip-asn.mmdb`, and place it in the |
| 1202 | standard auto-detected cache directory. The plugin reloads the generated MMDB |
| 1203 | automatically when the file changes. |
| 1204 | supported_platforms: |
| 1205 | include: |
| 1206 | - Linux |
| 1207 | exclude: [] |
| 1208 | multi_instance: false |
| 1209 | additional_permissions: |
| 1210 | description: "" |
| 1211 | default_behavior: |
| 1212 | auto_detection: |
| 1213 | description: "Not used by the default install. After the downloader writes `topology-ip-asn.mmdb` into the standard cache path, the plugin auto-detects the generated file." |
| 1214 | limits: |
| 1215 | description: "ASN number only. No AS organization name, country, state, city, or coordinates are available from this source." |
| 1216 | performance_impact: |
| 1217 | description: "Lookups are local MMDB reads after conversion. Memory use is mostly the mapped database file and the kernel page cache needed to keep active pages hot." |
| 1218 | setup: |
| 1219 | prerequisites: |
| 1220 | list: |
| 1221 | - title: Downloader with CAIDA Prefix-to-AS support |
| 1222 | description: | |
| 1223 | Run the bundled downloader when available. Packaged 32-bit installs do not |
| 1224 | include this binary: |
| 1225 | |
| 1226 | ```bash |
| 1227 | sudo /usr/sbin/topology-ip-intel-downloader \ |
| 1228 | --asn caida:prefix2as \ |
| 1229 | --geo dbip:city-lite |
| 1230 | ``` |
| 1231 | |
| 1232 | `caida:prefix2as` is valid only for `--asn`. The example keeps DB-IP |
| 1233 | city-lite for geo fields; substitute any supported `--geo` provider if |
| 1234 | country-only enrichment is sufficient. |
| 1235 | - title: CAIDA acceptable use terms |
| 1236 | description: | |
| 1237 | CAIDA documents acceptable-use and attribution requirements on the dataset |
| 1238 | page. Review the terms before redistributing generated datasets or derived |
| 1239 | reports. |
| 1240 | configuration: |
| 1241 | file: |
| 1242 | name: netflow.yaml |
| 1243 | options: |
| 1244 | description: | |
| 1245 | Once the downloader has produced a CAIDA-derived ASN MMDB in the cache |
| 1246 | directory, the plugin auto-detects it. To pin paths explicitly, set |
| 1247 | `enrichment.geoip.asn_database` and keep `geo_database` pointed at your chosen |
| 1248 | geographic provider. |
| 1249 | folding: |
| 1250 | title: Config options |
| 1251 | enabled: true |
| 1252 | list: |
| 1253 | - name: enrichment.geoip.asn_database |
| 1254 | description: Path to the CAIDA-derived ASN MMDB. Empty = auto-detect from the cache directory. |
| 1255 | default_value: "[]" |
| 1256 | required: false |
| 1257 | - name: enrichment.geoip.geo_database |
| 1258 | description: Optional geographic MMDB paths supplied by another provider. |
| 1259 | default_value: "[]" |
| 1260 | required: false |
| 1261 | - name: enrichment.geoip.optional |
| 1262 | description: When true, missing or unreadable MMDBs are warnings, not fatal at startup. |
| 1263 | default_value: "false (true when auto-detected)" |
| 1264 | required: false |
| 1265 | examples: |
| 1266 | folding: |
| 1267 | title: Config |
| 1268 | enabled: true |
| 1269 | list: |
| 1270 | - name: After running the downloader with CAIDA ASN and DB-IP geo |
| 1271 | folding: |
| 1272 | enabled: false |
| 1273 | description: Auto-detection picks up both generated cache files. |
| 1274 | config: | |
| 1275 | enrichment: |
| 1276 | geoip: |
| 1277 | asn_database: [] |
| 1278 | geo_database: [] |
| 1279 | optional: true |
| 1280 | - name: Explicit CAIDA ASN path |
| 1281 | folding: |
| 1282 | enabled: true |
| 1283 | description: Pin CAIDA for AS numbers and keep a separate geo source for country/city fields. |
| 1284 | config: | |
| 1285 | enrichment: |
| 1286 | geoip: |
| 1287 | asn_database: |
| 1288 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 1289 | geo_database: |
| 1290 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 1291 | optional: false |
| 1292 | troubleshooting: |
| 1293 | problems: |
| 1294 | list: |
| 1295 | - name: AS names are empty |
| 1296 | description: | |
| 1297 | CAIDA Prefix-to-AS contains AS numbers, not organization names. Layer a |
| 1298 | provider with AS organization data, such as MaxMind GeoLite2 ASN or DB-IP |
| 1299 | ASN Lite, after the CAIDA MMDB in `asn_database` if AS names matter. |
| 1300 | - name: Country fields are empty |
| 1301 | description: | |
| 1302 | This source is ASN-only. Configure a `--geo` provider in the downloader or |
| 1303 | point `enrichment.geoip.geo_database` at a separate geo MMDB. |
| 1304 | - name: Multi-origin AS simplification |
| 1305 | description: | |
| 1306 | CAIDA can represent multi-origin prefixes. The downloader uses the first |
| 1307 | listed AS for the generated single-value MMDB record. If you need full MOAS |
| 1308 | semantics, use BMP or BioRIS routing enrichment instead of an offline ASN |
| 1309 | lookup database. |
| 1310 | alerts: [] |
| 1311 | metrics: |
| 1312 | folding: |
| 1313 | title: Metrics |
| 1314 | enabled: false |
| 1315 | description: | |
| 1316 | Enriches flow records with AS numbers; produces no metrics of its own. |
| 1317 | Verify on the Network Flows view via `SRC_AS` and `DST_AS`. |
| 1318 | availability: [] |
| 1319 | scopes: [] |
| 1320 | - meta: |
| 1321 | plugin_name: netflow-plugin |
| 1322 | module_name: ip2location |
| 1323 | monitored_instance: |
| 1324 | name: IP2Location LITE IP-Country |
| 1325 | link: https://lite.ip2location.com/database/db1-ip-country?lang=en_US |
| 1326 | categories: |
| 1327 | - flows.enrichment-methods |
| 1328 | icon_filename: network-wired.svg |
| 1329 | keywords: |
| 1330 | - ip2location |
| 1331 | - ip2location lite |
| 1332 | - geoip |
| 1333 | - country |
| 1334 | - csv |
| 1335 | - ip intelligence |
| 1336 | - flow enrichment |
| 1337 | related_resources: |
| 1338 | integrations: |
| 1339 | list: |
| 1340 | - plugin_name: netflow-plugin |
| 1341 | module_name: dbip |
| 1342 | - plugin_name: netflow-plugin |
| 1343 | module_name: maxmind |
| 1344 | - plugin_name: netflow-plugin |
| 1345 | module_name: iptoasn |
| 1346 | - plugin_name: netflow-plugin |
| 1347 | module_name: custom-mmdb |
| 1348 | info_provided_to_referring_integrations: |
| 1349 | description: "" |
| 1350 | overview: |
| 1351 | data_collection: |
| 1352 | metrics_description: | |
| 1353 | Enrich network flows with country codes from the IP2Location LITE |
| 1354 | IP-Country database. The supported downloader source is the DB1 LITE CSV ZIP |
| 1355 | (`IP2LOCATION-LITE-DB1.CSV.ZIP`), whose documented fields are |
| 1356 | `ip_from`, `ip_to`, `country_code`, and `country_name`. |
| 1357 | |
| 1358 | IP2Location DB1 is country-only. It does not provide ASN, AS organization, |
| 1359 | state, city, latitude, or longitude. Pair it with CAIDA, IPtoASN, DB-IP, or |
| 1360 | MaxMind ASN data when you also need AS fields. |
| 1361 | |
| 1362 | For the IP-intelligence mechanism (lookup order, MMDB composition, refresh, |
| 1363 | provider chains), see |
| 1364 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 1365 | method_description: | |
| 1366 | The netflow plugin reads MMDB files only. Use Netdata's |
| 1367 | `topology-ip-intel-downloader` to fetch the IP2Location LITE CSV ZIP, convert |
| 1368 | the ranges to `topology-ip-geo.mmdb`, and write the file into the standard |
| 1369 | auto-detected cache directory. The plugin reloads the generated MMDB |
| 1370 | automatically when the file changes. |
| 1371 | supported_platforms: |
| 1372 | include: |
| 1373 | - Linux |
| 1374 | exclude: [] |
| 1375 | multi_instance: false |
| 1376 | additional_permissions: |
| 1377 | description: "" |
| 1378 | default_behavior: |
| 1379 | auto_detection: |
| 1380 | description: "Not used by the default install. After the downloader writes `topology-ip-geo.mmdb` into the standard cache path, the plugin auto-detects the generated file." |
| 1381 | limits: |
| 1382 | description: "Country only. No ASN, AS name, state, city, latitude, or longitude are available from this source." |
| 1383 | performance_impact: |
| 1384 | description: "Lookups are local MMDB reads after conversion. Memory use is mostly the mapped database file and the kernel page cache needed to keep active pages hot." |
| 1385 | setup: |
| 1386 | prerequisites: |
| 1387 | list: |
| 1388 | - title: Downloader with IP2Location LITE support |
| 1389 | description: | |
| 1390 | Run the bundled downloader when available. Packaged 32-bit installs do not |
| 1391 | include this binary: |
| 1392 | |
| 1393 | ```bash |
| 1394 | sudo /usr/sbin/topology-ip-intel-downloader \ |
| 1395 | --asn iptoasn:combined \ |
| 1396 | --geo ip2location:country-lite |
| 1397 | ``` |
| 1398 | |
| 1399 | `ip2location:country-lite` is valid only for `--geo`. The example pairs it |
| 1400 | with IPtoASN so the resulting cache contains both AS and country fields. |
| 1401 | - title: IP2Location LITE license |
| 1402 | description: | |
| 1403 | Review the IP2Location LITE terms before redistributing the data or |
| 1404 | generated derivative files. The plugin consumes only the generated MMDB and |
| 1405 | does not contact IP2Location during flow decoding. |
| 1406 | configuration: |
| 1407 | file: |
| 1408 | name: netflow.yaml |
| 1409 | options: |
| 1410 | description: | |
| 1411 | Once the downloader has produced an IP2Location-derived geo MMDB in the cache |
| 1412 | directory, the plugin auto-detects it. To pin paths explicitly, set |
| 1413 | `enrichment.geoip.geo_database` and keep `asn_database` pointed at your chosen |
| 1414 | ASN provider. |
| 1415 | folding: |
| 1416 | title: Config options |
| 1417 | enabled: true |
| 1418 | list: |
| 1419 | - name: enrichment.geoip.asn_database |
| 1420 | description: Optional ASN MMDB paths supplied by another provider. |
| 1421 | default_value: "[]" |
| 1422 | required: false |
| 1423 | - name: enrichment.geoip.geo_database |
| 1424 | description: Path to the IP2Location-derived geo MMDB. Empty = auto-detect from the cache directory. |
| 1425 | default_value: "[]" |
| 1426 | required: false |
| 1427 | - name: enrichment.geoip.optional |
| 1428 | description: When true, missing or unreadable MMDBs are warnings, not fatal at startup. |
| 1429 | default_value: "false (true when auto-detected)" |
| 1430 | required: false |
| 1431 | examples: |
| 1432 | folding: |
| 1433 | title: Config |
| 1434 | enabled: true |
| 1435 | list: |
| 1436 | - name: After running the downloader with IPtoASN ASN and IP2Location country |
| 1437 | folding: |
| 1438 | enabled: false |
| 1439 | description: Auto-detection picks up both generated cache files. |
| 1440 | config: | |
| 1441 | enrichment: |
| 1442 | geoip: |
| 1443 | asn_database: [] |
| 1444 | geo_database: [] |
| 1445 | optional: true |
| 1446 | - name: Explicit IP2Location country path |
| 1447 | folding: |
| 1448 | enabled: true |
| 1449 | description: Pin IP2Location for country and keep a separate ASN source. |
| 1450 | config: | |
| 1451 | enrichment: |
| 1452 | geoip: |
| 1453 | asn_database: |
| 1454 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 1455 | geo_database: |
| 1456 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 1457 | optional: false |
| 1458 | troubleshooting: |
| 1459 | problems: |
| 1460 | list: |
| 1461 | - name: ASN fields are empty |
| 1462 | description: | |
| 1463 | IP2Location DB1 is country-only. Configure `--asn iptoasn:combined`, |
| 1464 | `--asn caida:prefix2as`, or another ASN provider in the downloader. |
| 1465 | - name: City or coordinates are empty |
| 1466 | description: | |
| 1467 | The DB1 LITE source does not include city, state, latitude, or longitude. |
| 1468 | Use DB-IP city-lite, MaxMind GeoLite2 City, GeoIP2 City, or a custom MMDB |
| 1469 | when those fields are required. |
| 1470 | - name: Stale country data |
| 1471 | description: | |
| 1472 | Check the upstream IP2Location LITE page for the current version and next |
| 1473 | update date, then schedule the downloader accordingly. The plugin reloads |
| 1474 | the generated MMDB in place when the file signature changes. |
| 1475 | alerts: [] |
| 1476 | metrics: |
| 1477 | folding: |
| 1478 | title: Metrics |
| 1479 | enabled: false |
| 1480 | description: | |
| 1481 | Enriches flow records with country codes; produces no metrics of its own. |
| 1482 | Verify on the Network Flows view via `SRC_COUNTRY` and `DST_COUNTRY`. |
| 1483 | availability: [] |
| 1484 | scopes: [] |
| 1485 | - meta: |
| 1486 | plugin_name: netflow-plugin |
| 1487 | module_name: ipdeny |
| 1488 | monitored_instance: |
| 1489 | name: IPDeny Country Zones |
| 1490 | link: https://www.ipdeny.com/ipblocks/ |
| 1491 | categories: |
| 1492 | - flows.enrichment-methods |
| 1493 | icon_filename: network-wired.svg |
| 1494 | keywords: |
| 1495 | - ipdeny |
| 1496 | - country zones |
| 1497 | - geoip |
| 1498 | - country |
| 1499 | - cidr |
| 1500 | - ip intelligence |
| 1501 | - flow enrichment |
| 1502 | related_resources: |
| 1503 | integrations: |
| 1504 | list: |
| 1505 | - plugin_name: netflow-plugin |
| 1506 | module_name: dbip |
| 1507 | - plugin_name: netflow-plugin |
| 1508 | module_name: maxmind |
| 1509 | - plugin_name: netflow-plugin |
| 1510 | module_name: iptoasn |
| 1511 | - plugin_name: netflow-plugin |
| 1512 | module_name: custom-mmdb |
| 1513 | info_provided_to_referring_integrations: |
| 1514 | description: "" |
| 1515 | overview: |
| 1516 | data_collection: |
| 1517 | metrics_description: | |
| 1518 | Enrich network flows with country codes from IPDeny country zone files. |
| 1519 | IPDeny publishes country-specific CIDR lists and an `all-zones.tar.gz` |
| 1520 | archive. The Netdata downloader reads the IPv4 country-zone archive and maps |
| 1521 | each `<country>.zone` member to that ISO country code. |
| 1522 | |
| 1523 | IPDeny Country Zones are country-only and the current built-in source is IPv4 |
| 1524 | only. They do not provide ASN, AS organization, state, city, latitude, or |
| 1525 | longitude. Pair IPDeny with another ASN source when you also need AS fields, |
| 1526 | and use DB-IP, MaxMind, IP2Location, or a custom MMDB when IPv6 country |
| 1527 | coverage is required. |
| 1528 | |
| 1529 | For the IP-intelligence mechanism (lookup order, MMDB composition, refresh, |
| 1530 | provider chains), see |
| 1531 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 1532 | method_description: | |
| 1533 | The netflow plugin reads MMDB files only. Use Netdata's |
| 1534 | `topology-ip-intel-downloader` to fetch IPDeny's HTTPS country-zone archive, |
| 1535 | convert the CIDR lists to `topology-ip-geo.mmdb`, and write the file into the |
| 1536 | standard auto-detected cache directory. The plugin reloads the generated MMDB |
| 1537 | automatically when the file changes. |
| 1538 | supported_platforms: |
| 1539 | include: |
| 1540 | - Linux |
| 1541 | exclude: [] |
| 1542 | multi_instance: false |
| 1543 | additional_permissions: |
| 1544 | description: "" |
| 1545 | default_behavior: |
| 1546 | auto_detection: |
| 1547 | description: "Not used by the default install. After the downloader writes `topology-ip-geo.mmdb` into the standard cache path, the plugin auto-detects the generated file." |
| 1548 | limits: |
| 1549 | description: "Country-only IPv4 source. No ASN, AS name, IPv6 coverage, state, city, latitude, or longitude are available from this built-in source." |
| 1550 | performance_impact: |
| 1551 | description: "Lookups are local MMDB reads after conversion. Memory use is mostly the mapped database file and the kernel page cache needed to keep active pages hot." |
| 1552 | setup: |
| 1553 | prerequisites: |
| 1554 | list: |
| 1555 | - title: Downloader with IPDeny support |
| 1556 | description: | |
| 1557 | Run the bundled downloader when available. Packaged 32-bit installs do not |
| 1558 | include this binary: |
| 1559 | |
| 1560 | ```bash |
| 1561 | sudo /usr/sbin/topology-ip-intel-downloader \ |
| 1562 | --asn iptoasn:combined \ |
| 1563 | --geo ipdeny:country-zones |
| 1564 | ``` |
| 1565 | |
| 1566 | `ipdeny:country-zones` is valid only for `--geo`. The example pairs it |
| 1567 | with IPtoASN so the resulting cache contains both AS and country fields. |
| 1568 | - title: IPDeny usage policy |
| 1569 | description: | |
| 1570 | IPDeny publishes usage limits and copyright policy on its site. Review |
| 1571 | those terms before mirroring, redistributing, or downloading more often |
| 1572 | than needed. |
| 1573 | configuration: |
| 1574 | file: |
| 1575 | name: netflow.yaml |
| 1576 | options: |
| 1577 | description: | |
| 1578 | Once the downloader has produced an IPDeny-derived geo MMDB in the cache |
| 1579 | directory, the plugin auto-detects it. To pin paths explicitly, set |
| 1580 | `enrichment.geoip.geo_database` and keep `asn_database` pointed at your chosen |
| 1581 | ASN provider. |
| 1582 | folding: |
| 1583 | title: Config options |
| 1584 | enabled: true |
| 1585 | list: |
| 1586 | - name: enrichment.geoip.asn_database |
| 1587 | description: Optional ASN MMDB paths supplied by another provider. |
| 1588 | default_value: "[]" |
| 1589 | required: false |
| 1590 | - name: enrichment.geoip.geo_database |
| 1591 | description: Path to the IPDeny-derived geo MMDB. Empty = auto-detect from the cache directory. |
| 1592 | default_value: "[]" |
| 1593 | required: false |
| 1594 | - name: enrichment.geoip.optional |
| 1595 | description: When true, missing or unreadable MMDBs are warnings, not fatal at startup. |
| 1596 | default_value: "false (true when auto-detected)" |
| 1597 | required: false |
| 1598 | examples: |
| 1599 | folding: |
| 1600 | title: Config |
| 1601 | enabled: true |
| 1602 | list: |
| 1603 | - name: After running the downloader with IPtoASN ASN and IPDeny country |
| 1604 | folding: |
| 1605 | enabled: false |
| 1606 | description: Auto-detection picks up both generated cache files. |
| 1607 | config: | |
| 1608 | enrichment: |
| 1609 | geoip: |
| 1610 | asn_database: [] |
| 1611 | geo_database: [] |
| 1612 | optional: true |
| 1613 | - name: Explicit IPDeny country path |
| 1614 | folding: |
| 1615 | enabled: true |
| 1616 | description: Pin IPDeny for country and keep a separate ASN source. |
| 1617 | config: | |
| 1618 | enrichment: |
| 1619 | geoip: |
| 1620 | asn_database: |
| 1621 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 1622 | geo_database: |
| 1623 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 1624 | optional: false |
| 1625 | troubleshooting: |
| 1626 | problems: |
| 1627 | list: |
| 1628 | - name: IPv6 country fields are empty |
| 1629 | description: | |
| 1630 | The current built-in IPDeny downloader source uses the IPv4 |
| 1631 | `all-zones.tar.gz` archive. Use DB-IP, MaxMind, IP2Location, or a custom |
| 1632 | MMDB if IPv6 country enrichment is required. |
| 1633 | - name: ASN fields are empty |
| 1634 | description: | |
| 1635 | IPDeny zone files carry country CIDRs only. Configure a separate ASN |
| 1636 | provider in the downloader. |
| 1637 | - name: Download blocked or rate-limited |
| 1638 | description: | |
| 1639 | Check IPDeny's usage limits and retry later. The plugin continues using the |
| 1640 | last generated MMDB file until the downloader publishes a replacement. |
| 1641 | alerts: [] |
| 1642 | metrics: |
| 1643 | folding: |
| 1644 | title: Metrics |
| 1645 | enabled: false |
| 1646 | description: | |
| 1647 | Enriches IPv4 flow records with country codes; produces no metrics of its own. |
| 1648 | Verify on the Network Flows view via `SRC_COUNTRY` and `DST_COUNTRY`. |
| 1649 | availability: [] |
| 1650 | scopes: [] |
| 1651 | - meta: |
| 1652 | plugin_name: netflow-plugin |
| 1653 | module_name: ipip |
| 1654 | monitored_instance: |
| 1655 | name: IPIP Country Database |
| 1656 | link: https://en.ipip.net/about.html |
| 1657 | categories: |
| 1658 | - flows.enrichment-methods |
| 1659 | icon_filename: network-wired.svg |
| 1660 | keywords: |
| 1661 | - ipip |
| 1662 | - 17mon |
| 1663 | - geoip |
| 1664 | - country |
| 1665 | - ip intelligence |
| 1666 | - flow enrichment |
| 1667 | related_resources: |
| 1668 | integrations: |
| 1669 | list: |
| 1670 | - plugin_name: netflow-plugin |
| 1671 | module_name: dbip |
| 1672 | - plugin_name: netflow-plugin |
| 1673 | module_name: maxmind |
| 1674 | - plugin_name: netflow-plugin |
| 1675 | module_name: iptoasn |
| 1676 | - plugin_name: netflow-plugin |
| 1677 | module_name: custom-mmdb |
| 1678 | info_provided_to_referring_integrations: |
| 1679 | description: "" |
| 1680 | overview: |
| 1681 | data_collection: |
| 1682 | metrics_description: | |
| 1683 | Enrich network flows with country codes from IPIP's country database. The |
| 1684 | supported downloader source fetches IPIP's `country.zip` archive and reads the |
| 1685 | `country.txt` file, where each line maps a prefix to a country token. |
| 1686 | |
| 1687 | IPIP Country is country-only. The current `country.zip` source validated during |
| 1688 | this work contained IPv4 prefixes only. It does not provide ASN, AS |
| 1689 | organization, state, city, latitude, or longitude. Pair IPIP with another ASN |
| 1690 | source when you also need AS fields, and use DB-IP, MaxMind, IP2Location, or a |
| 1691 | custom MMDB when IPv6 country coverage is required. |
| 1692 | |
| 1693 | For the IP-intelligence mechanism (lookup order, MMDB composition, refresh, |
| 1694 | provider chains), see |
| 1695 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 1696 | method_description: | |
| 1697 | The netflow plugin reads MMDB files only. Use Netdata's |
| 1698 | `topology-ip-intel-downloader` to fetch the IPIP country ZIP, convert the |
| 1699 | prefix list to `topology-ip-geo.mmdb`, and write the file into the standard |
| 1700 | auto-detected cache directory. The plugin reloads the generated MMDB |
| 1701 | automatically when the file changes. |
| 1702 | supported_platforms: |
| 1703 | include: |
| 1704 | - Linux |
| 1705 | exclude: [] |
| 1706 | multi_instance: false |
| 1707 | additional_permissions: |
| 1708 | description: "" |
| 1709 | default_behavior: |
| 1710 | auto_detection: |
| 1711 | description: "Not used by the default install. After the downloader writes `topology-ip-geo.mmdb` into the standard cache path, the plugin auto-detects the generated file." |
| 1712 | limits: |
| 1713 | description: "Country-only source. The currently validated `country.zip` payload is IPv4-only and carries no ASN, AS name, state, city, latitude, or longitude." |
| 1714 | performance_impact: |
| 1715 | description: "Lookups are local MMDB reads after conversion. Memory use is mostly the mapped database file and the kernel page cache needed to keep active pages hot." |
| 1716 | setup: |
| 1717 | prerequisites: |
| 1718 | list: |
| 1719 | - title: Downloader with IPIP support |
| 1720 | description: | |
| 1721 | Run the bundled downloader when available. Packaged 32-bit installs do not |
| 1722 | include this binary: |
| 1723 | |
| 1724 | ```bash |
| 1725 | sudo /usr/sbin/topology-ip-intel-downloader \ |
| 1726 | --asn iptoasn:combined \ |
| 1727 | --geo ipip:country |
| 1728 | ``` |
| 1729 | |
| 1730 | `ipip:country` is valid only for `--geo`. The example pairs it with |
| 1731 | IPtoASN so the resulting cache contains both AS and country fields. |
| 1732 | - title: IPIP data terms |
| 1733 | description: | |
| 1734 | Review IPIP's current terms before redistributing the data or generated |
| 1735 | derivative files. The plugin consumes only the generated MMDB and does not |
| 1736 | contact IPIP during flow decoding. |
| 1737 | configuration: |
| 1738 | file: |
| 1739 | name: netflow.yaml |
| 1740 | options: |
| 1741 | description: | |
| 1742 | Once the downloader has produced an IPIP-derived geo MMDB in the cache |
| 1743 | directory, the plugin auto-detects it. To pin paths explicitly, set |
| 1744 | `enrichment.geoip.geo_database` and keep `asn_database` pointed at your chosen |
| 1745 | ASN provider. |
| 1746 | folding: |
| 1747 | title: Config options |
| 1748 | enabled: true |
| 1749 | list: |
| 1750 | - name: enrichment.geoip.asn_database |
| 1751 | description: Optional ASN MMDB paths supplied by another provider. |
| 1752 | default_value: "[]" |
| 1753 | required: false |
| 1754 | - name: enrichment.geoip.geo_database |
| 1755 | description: Path to the IPIP-derived geo MMDB. Empty = auto-detect from the cache directory. |
| 1756 | default_value: "[]" |
| 1757 | required: false |
| 1758 | - name: enrichment.geoip.optional |
| 1759 | description: When true, missing or unreadable MMDBs are warnings, not fatal at startup. |
| 1760 | default_value: "false (true when auto-detected)" |
| 1761 | required: false |
| 1762 | examples: |
| 1763 | folding: |
| 1764 | title: Config |
| 1765 | enabled: true |
| 1766 | list: |
| 1767 | - name: After running the downloader with IPtoASN ASN and IPIP country |
| 1768 | folding: |
| 1769 | enabled: false |
| 1770 | description: Auto-detection picks up both generated cache files. |
| 1771 | config: | |
| 1772 | enrichment: |
| 1773 | geoip: |
| 1774 | asn_database: [] |
| 1775 | geo_database: [] |
| 1776 | optional: true |
| 1777 | - name: Explicit IPIP country path |
| 1778 | folding: |
| 1779 | enabled: true |
| 1780 | description: Pin IPIP for country and keep a separate ASN source. |
| 1781 | config: | |
| 1782 | enrichment: |
| 1783 | geoip: |
| 1784 | asn_database: |
| 1785 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 1786 | geo_database: |
| 1787 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 1788 | optional: false |
| 1789 | troubleshooting: |
| 1790 | problems: |
| 1791 | list: |
| 1792 | - name: IPv6 country fields are empty |
| 1793 | description: | |
| 1794 | The currently validated IPIP `country.zip` source contains IPv4 prefixes. |
| 1795 | Use DB-IP, MaxMind, IP2Location, or a custom MMDB if IPv6 country |
| 1796 | enrichment is required. |
| 1797 | - name: ASN fields are empty |
| 1798 | description: | |
| 1799 | IPIP country data carries country prefixes only. Configure a separate ASN |
| 1800 | provider in the downloader. |
| 1801 | - name: City or coordinates are empty |
| 1802 | description: | |
| 1803 | The supported IPIP source is country-only. Use DB-IP city-lite, MaxMind |
| 1804 | GeoLite2 City, GeoIP2 City, or a custom MMDB for richer geo fields. |
| 1805 | alerts: [] |
| 1806 | metrics: |
| 1807 | folding: |
| 1808 | title: Metrics |
| 1809 | enabled: false |
| 1810 | description: | |
| 1811 | Enriches flow records with country codes; produces no metrics of its own. |
| 1812 | Verify on the Network Flows view via `SRC_COUNTRY` and `DST_COUNTRY`. |
| 1813 | availability: [] |
| 1814 | scopes: [] |
| 1815 | - meta: |
| 1816 | plugin_name: netflow-plugin |
| 1817 | module_name: custom-mmdb |
| 1818 | monitored_instance: |
| 1819 | name: Custom MMDB Database |
| 1820 | link: https://maxmind.github.io/MaxMind-DB/ |
| 1821 | categories: |
| 1822 | - flows.enrichment-methods |
| 1823 | icon_filename: network-wired.svg |
| 1824 | keywords: |
| 1825 | - mmdb |
| 1826 | - custom database |
| 1827 | - bring your own |
| 1828 | - mmdbwriter |
| 1829 | - internal asn |
| 1830 | - internal ipam |
| 1831 | - ip intelligence |
| 1832 | related_resources: |
| 1833 | integrations: |
| 1834 | list: |
| 1835 | - plugin_name: netflow-plugin |
| 1836 | module_name: dbip |
| 1837 | - plugin_name: netflow-plugin |
| 1838 | module_name: maxmind |
| 1839 | - plugin_name: netflow-plugin |
| 1840 | module_name: iptoasn |
| 1841 | - plugin_name: netflow-plugin |
| 1842 | module_name: caida-prefix2as |
| 1843 | - plugin_name: netflow-plugin |
| 1844 | module_name: ip2location |
| 1845 | - plugin_name: netflow-plugin |
| 1846 | module_name: ipdeny |
| 1847 | - plugin_name: netflow-plugin |
| 1848 | module_name: ipip |
| 1849 | info_provided_to_referring_integrations: |
| 1850 | description: "" |
| 1851 | overview: |
| 1852 | data_collection: |
| 1853 | metrics_description: | |
| 1854 | Enrich network flows with ASN, geographic, or internal network context from |
| 1855 | operator-built MMDB databases. Use Custom MMDB Database when you want the netflow |
| 1856 | plugin to read a MaxMind-format binary database (`.mmdb`) that your |
| 1857 | own tooling produces -- typical use cases are joining the upstream DB-IP / |
| 1858 | GeoLite2 data with internal AS labels, or producing a CIDR-tagged custom file |
| 1859 | directly from an internal IPAM. The plugin only requires the file to conform to |
| 1860 | the [MaxMind DB binary format](https://maxmind.github.io/MaxMind-DB/); whatever |
| 1861 | fields it exposes are picked up by the same decoder path used for DB-IP and |
| 1862 | MaxMind. |
| 1863 | |
| 1864 | The decoder reads `autonomous_system_number`, `autonomous_system_organization`, |
| 1865 | and (optionally) a string `asn` field from the ASN database, and `country.iso_code`, |
| 1866 | `subdivisions[].iso_code`, `city.names.en`, `location.latitude`, |
| 1867 | `location.longitude`, plus the Netdata-specific `netdata.ip_class` flag from the |
| 1868 | geo database. Any other vendor-specific keys are ignored. |
| 1869 | |
| 1870 | Note that `*_GEO_CITY`, `*_GEO_LATITUDE`, and `*_GEO_LONGITUDE` are written to |
| 1871 | the raw journal tier only -- the rollup tiers drop these high-cardinality fields. |
| 1872 | State and country survive into all four tiers. |
| 1873 | |
| 1874 | For the full IP-intelligence concept (composition rule, ASN provider chain, |
| 1875 | auto-detect path order, dual-stack handling, hot reload semantics), see |
| 1876 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 1877 | method_description: | |
| 1878 | You build (or download) the MMDB file with your own tooling, place it on the |
| 1879 | agent host, and point `enrichment.geoip.asn_database` and / or |
| 1880 | `enrichment.geoip.geo_database` at it in `netflow.yaml`. The plugin reloads on |
| 1881 | file change automatically (mechanism documented on the IP Intelligence page). |
| 1882 | supported_platforms: |
| 1883 | include: |
| 1884 | - Linux |
| 1885 | exclude: [] |
| 1886 | multi_instance: false |
| 1887 | additional_permissions: |
| 1888 | description: "" |
| 1889 | default_behavior: |
| 1890 | auto_detection: |
| 1891 | description: "Not auto-detected. You must configure paths explicitly." |
| 1892 | limits: |
| 1893 | description: "Coverage, schema quality, and freshness are entirely controlled by the custom MMDB build you provide. Invalid or missing files fail startup unless marked optional." |
| 1894 | performance_impact: |
| 1895 | description: "Lookups are local MMDB reads with no per-flow network call. Memory use depends on the size and number of custom databases you configure." |
| 1896 | setup: |
| 1897 | prerequisites: |
| 1898 | list: |
| 1899 | - title: Build or obtain a standards-compliant MMDB file |
| 1900 | description: | |
| 1901 | The file must conform to the |
| 1902 | [MaxMind DB binary format spec](https://maxmind.github.io/MaxMind-DB/) -- |
| 1903 | a binary search tree over IP prefixes plus a data section of |
| 1904 | type-length-value records. Validate with `mmdblookup` from the |
| 1905 | `libmaxminddb-tools` package before deploying: |
| 1906 | |
| 1907 | ```bash |
| 1908 | mmdblookup --file your-custom.mmdb --ip 8.8.8.8 |
| 1909 | ``` |
| 1910 | |
| 1911 | Common ways to produce a file: |
| 1912 | |
| 1913 | - **Go**: [`github.com/maxmind/mmdbwriter`](https://github.com/maxmind/mmdbwriter) |
| 1914 | -- MaxMind's official Go writer. Lets you start from an upstream MMDB |
| 1915 | (DB-IP, GeoLite2) and overlay extra records, then write a new file. |
| 1916 | Pairs well with the read-side library |
| 1917 | [`github.com/oschwald/maxminddb-golang`](https://github.com/oschwald/maxminddb-golang). |
| 1918 | - **Perl**: [`MaxMind::DB::Writer`](https://metacpan.org/pod/MaxMind::DB::Writer) |
| 1919 | -- the original reference writer. |
| 1920 | - **Python**: community ports such as `mmdb_writer` (PyPI) wrap the same |
| 1921 | format. |
| 1922 | |
| 1923 | Whatever fields you encode are read by the plugin only if they match the |
| 1924 | standard schema names listed in the metrics description above. |
| 1925 | configuration: |
| 1926 | file: |
| 1927 | name: netflow.yaml |
| 1928 | options: |
| 1929 | description: | |
| 1930 | Point `enrichment.geoip.asn_database` and / or `enrichment.geoip.geo_database` |
| 1931 | at your custom MMDB file paths. Set `optional: true` while you iterate so a |
| 1932 | missing or malformed file does not block plugin startup. |
| 1933 | folding: |
| 1934 | title: Config options |
| 1935 | enabled: true |
| 1936 | list: |
| 1937 | - name: enrichment.geoip.asn_database |
| 1938 | description: List of MMDB paths providing AS data. Multiple files compose -- per field, the last database returning a non-empty value wins. |
| 1939 | default_value: "[]" |
| 1940 | required: false |
| 1941 | - name: enrichment.geoip.geo_database |
| 1942 | description: List of MMDB paths providing geographic data. Same composition rule as the ASN list. |
| 1943 | default_value: "[]" |
| 1944 | required: false |
| 1945 | - name: enrichment.geoip.optional |
| 1946 | description: When true, missing or unreadable files at startup are tolerated (the resolver starts with no databases). Recommended while testing a custom file. |
| 1947 | default_value: "false" |
| 1948 | required: false |
| 1949 | examples: |
| 1950 | folding: |
| 1951 | title: Config |
| 1952 | enabled: true |
| 1953 | list: |
| 1954 | - name: Internal MMDB built with mmdbwriter |
| 1955 | folding: |
| 1956 | enabled: false |
| 1957 | description: A custom build that combines public BGP data with internal CIDR labels, written atomically into the plugin's lookup directory by the operator's own pipeline. |
| 1958 | config: | |
| 1959 | enrichment: |
| 1960 | geoip: |
| 1961 | asn_database: |
| 1962 | - /etc/netdata/internal-asn.mmdb |
| 1963 | geo_database: |
| 1964 | - /etc/netdata/internal-geo.mmdb |
| 1965 | optional: false |
| 1966 | - name: Overlay a custom ASN file on top of the bundled DB-IP geo |
| 1967 | folding: |
| 1968 | enabled: true |
| 1969 | description: Keep DB-IP for geographic data and use a custom MMDB only for AS labels (your IPAM-derived prefixes win because they appear last in the list -- the resolver merges per field, last non-empty wins). |
| 1970 | config: | |
| 1971 | enrichment: |
| 1972 | geoip: |
| 1973 | asn_database: |
| 1974 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 1975 | - /etc/netdata/internal-asn.mmdb |
| 1976 | geo_database: |
| 1977 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 1978 | optional: true |
| 1979 | - name: Iterating on a new custom MMDB |
| 1980 | folding: |
| 1981 | enabled: true |
| 1982 | description: While you tune the build pipeline, mark the database optional so an absent or malformed file becomes a startup warning instead of a fatal error. |
| 1983 | config: | |
| 1984 | enrichment: |
| 1985 | geoip: |
| 1986 | asn_database: |
| 1987 | - /etc/netdata/wip-asn.mmdb |
| 1988 | geo_database: [] |
| 1989 | optional: true |
| 1990 | troubleshooting: |
| 1991 | problems: |
| 1992 | list: |
| 1993 | - name: Lookups silently return empty |
| 1994 | description: | |
| 1995 | The custom file's schema is non-standard (for example, the ASN is encoded |
| 1996 | as a string instead of `autonomous_system_number: uint32`, or the country |
| 1997 | is at a non-standard path). The plugin only reads the standard MaxMind |
| 1998 | field names listed in the data-collection description. Validate with |
| 1999 | `mmdblookup --file your-custom.mmdb --ip <known-public-ip>` and confirm the |
| 2000 | standard fields are present at the expected paths. |
| 2001 | - name: Plugin fails to start with optional=false |
| 2002 | description: | |
| 2003 | File missing, unreadable, or not a valid MMDB at the configured path. |
| 2004 | Check permissions (the netdata user must be able to read the file) and |
| 2005 | re-validate with `mmdblookup`. Set `optional: true` while iterating. |
| 2006 | - name: Custom file refresh did not take effect |
| 2007 | description: | |
| 2008 | The plugin reloads custom databases the same way it reloads stock ones |
| 2009 | (signature-driven, in place). If the file was rewritten in place the |
| 2010 | plugin will pick it up; if it was edited byte-by-byte (rare with MMDB |
| 2011 | builders) verify size or mtime actually changed. The reload mechanism |
| 2012 | and its 30-second cadence are documented in |
| 2013 | [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 2014 | Operational practice: have your build pipeline write the new MMDB to a |
| 2015 | temp file in the same directory and `rename(2)` it over the live path so |
| 2016 | the swap is atomic. |
| 2017 | alerts: [] |
| 2018 | metrics: |
| 2019 | folding: |
| 2020 | title: Metrics |
| 2021 | enabled: false |
| 2022 | description: | |
| 2023 | Enriches flow records; produces no metrics of its own. |
| 2024 | availability: [] |
| 2025 | scopes: [] |
| 2026 | - meta: |
| 2027 | plugin_name: netflow-plugin |
| 2028 | module_name: bmp |
| 2029 | monitored_instance: |
| 2030 | name: BMP (BGP Monitoring Protocol) |
| 2031 | link: https://www.rfc-editor.org/rfc/rfc7854 |
| 2032 | categories: |
| 2033 | - flows.enrichment-methods |
| 2034 | icon_filename: network-wired.svg |
| 2035 | keywords: |
| 2036 | - bmp |
| 2037 | - bgp |
| 2038 | - rfc 7854 |
| 2039 | - route monitoring |
| 2040 | - cisco |
| 2041 | - juniper |
| 2042 | - arista |
| 2043 | - frr |
| 2044 | - nokia |
| 2045 | related_resources: |
| 2046 | integrations: |
| 2047 | list: |
| 2048 | - plugin_name: netflow-plugin |
| 2049 | module_name: bioris |
| 2050 | info_provided_to_referring_integrations: |
| 2051 | description: "" |
| 2052 | overview: |
| 2053 | data_collection: |
| 2054 | metrics_description: | |
| 2055 | Enrich network flows with BGP next-hop, AS path, communities, and routing |
| 2056 | context received directly from routers. BMP (BGP Monitoring Protocol, RFC 7854) |
| 2057 | lets a router push its BGP route updates to a passive collector. With this |
| 2058 | integration enabled, Netdata is that collector -- it listens for BMP TCP |
| 2059 | connections from your routers, parses BGP UPDATE messages carried inside |
| 2060 | RouteMonitoring frames, and builds an in-memory routing trie that flow enrichment |
| 2061 | then reads from. |
| 2062 | |
| 2063 | Every flow whose source or destination IP matches a learned prefix gains: |
| 2064 | `SRC_AS` / `DST_AS` (when the `routing` provider in `asn_providers` reaches BGP |
| 2065 | data), `SRC_MASK` / `DST_MASK` (when `routing` reaches BGP data in `net_providers`), |
| 2066 | plus -- for the destination side only -- `NEXT_HOP`, `DST_AS_PATH`, |
| 2067 | `DST_COMMUNITIES`, and `DST_LARGE_COMMUNITIES` (RFC 8092). Source-side AS path and |
| 2068 | communities are not surfaced; BGP path attributes are most meaningful for the |
| 2069 | destination of the traffic. AS *names* (`*_AS_NAME`) come from the GeoIP/ASN MMDB, |
| 2070 | not from BMP -- BMP gives you accurate AS *numbers* and path/communities. |
| 2071 | |
| 2072 | AS path, communities, and large communities are written to the **raw** flow journal |
| 2073 | only -- the rollup tiers do not carry them. `NEXT_HOP` is carried in both raw and |
| 2074 | rollup. |
| 2075 | |
| 2076 | For the cross-cutting Enrichment concept (provider chains, shared trie with |
| 2077 | BioRIS, withdrawal handling, restart convergence), see |
| 2078 | [Enrichment](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 2079 | method_description: | |
| 2080 | The plugin runs a TCP listener on `0.0.0.0:10179`. This port is the Akvorado |
| 2081 | convention -- RFC 7854 does not register a port, and IANA does not assign one |
| 2082 | for BMP. Each connecting router must first send an Initiation message; the plugin |
| 2083 | then processes RouteMonitoring (carrying BGP UPDATE), PeerDownNotification, and |
| 2084 | Termination frames. PeerUp, StatisticsReport, and RouteMirroring frames are |
| 2085 | accepted but not acted on. Only BMP **version 3** is processed. Version 4 frames |
| 2086 | are decoded but ignored; draft v1/v2 frames are decode errors and count toward |
| 2087 | the consecutive decode-error threshold. |
| 2088 | |
| 2089 | NLRI families parsed: IPv4/IPv6 unicast, IPv4/IPv6 MPLS-labelled, VPNv4, VPNv6, |
| 2090 | and EVPN IP-prefix routes. |
| 2091 | |
| 2092 | BMP and BioRIS share a single in-memory routing trie. A full IPv4+IPv6 BGP table |
| 2093 | is roughly 1.2M prefixes per peer; each entry stores `Vec<u32>` AS-path, |
| 2094 | `Vec<u32>` communities, `Vec<(u32,u32,u32)>` large communities, plus a route_key |
| 2095 | string per path. Expect several hundred MB of resident memory per peer with a full |
| 2096 | feed. The trie has no time-based eviction -- routes leave only via explicit BGP |
| 2097 | withdrawal (`MP_UNREACH` or `withdraw_routes`), PeerDown, or session disconnect |
| 2098 | followed by the `keep` interval expiring (default 5 minutes). |
| 2099 | supported_platforms: |
| 2100 | include: |
| 2101 | - Linux |
| 2102 | exclude: [] |
| 2103 | multi_instance: false |
| 2104 | additional_permissions: |
| 2105 | description: "" |
| 2106 | default_behavior: |
| 2107 | auto_detection: |
| 2108 | description: "Disabled by default. Set enrichment.routing_dynamic.bmp.enabled to true and configure your routers to dial in." |
| 2109 | limits: |
| 2110 | description: "Memory and CPU scale with the number of BMP sessions, routing tables, prefixes, AS paths, and communities. Full-table router feeds can consume hundreds of MB per peer." |
| 2111 | performance_impact: |
| 2112 | description: "Disabled until BMP is configured. Once active, BMP updates maintain an in-memory routing trie used for enrichment, so resource use scales with routing-table size and update rate." |
| 2113 | setup: |
| 2114 | prerequisites: |
| 2115 | list: |
| 2116 | - title: BMP-capable routers |
| 2117 | description: | |
| 2118 | Common vendor configuration patterns: |
| 2119 | |
| 2120 | - **Cisco IOS-XR** -- `bmp server N` global block plus `bmp-activate server N` |
| 2121 | under `router bgp ... neighbor`. |
| 2122 | - **Cisco IOS-XE 3.12 / 15.4 or later** -- `bmp server N` nested inside |
| 2123 | `router bgp` with `activate`. |
| 2124 | - **Juniper JunOS** (BMP support since 13.3, RFC 8671 Adj-RIB-In since 18.3R1) |
| 2125 | -- `routing-options bmp ...` with one or more named stations. |
| 2126 | - **Arista EOS** -- `router bgp ... bgp monitoring` with one or more |
| 2127 | `monitoring station` blocks (active connection). |
| 2128 | - **Nokia SR OS** (MD-CLI) -- `/configure bmp` plus per-router `bgp monitor`. |
| 2129 | - **FRR (bgpd)** -- `bmp targets` block under `router bgp`. Note the loadable |
| 2130 | module: `bgpd` must be started with `-M bmp` or every BMP command silently |
| 2131 | fails. |
| 2132 | |
| 2133 | The plugin parses RFC 7854 BMP **version 3** only. Version 4 frames are decoded |
| 2134 | but ignored. Older draft versions (v1, v2) are decode errors and count toward |
| 2135 | the consecutive decode-error threshold. |
| 2136 | - title: TCP reachability between routers and the agent |
| 2137 | description: | |
| 2138 | Routers initiate the connection -- the plugin is a passive listener. Allow |
| 2139 | inbound TCP on the configured port (default 10179) from each BMP-speaking |
| 2140 | router to the agent. The plugin does not retry; it waits for the router to |
| 2141 | reconnect. |
| 2142 | - title: No TLS, no authentication |
| 2143 | description: | |
| 2144 | The listener accepts plain TCP only. Restrict access at the firewall and use |
| 2145 | a dedicated management network -- never expose 10179 to the public internet. |
| 2146 | BMP carries your full routing table; treat it as sensitive. |
| 2147 | configuration: |
| 2148 | file: |
| 2149 | name: netflow.yaml |
| 2150 | options: |
| 2151 | description: | |
| 2152 | All BMP options live under `enrichment.routing_dynamic.bmp` in `netflow.yaml`. |
| 2153 | folding: |
| 2154 | title: Config options |
| 2155 | enabled: true |
| 2156 | list: |
| 2157 | - name: enabled |
| 2158 | description: Master switch. Set to true to start the listener. |
| 2159 | default_value: "false" |
| 2160 | required: false |
| 2161 | - name: listen |
| 2162 | description: TCP bind address (host:port). |
| 2163 | default_value: "0.0.0.0:10179" |
| 2164 | required: false |
| 2165 | - name: keep |
| 2166 | description: Grace window after a BMP disconnect before purging that session's routes from the trie. |
| 2167 | default_value: "5m" |
| 2168 | required: false |
| 2169 | - name: max_consecutive_decode_errors |
| 2170 | description: Close the session after N consecutive decode errors. |
| 2171 | default_value: "8" |
| 2172 | required: false |
| 2173 | - name: receive_buffer |
| 2174 | description: Optional SO_RCVBUF per connection in bytes (0 = OS default). |
| 2175 | default_value: "0" |
| 2176 | required: false |
| 2177 | - name: collect_asns |
| 2178 | description: When false, AS numbers from BMP are forced to 0 before storage. |
| 2179 | default_value: "true" |
| 2180 | required: false |
| 2181 | - name: collect_as_paths |
| 2182 | description: When false, AS paths are dropped before storage. |
| 2183 | default_value: "true" |
| 2184 | required: false |
| 2185 | - name: collect_communities |
| 2186 | description: When false, communities and large communities are dropped before storage. |
| 2187 | default_value: "true" |
| 2188 | required: false |
| 2189 | - name: rds |
| 2190 | description: | |
| 2191 | Whitelist of accepted Route Distinguishers for L3VPN peers. Empty list |
| 2192 | accepts everything. Formats: numeric `0`, `"ASN:idx"`, `"IPv4:idx"`, or |
| 2193 | full text RD. |
| 2194 | default_value: "[]" |
| 2195 | required: false |
| 2196 | examples: |
| 2197 | folding: |
| 2198 | title: Config |
| 2199 | enabled: true |
| 2200 | list: |
| 2201 | - name: Enable BMP listener |
| 2202 | folding: |
| 2203 | enabled: false |
| 2204 | description: Start the listener on the default port. |
| 2205 | config: | |
| 2206 | enrichment: |
| 2207 | routing_dynamic: |
| 2208 | bmp: |
| 2209 | enabled: true |
| 2210 | listen: "0.0.0.0:10179" |
| 2211 | keep: 5m |
| 2212 | - name: Cisco IOS-XR router config |
| 2213 | folding: |
| 2214 | enabled: true |
| 2215 | description: | |
| 2216 | Vendor-side config to send BMP to Netdata. The `bmp server` block is |
| 2217 | global, not under `router bgp`. Each neighbor that should be exported |
| 2218 | needs `bmp-activate server N`. IOS-XR's default route monitoring is |
| 2219 | pre-policy (Adj-RIB-In before inbound policy). Set |
| 2220 | `route-monitoring policy post inbound` if you prefer post-policy. |
| 2221 | config: | |
| 2222 | bmp server 1 |
| 2223 | host 10.0.0.10 port 10179 |
| 2224 | description "Netdata BMP collector" |
| 2225 | update-source Loopback0 |
| 2226 | initial-delay 5 |
| 2227 | stats-reporting-period 60 |
| 2228 | initial-refresh delay 30 spread 2 |
| 2229 | ! |
| 2230 | router bgp 65000 |
| 2231 | neighbor 192.0.2.1 |
| 2232 | bmp-activate server 1 |
| 2233 | - name: Cisco IOS-XE router config |
| 2234 | folding: |
| 2235 | enabled: true |
| 2236 | description: | |
| 2237 | IOS-XE 3.12 / 15.4 or later. The `bmp server N` block lives inside |
| 2238 | `router bgp`, unlike IOS-XR. |
| 2239 | config: | |
| 2240 | router bgp 65000 |
| 2241 | bmp server 1 |
| 2242 | address 10.0.0.10 port-number 10179 |
| 2243 | description "Netdata BMP collector" |
| 2244 | initial-delay 10 |
| 2245 | stats-reporting-period 60 |
| 2246 | update-source GigabitEthernet1 |
| 2247 | activate |
| 2248 | exit-bmp-server-mode |
| 2249 | ! |
| 2250 | neighbor 192.0.2.1 bmp-activate all |
| 2251 | - name: Juniper JunOS router config |
| 2252 | folding: |
| 2253 | enabled: true |
| 2254 | description: | |
| 2255 | Named station form. JunOS supports both pre-policy (RFC 7854) and |
| 2256 | post-policy / Adj-RIB-In (RFC 8671, JunOS 18.3R1+). |
| 2257 | config: | |
| 2258 | set routing-options bmp station netdata station-address 10.0.0.10 |
| 2259 | set routing-options bmp station netdata station-port 10179 |
| 2260 | set routing-options bmp station netdata connection-mode active |
| 2261 | set routing-options bmp station netdata local-address 10.0.0.1 |
| 2262 | set routing-options bmp station netdata statistics-timeout 60 |
| 2263 | set routing-options bmp station netdata route-monitoring pre-policy |
| 2264 | set routing-options bmp station netdata monitor enable |
| 2265 | - name: Arista EOS router config |
| 2266 | folding: |
| 2267 | enabled: true |
| 2268 | description: | |
| 2269 | EOS uses `bgp monitoring` plus one or more `monitoring station` blocks |
| 2270 | inside `router bgp`. Active connection mode is the equivalent of all |
| 2271 | other vendors (router dials Netdata). |
| 2272 | config: | |
| 2273 | router bgp 65000 |
| 2274 | bgp monitoring |
| 2275 | monitoring station netdata |
| 2276 | update-source Management1 |
| 2277 | connection address 10.0.0.10 |
| 2278 | connection mode active port 10179 |
| 2279 | export-policy received routes post-policy |
| 2280 | export-policy bgp rib bestpaths |
| 2281 | - name: FRR (bgpd) router config |
| 2282 | folding: |
| 2283 | enabled: true |
| 2284 | description: | |
| 2285 | Critical -- BMP is a runtime module in FRR. Without `-M bmp` in |
| 2286 | `/etc/frr/daemons` (`bgpd_options`), every BMP command silently fails. |
| 2287 | config: | |
| 2288 | # /etc/frr/daemons: |
| 2289 | # bgpd_options=" -A 127.0.0.1 -M bmp" |
| 2290 | router bgp 65000 |
| 2291 | bmp targets netdata |
| 2292 | bmp connect 10.0.0.10 port 10179 min-retry 5000 max-retry 60000 |
| 2293 | bmp stats interval 60000 |
| 2294 | bmp monitor ipv4 unicast pre-policy |
| 2295 | bmp monitor ipv6 unicast pre-policy |
| 2296 | exit |
| 2297 | - name: Nokia SR OS (MD-CLI) router config |
| 2298 | folding: |
| 2299 | enabled: true |
| 2300 | description: Active connection from one or more BGP routing instances to a named station. |
| 2301 | config: | |
| 2302 | /configure bmp admin-state enable |
| 2303 | /configure bmp station "netdata" admin-state enable |
| 2304 | /configure bmp station "netdata" connection local-address 10.0.0.1 |
| 2305 | /configure bmp station "netdata" connection station-address ip-address 10.0.0.10 |
| 2306 | /configure bmp station "netdata" connection station-address port 10179 |
| 2307 | /configure bmp station "netdata" family ipv4 true |
| 2308 | /configure bmp station "netdata" family ipv6 true |
| 2309 | /configure router "Base" bgp monitor admin-state enable |
| 2310 | /configure router "Base" bgp monitor route-monitoring post-policy true |
| 2311 | /configure router "Base" bgp monitor station "netdata" { } |
| 2312 | - name: Drop AS path and communities |
| 2313 | folding: |
| 2314 | enabled: true |
| 2315 | description: | |
| 2316 | Useful if you only care about the AS number for traffic attribution and |
| 2317 | want to keep the journal small. |
| 2318 | config: | |
| 2319 | enrichment: |
| 2320 | routing_dynamic: |
| 2321 | bmp: |
| 2322 | enabled: true |
| 2323 | collect_asns: true |
| 2324 | collect_as_paths: false |
| 2325 | collect_communities: false |
| 2326 | - name: Restrict to specific Route Distinguishers |
| 2327 | folding: |
| 2328 | enabled: true |
| 2329 | description: | |
| 2330 | For L3VPN peers, only accept routes whose RD matches the whitelist. |
| 2331 | Other peer types are unaffected. |
| 2332 | config: | |
| 2333 | enrichment: |
| 2334 | routing_dynamic: |
| 2335 | bmp: |
| 2336 | enabled: true |
| 2337 | rds: |
| 2338 | - "65000:100" |
| 2339 | - "65000:200" |
| 2340 | troubleshooting: |
| 2341 | problems: |
| 2342 | list: |
| 2343 | - name: Listener not receiving BMP sessions |
| 2344 | description: | |
| 2345 | The plugin is a passive listener -- it never dials. Check the router side: |
| 2346 | `show bmp` (Cisco), `show bmp connections` / `show bgp monitoring station` |
| 2347 | (Juniper, Arista), `show bmp targets` (FRR). Confirm the firewall allows |
| 2348 | inbound TCP on port 10179. For FRR specifically, verify `bgpd` was started |
| 2349 | with `-M bmp` -- without it, every BMP command is silently accepted but |
| 2350 | no connection is ever opened. |
| 2351 | - name: Convergence takes minutes after restart |
| 2352 | description: | |
| 2353 | The trie is not persisted. After a plugin restart, routers re-send |
| 2354 | Initiation followed by their Adj-RIB-In as RouteMonitoring updates. FRR |
| 2355 | re-emits everything in seconds. Cisco IOS-XR's `initial-refresh` is |
| 2356 | configurably spread (defaults to a per-peer delay) so a full re-feed can |
| 2357 | take minutes. Juniper varies between seconds and minutes depending on |
| 2358 | station options. Schedule restarts off-peak when BGP attribution matters. |
| 2359 | - name: Memory growth without bound |
| 2360 | description: | |
| 2361 | A full BGP feed adds ~1.2M prefixes per peer permanently -- there is no |
| 2362 | time-based eviction in the trie. Routes are removed only by explicit BGP |
| 2363 | withdrawal, PeerDown, or session disconnect followed by the `keep` |
| 2364 | interval. Plan capacity before connecting full-table peers. |
| 2365 | - name: AS path inconsistent with the exporter's view |
| 2366 | description: | |
| 2367 | The exporter and the BMP-feeding router are usually different boxes with |
| 2368 | different routing tables. Different vantage points see different AS paths; |
| 2369 | this is normal. The `routing` provider in the `asn_providers` chain decides |
| 2370 | which source wins (default order is `flow, routing, geoip` -- exporter first). |
| 2371 | - name: Empty BGP enrichment after enabling |
| 2372 | description: | |
| 2373 | Confirm the router actually established the BMP session (vendor-side `show` |
| 2374 | command above). Confirm `enrichment.asn_providers` includes `routing` (or |
| 2375 | the `bmp` alias) -- if `routing` is removed from both `asn_providers` and |
| 2376 | `net_providers`, the trie is built but never read. |
| 2377 | - name: Validate BGP enrichment after enabling |
| 2378 | description: | |
| 2379 | BGP-derived enrichment depends on router export policy, peer state, and |
| 2380 | route visibility. Validate against your specific router firmware before |
| 2381 | depending on this for capacity or security decisions. |
| 2382 | alerts: [] |
| 2383 | metrics: |
| 2384 | folding: |
| 2385 | title: Metrics |
| 2386 | enabled: false |
| 2387 | description: | |
| 2388 | Enriches flow records with `SRC_AS` / `DST_AS` (when the `routing` provider in |
| 2389 | `asn_providers` reaches BGP), `SRC_MASK` / `DST_MASK` (via `net_providers`), |
| 2390 | `NEXT_HOP`, `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES`. Source-side |
| 2391 | AS path and communities are not surfaced. AS *names* come from the GeoIP/ASN MMDB, |
| 2392 | not from BMP. AS path, communities, and large communities are RAW-tier-only -- |
| 2393 | rollups carry only `NEXT_HOP`. Verify in the Network Flows view by querying the |
| 2394 | AS-path and communities columns. |
| 2395 | availability: [] |
| 2396 | scopes: [] |
| 2397 | - meta: |
| 2398 | plugin_name: netflow-plugin |
| 2399 | module_name: bioris |
| 2400 | monitored_instance: |
| 2401 | name: bio-rd / RIPE RIS |
| 2402 | link: https://github.com/bio-routing/bio-rd |
| 2403 | categories: |
| 2404 | - flows.enrichment-methods |
| 2405 | icon_filename: network-wired.svg |
| 2406 | keywords: |
| 2407 | - bioris |
| 2408 | - bio-rd |
| 2409 | - ripe ris |
| 2410 | - bgp |
| 2411 | - grpc |
| 2412 | - route information service |
| 2413 | - as path |
| 2414 | - bgp communities |
| 2415 | related_resources: |
| 2416 | integrations: |
| 2417 | list: |
| 2418 | - plugin_name: netflow-plugin |
| 2419 | module_name: bmp |
| 2420 | info_provided_to_referring_integrations: |
| 2421 | description: "" |
| 2422 | overview: |
| 2423 | data_collection: |
| 2424 | metrics_description: | |
| 2425 | Enrich network flows with BGP routing context received from a bio-rd-compatible |
| 2426 | `RoutingInformationService` gRPC endpoint. [bio-rd](https://github.com/bio-routing/bio-rd) |
| 2427 | is a Go-based BGP/BMP daemon that you run yourself. Its `ris` daemon receives BMP |
| 2428 | sessions from routers or collectors and exposes the resulting RIB through gRPC. |
| 2429 | Netdata is a **client** of that gRPC interface only. |
| 2430 | |
| 2431 | Netdata does **not** connect directly to RIPE NCC RIS Live, RIPEstat, RIS MRT dumps, |
| 2432 | or RIPE route collector sessions. RIPE RIS public access is WebSocket JSON, HTTP API, |
| 2433 | and MRT dump based. To use a RIPE-derived external view, run a bridge or service that |
| 2434 | imports that data into a bio-rd-compatible `RoutingInformationService`, then point |
| 2435 | Netdata at that service. |
| 2436 | |
| 2437 | Pick this when bio-rd is already part of your routing toolbox, or when you want Netdata |
| 2438 | to consume a routing view that is already exposed through the bio-rd RIS gRPC API. If |
| 2439 | your routers can speak BMP straight to Netdata, the `bmp` integration is simpler. |
| 2440 | |
| 2441 | BioRIS populates the same flow-record fields as BMP -- both feed a single shared |
| 2442 | in-memory routing trie. See the BMP integration card for the side-by-side comparison |
| 2443 | and combined-deployment notes. |
| 2444 | method_description: | |
| 2445 | The plugin connects as a gRPC client to one or more user-provided bio-rd `ris` |
| 2446 | endpoints (`grpc_addr`) and runs three RPCs against each one: |
| 2447 | |
| 2448 | 1. `GetRouters` -- discover which routers/VRFs the bio-rd instance is exposing. |
| 2449 | 2. `DumpRIB` -- pull a baseline RIB for each (router, AFI/SAFI) tuple. This is the |
| 2450 | expensive call -- full IPv4+IPv6 feeds run to millions of prefixes. |
| 2451 | 3. `ObserveRIB` -- subscribe to incremental updates so the trie tracks live changes. |
| 2452 | |
| 2453 | Per refresh cycle (default 30 minutes), the plugin re-runs `GetRouters` + |
| 2454 | `DumpRIB`, then keeps `ObserveRIB` streams open between cycles. Routers that |
| 2455 | disappear between refreshes have their routes purged. |
| 2456 | |
| 2457 | Multiple `ris_instances` are **additive**, not failover -- routes from every |
| 2458 | configured endpoint merge into the same trie. The trie is also shared with the |
| 2459 | `bmp` integration, so if both are enabled their routes coexist and lookups pick |
| 2460 | the best match across both sources. |
| 2461 | |
| 2462 | Connection is plain gRPC over HTTP/2 by default; set `grpc_secure: true` to use |
| 2463 | TLS with the system CA bundle. There is no client-cert / mTLS and no application |
| 2464 | auth -- restrict access at the network layer. |
| 2465 | |
| 2466 | **Fields populated** (same set as BMP): |
| 2467 | |
| 2468 | | Field | Side | Source | |
| 2469 | |---|---|---| |
| 2470 | | `SRC_AS` / `DST_AS` | both | When the `routing` provider in the `asn_providers` chain matches | |
| 2471 | | `SRC_MASK` / `DST_MASK` | both | Prefix length of the longest-match BGP route | |
| 2472 | | `NEXT_HOP` | dest only | BGP next-hop attribute from the destination route | |
| 2473 | | `DST_AS_PATH` | dest only | Full AS path, CSV of ASNs | |
| 2474 | | `DST_COMMUNITIES` | dest only | Standard BGP communities (CSV of u32) | |
| 2475 | | `DST_LARGE_COMMUNITIES` | dest only | RFC 8092 large communities | |
| 2476 | |
| 2477 | Source-side AS path and communities are **not** surfaced; BGP path attributes are |
| 2478 | most meaningful for the destination of the traffic. AS *names* (`SRC_AS_NAME`, |
| 2479 | `DST_AS_NAME`) come from the GeoIP/ASN MMDB, not from BGP -- BioRIS gives you |
| 2480 | accurate AS *numbers* and path/community attributes; the names come from the ASN |
| 2481 | database integration. |
| 2482 | |
| 2483 | **Storage tier:** `DST_AS_PATH`, `DST_COMMUNITIES`, and `DST_LARGE_COMMUNITIES` |
| 2484 | are written only into the raw journal tier. The 1-minute / 5-minute / 1-hour |
| 2485 | rollup tiers do not carry them. Queries that need AS path or community data |
| 2486 | must run against a window that the raw retention still covers. |
| 2487 | supported_platforms: |
| 2488 | include: |
| 2489 | - Linux |
| 2490 | exclude: [] |
| 2491 | multi_instance: false |
| 2492 | additional_permissions: |
| 2493 | description: "" |
| 2494 | default_behavior: |
| 2495 | auto_detection: |
| 2496 | description: "Disabled by default. Set enrichment.routing_dynamic.bioris.enabled to true and provide at least one ris_instances entry." |
| 2497 | limits: |
| 2498 | description: "Memory scales with the number of RIS instances, peers, routing tables, prefixes, AS paths, and communities. Full-table feeds can consume hundreds of MB per peer." |
| 2499 | performance_impact: |
| 2500 | description: | |
| 2501 | Disabled until BioRIS is configured. Once active, Netdata maintains an |
| 2502 | in-memory routing trie for the received RIB and updates. Plan capacity from |
| 2503 | the number of peers and tables you import, and watch the agent's RSS. |
| 2504 | setup: |
| 2505 | prerequisites: |
| 2506 | list: |
| 2507 | - title: A running bio-rd 'ris' daemon |
| 2508 | description: | |
| 2509 | bio-rd is a separate project. The plugin only consumes its gRPC interface; it |
| 2510 | does not bundle bio-rd. You install and operate it yourself: |
| 2511 | |
| 2512 | ```bash |
| 2513 | # Install Go (>=1.20), then: |
| 2514 | git clone https://github.com/bio-routing/bio-rd.git |
| 2515 | cd bio-rd/cmd/ris |
| 2516 | go build -o /usr/local/bin/ris . |
| 2517 | ``` |
| 2518 | |
| 2519 | Configure `ris` with one or more BMP sources using bio-rd's `bmp_addr` / |
| 2520 | `bmp_servers` configuration. Refer to the bio-rd documentation for the BMP setup |
| 2521 | -- this is bio-rd's configuration, not Netdata's. |
| 2522 | |
| 2523 | Do not set Netdata's `grpc_addr` to a RIPE RIS Live URL, RIPEstat API URL, MRT dump |
| 2524 | URL, or route collector session address. Those endpoints are not the |
| 2525 | `RoutingInformationService` gRPC API that Netdata consumes. |
| 2526 | |
| 2527 | Run the daemon with a gRPC port: |
| 2528 | `/usr/local/bin/ris --grpc_port 50051 --config.file /etc/bio-rd.yml` |
| 2529 | - title: Network reachability + no auth |
| 2530 | description: | |
| 2531 | The gRPC connection is plain HTTP/2 by default, or TLS with the system CA |
| 2532 | bundle when `grpc_secure: true`. There is no client-cert / mTLS and no application |
| 2533 | authentication -- restrict access at the firewall, or run bio-rd on the same |
| 2534 | host as the agent and bind it to localhost. |
| 2535 | configuration: |
| 2536 | file: |
| 2537 | name: netflow.yaml |
| 2538 | options: |
| 2539 | description: | |
| 2540 | BioRIS options live under `enrichment.routing_dynamic.bioris` in |
| 2541 | `netflow.yaml`. |
| 2542 | folding: |
| 2543 | title: Config options |
| 2544 | enabled: true |
| 2545 | list: |
| 2546 | - name: enabled |
| 2547 | description: Master switch. |
| 2548 | default_value: "false" |
| 2549 | required: false |
| 2550 | - name: timeout |
| 2551 | description: Connect + per-RPC timeout for `GetRouters` and `ObserveRIB` setup. Default is aggressive; raise to 2-5s when reaching a remote bio-rd over the public internet. |
| 2552 | default_value: "200ms" |
| 2553 | required: false |
| 2554 | - name: refresh |
| 2555 | description: Cadence at which the plugin re-runs `GetRouters` + `DumpRIB` for every router. Floored to 10s. |
| 2556 | default_value: "30m" |
| 2557 | required: false |
| 2558 | - name: refresh_timeout |
| 2559 | description: Per-DumpRIB request timeout and per-message stream timeout for the baseline RIB pull. |
| 2560 | default_value: "10s" |
| 2561 | required: false |
| 2562 | - name: ris_instances |
| 2563 | description: | |
| 2564 | List of bio-rd endpoints. Each entry: `grpc_addr` (`host:port` or full |
| 2565 | `http(s)://` URI), `grpc_secure` (TLS with system CAs when true), `vrf` |
| 2566 | (string VRF name to filter on), `vrf_id` (numeric VRF ID, alternative to |
| 2567 | `vrf`). Multiple instances are additive (not failover) -- routes from every |
| 2568 | instance merge into the same trie. |
| 2569 | default_value: "[]" |
| 2570 | required: true |
| 2571 | examples: |
| 2572 | folding: |
| 2573 | title: Config |
| 2574 | enabled: true |
| 2575 | list: |
| 2576 | - name: Local bio-rd |
| 2577 | folding: |
| 2578 | enabled: false |
| 2579 | description: bio-rd running on the same host, plain gRPC. |
| 2580 | config: | |
| 2581 | enrichment: |
| 2582 | routing_dynamic: |
| 2583 | bioris: |
| 2584 | enabled: true |
| 2585 | timeout: 2s |
| 2586 | refresh: 30m |
| 2587 | refresh_timeout: 30s |
| 2588 | ris_instances: |
| 2589 | - grpc_addr: "127.0.0.1:50051" |
| 2590 | grpc_secure: false |
| 2591 | - name: Remote bio-rd over TLS |
| 2592 | folding: |
| 2593 | enabled: true |
| 2594 | description: Across a network, system CA bundle. No client cert / mTLS. |
| 2595 | config: | |
| 2596 | enrichment: |
| 2597 | routing_dynamic: |
| 2598 | bioris: |
| 2599 | enabled: true |
| 2600 | timeout: 5s |
| 2601 | ris_instances: |
| 2602 | - grpc_addr: "ris.example.internal:50051" |
| 2603 | grpc_secure: true |
| 2604 | vrf: "global" |
| 2605 | - name: Prefer BGP over the exporter's AS view |
| 2606 | folding: |
| 2607 | enabled: true |
| 2608 | description: | |
| 2609 | With the default provider chain, an exporter-supplied AS number wins over |
| 2610 | BGP. If your bio-rd feed is more accurate than what the exporter is |
| 2611 | stamping into flow records, reorder the chains so `routing` is consulted |
| 2612 | first. |
| 2613 | config: | |
| 2614 | enrichment: |
| 2615 | asn_providers: [routing, flow, geoip] |
| 2616 | net_providers: [routing, flow] |
| 2617 | routing_dynamic: |
| 2618 | bioris: |
| 2619 | enabled: true |
| 2620 | ris_instances: |
| 2621 | - grpc_addr: "127.0.0.1:50051" |
| 2622 | - name: Combined with BMP into a single trie |
| 2623 | folding: |
| 2624 | enabled: true |
| 2625 | description: | |
| 2626 | Run BMP from internal routers and BioRIS from a separate bio-rd-compatible RIS |
| 2627 | service, for example one populated from an external routing view. Both populate |
| 2628 | the same shared trie -- lookups pick the best-matching route across both sources |
| 2629 | at query time. |
| 2630 | config: | |
| 2631 | enrichment: |
| 2632 | routing_dynamic: |
| 2633 | bmp: |
| 2634 | enabled: true |
| 2635 | listen: "0.0.0.0:10179" |
| 2636 | bioris: |
| 2637 | enabled: true |
| 2638 | ris_instances: |
| 2639 | - grpc_addr: "127.0.0.1:50051" |
| 2640 | troubleshooting: |
| 2641 | problems: |
| 2642 | list: |
| 2643 | - name: Default 200ms timeout too aggressive |
| 2644 | description: | |
| 2645 | `timeout` covers the gRPC connect plus the `GetRouters` and `ObserveRIB` |
| 2646 | setup RPCs. Over the public internet to a remote bio-rd, 200ms is often not |
| 2647 | enough. If you see "deadline exceeded" errors in the journal, raise to 2-5s. |
| 2648 | `refresh_timeout` is separate and bounds the long-running `DumpRIB` and per- |
| 2649 | message stream reads (default 10s). |
| 2650 | - name: Initial dump takes minutes for full feeds |
| 2651 | description: | |
| 2652 | A full IPv4+IPv6 RIB from a full-table bio-rd source can contain millions of |
| 2653 | prefixes. The first `DumpRIB` after enabling (or after a plugin restart) takes |
| 2654 | time -- BGP attribution is incomplete until it finishes. Subsequent |
| 2655 | `ObserveRIB` streams are incremental. |
| 2656 | - name: Plugin restart wipes the trie |
| 2657 | description: | |
| 2658 | The trie is in-memory only -- restarting the netflow plugin loses every |
| 2659 | learned BGP route. Convergence over BioRIS depends on the upstream feed; a |
| 2660 | full DumpRIB from a full-table bio-rd source can take minutes. Schedule |
| 2661 | restarts off-peak if BGP attribution matters for your workflow. |
| 2662 | - name: Pointing grpc_addr at RIPE RIS does not work |
| 2663 | description: | |
| 2664 | `grpc_addr` must point to a bio-rd-compatible `RoutingInformationService` |
| 2665 | endpoint. RIPE RIS Live, RIPEstat, RIS MRT dumps, and route collector BGP |
| 2666 | sessions use different protocols, so they cannot be used directly as |
| 2667 | `ris_instances[].grpc_addr`. Put a converter or bio-rd-compatible service in |
| 2668 | front of RIPE-derived data if you need that external view. |
| 2669 | - name: Memory growth without bound |
| 2670 | description: | |
| 2671 | The trie has no time-based eviction. Routes are removed only when the |
| 2672 | upstream BGP source withdraws them, when a router disappears from |
| 2673 | `GetRouters` between refreshes, or on plugin shutdown. A full feed is |
| 2674 | several hundred MB of RSS per peer, permanently. |
| 2675 | - name: AS path / communities missing on older queries |
| 2676 | description: | |
| 2677 | `DST_AS_PATH`, `DST_COMMUNITIES`, and `DST_LARGE_COMMUNITIES` only exist in |
| 2678 | the raw journal tier. The 1m / 5m / 1h rollup tiers do not carry them. Queries that span beyond the raw |
| 2679 | retention horizon will not return BGP path data. |
| 2680 | - name: AS path inconsistent with the exporter's view |
| 2681 | description: | |
| 2682 | Different vantage points see different BGP paths. If your flow exporter and |
| 2683 | the BGP source bio-rd is peering with are different boxes with different |
| 2684 | routing tables, expect divergence. This is normal in BGP, not a bug. |
| 2685 | - name: Validate BioRIS enrichment after enabling |
| 2686 | description: | |
| 2687 | BioRIS-derived enrichment depends on the bio-rd version, upstream BGP source, |
| 2688 | route visibility, and refresh cadence. Validate against your bio-rd setup |
| 2689 | before relying on this for capacity or security decisions. |
| 2690 | alerts: [] |
| 2691 | metrics: |
| 2692 | folding: |
| 2693 | title: Metrics |
| 2694 | enabled: false |
| 2695 | description: | |
| 2696 | Enriches flow records with the same fields as BMP: `SRC_AS`, `DST_AS`, `SRC_MASK`, |
| 2697 | `DST_MASK`, `NEXT_HOP`, `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES`. |
| 2698 | Source-side AS path and communities are not surfaced. AS names come from the |
| 2699 | GeoIP/ASN MMDB, not from BGP. AS path and communities are written only to the raw |
| 2700 | journal tier; rollup tiers do not carry them. When BMP and BioRIS are both |
| 2701 | enabled, they contribute to a single shared in-memory trie and lookups pick the |
| 2702 | best-matching route across both sources. |
| 2703 | availability: [] |
| 2704 | scopes: [] |
| 2705 | - meta: |
| 2706 | plugin_name: netflow-plugin |
| 2707 | module_name: aws-ip-ranges |
| 2708 | monitored_instance: |
| 2709 | name: AWS IP Ranges |
| 2710 | link: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html |
| 2711 | categories: |
| 2712 | - flows.enrichment-methods |
| 2713 | icon_filename: network-wired.svg |
| 2714 | keywords: |
| 2715 | - aws |
| 2716 | - amazon |
| 2717 | - cloud |
| 2718 | - ip ranges |
| 2719 | - vpc |
| 2720 | - ec2 |
| 2721 | - prefix list |
| 2722 | - service tags |
| 2723 | related_resources: |
| 2724 | integrations: |
| 2725 | list: |
| 2726 | - plugin_name: netflow-plugin |
| 2727 | module_name: gcp-ip-ranges |
| 2728 | - plugin_name: netflow-plugin |
| 2729 | module_name: azure-ip-ranges |
| 2730 | - plugin_name: netflow-plugin |
| 2731 | module_name: netbox |
| 2732 | - plugin_name: netflow-plugin |
| 2733 | module_name: generic-ipam |
| 2734 | info_provided_to_referring_integrations: |
| 2735 | description: "" |
| 2736 | overview: |
| 2737 | data_collection: |
| 2738 | metrics_description: | |
| 2739 | Annotate network flows with AWS service and region labels from AWS public IP |
| 2740 | ranges. AWS publishes its current public IP allocations as a single JSON document at |
| 2741 | [`https://ip-ranges.amazonaws.com/ip-ranges.json`](https://ip-ranges.amazonaws.com/ip-ranges.json). |
| 2742 | The file lists every prefix AWS uses for a published service, tagged with the AWS |
| 2743 | region and the AWS service that owns the prefix (`AMAZON`, `EC2`, `S3`, |
| 2744 | `CLOUDFRONT`, `API_GATEWAY`, ...). This integration fetches the file periodically, |
| 2745 | transforms it via a jq expression, and labels matching flow records with provider, |
| 2746 | region, and service tags. |
| 2747 | |
| 2748 | Once configured, traffic to / from AWS shows up in dashboards as `tenant=amazon` |
| 2749 | with per-region (`region=us-east-1`, `region=eu-central-1`, ...) and per-service |
| 2750 | (`role=s3`, `role=cloudfront`, ...) breakdown. This is **complementary to GeoIP |
| 2751 | and ASN enrichment**: instead of a flow being labeled simply `AS16509 Amazon`, |
| 2752 | it becomes labeled with the specific AWS service and region carrying the |
| 2753 | traffic -- without parsing AS-name strings. |
| 2754 | |
| 2755 | The file is public; no AWS credentials are needed. |
| 2756 | |
| 2757 | Schema of `ip-ranges.json`: |
| 2758 | |
| 2759 | - top-level: `syncToken`, `createDate`, `prefixes`, `ipv6_prefixes` |
| 2760 | - each entry in `prefixes[]`: `ip_prefix`, `region`, `service`, `network_border_group` |
| 2761 | - each entry in `ipv6_prefixes[]`: `ipv6_prefix`, `region`, `service`, `network_border_group` |
| 2762 | |
| 2763 | For the cross-cutting concepts (how multiple network sources merge, the lookup |
| 2764 | priority vs static `networks` config, TLS verification rules, the jq engine and |
| 2765 | its expected output schema, fetch loop and retry/backoff), see |
| 2766 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 2767 | method_description: | |
| 2768 | The plugin issues a periodic GET to `https://ip-ranges.amazonaws.com/ip-ranges.json`, |
| 2769 | parses the JSON body, runs the configured jq transform via the |
| 2770 | [jaq](https://github.com/01mf02/jaq) library, and merges the resulting |
| 2771 | prefix-labeled rows into the network-attributes trie shared by all enrichment |
| 2772 | sources. |
| 2773 | |
| 2774 | AWS does not publish a fixed refresh cadence; the file is updated whenever the |
| 2775 | AWS IP space changes (typically several times per day) and the `syncToken` / |
| 2776 | `createDate` fields advance on every change. AWS recommends polling no faster |
| 2777 | than the file actually changes; daily is enough for most flow-attribution use |
| 2778 | cases. The plugin floors the configured `interval` at 60s. |
| 2779 | supported_platforms: |
| 2780 | include: |
| 2781 | - Linux |
| 2782 | exclude: [] |
| 2783 | multi_instance: true |
| 2784 | additional_permissions: |
| 2785 | description: "" |
| 2786 | default_behavior: |
| 2787 | auto_detection: |
| 2788 | description: "Disabled by default. Add an entry under `enrichment.network_sources` to enable." |
| 2789 | limits: |
| 2790 | description: "One full AWS prefix document is fetched per refresh. Resource use scales with the number of AWS prefixes selected by your transform and the refresh interval." |
| 2791 | performance_impact: |
| 2792 | description: "One HTTPS request per refresh interval plus a jq transform over the AWS prefix document. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records." |
| 2793 | setup: |
| 2794 | prerequisites: |
| 2795 | list: |
| 2796 | - title: Outbound HTTPS to AWS |
| 2797 | description: | |
| 2798 | The agent host must be able to reach `https://ip-ranges.amazonaws.com/ip-ranges.json`. |
| 2799 | No AWS credentials are needed -- the file is public. |
| 2800 | configuration: |
| 2801 | file: |
| 2802 | name: netflow.yaml |
| 2803 | options: |
| 2804 | description: | |
| 2805 | Add a named entry under `enrichment.network_sources`. The map key (e.g. `aws`) |
| 2806 | is the source name; it is used in plugin logs but is **not** automatically |
| 2807 | attached to flow records -- the record labels come entirely from your `transform` |
| 2808 | output (`tenant`, `region`, `role`, `name`, `site`, `city`, `country`, `state`, |
| 2809 | `asn`, `asn_name`). |
| 2810 | |
| 2811 | The expected `transform` output is a stream of objects, each carrying a `prefix` |
| 2812 | (CIDR string) plus any subset of the optional attribute fields. See the |
| 2813 | cross-cutting [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment) |
| 2814 | page for the full output schema. |
| 2815 | folding: |
| 2816 | title: Config options |
| 2817 | enabled: true |
| 2818 | list: |
| 2819 | - name: url |
| 2820 | description: AWS publishes the master file at this URL. Use it as-is unless you mirror it locally for air-gapped environments. |
| 2821 | default_value: "https://ip-ranges.amazonaws.com/ip-ranges.json" |
| 2822 | required: true |
| 2823 | - name: interval |
| 2824 | description: | |
| 2825 | How often to fetch the file. AWS updates the document whenever its IP space |
| 2826 | changes (often several times per day, sometimes more); daily is enough for |
| 2827 | most uses. The configured value is floored at 60s. |
| 2828 | default_value: "60s" |
| 2829 | required: false |
| 2830 | - name: timeout |
| 2831 | description: Per-request timeout for the HTTPS GET. |
| 2832 | default_value: "60s" |
| 2833 | required: false |
| 2834 | - name: method |
| 2835 | description: HTTP method. AWS serves the file via GET; leave at the default. |
| 2836 | default_value: "GET" |
| 2837 | required: false |
| 2838 | - name: headers |
| 2839 | description: | |
| 2840 | Extra HTTP headers added to the request. Not required for the public AWS |
| 2841 | URL; only needed if you front the file behind your own authenticated mirror. |
| 2842 | default_value: "{}" |
| 2843 | required: false |
| 2844 | - name: transform |
| 2845 | description: | |
| 2846 | jq expression (compiled by jaq) that converts the AWS JSON into a stream |
| 2847 | of `{prefix, ...}` objects. The default `.` does **not** match the AWS |
| 2848 | schema -- you must supply a real transform (see examples below) or fetches fail |
| 2849 | because output rows cannot be mapped to the required `{prefix, ...}` schema. |
| 2850 | default_value: "." |
| 2851 | required: true |
| 2852 | examples: |
| 2853 | folding: |
| 2854 | title: Config |
| 2855 | enabled: true |
| 2856 | list: |
| 2857 | - name: Tag all AWS prefixes by region and service (IPv4 + IPv6) |
| 2858 | folding: |
| 2859 | enabled: false |
| 2860 | description: | |
| 2861 | Recommended starting point. Sets `tenant=amazon`, `region=<aws-region>`, |
| 2862 | `role=<service-name-lowercased>`. Covers both `prefixes[]` and |
| 2863 | `ipv6_prefixes[]` in one stream so v4 and v6 traffic are tagged |
| 2864 | consistently. |
| 2865 | config: | |
| 2866 | enrichment: |
| 2867 | network_sources: |
| 2868 | aws: |
| 2869 | url: "https://ip-ranges.amazonaws.com/ip-ranges.json" |
| 2870 | interval: 24h |
| 2871 | timeout: 60s |
| 2872 | transform: | |
| 2873 | (.prefixes + .ipv6_prefixes)[] | { |
| 2874 | prefix: (.ip_prefix // .ipv6_prefix), |
| 2875 | tenant: "amazon", |
| 2876 | region: .region, |
| 2877 | role: (.service | ascii_downcase) |
| 2878 | } |
| 2879 | - name: AWS S3 only |
| 2880 | folding: |
| 2881 | enabled: true |
| 2882 | description: | |
| 2883 | Filter to a single AWS service for narrower tagging. Useful when you only |
| 2884 | care about identifying S3 traffic (e.g. egress-cost attribution). |
| 2885 | config: | |
| 2886 | enrichment: |
| 2887 | network_sources: |
| 2888 | aws-s3: |
| 2889 | url: "https://ip-ranges.amazonaws.com/ip-ranges.json" |
| 2890 | interval: 24h |
| 2891 | transform: | |
| 2892 | (.prefixes + .ipv6_prefixes)[] |
| 2893 | | select(.service == "S3") |
| 2894 | | { |
| 2895 | prefix: (.ip_prefix // .ipv6_prefix), |
| 2896 | tenant: "amazon", |
| 2897 | role: "s3", |
| 2898 | region: .region |
| 2899 | } |
| 2900 | - name: Use network_border_group as the site label |
| 2901 | folding: |
| 2902 | enabled: true |
| 2903 | description: | |
| 2904 | AWS exposes a `network_border_group` field that distinguishes Local Zones |
| 2905 | / Wavelength Zones from the parent Region. Map it to the `site` attribute |
| 2906 | if you want that distinction visible in flow dashboards. |
| 2907 | config: | |
| 2908 | enrichment: |
| 2909 | network_sources: |
| 2910 | aws: |
| 2911 | url: "https://ip-ranges.amazonaws.com/ip-ranges.json" |
| 2912 | interval: 24h |
| 2913 | transform: | |
| 2914 | (.prefixes + .ipv6_prefixes)[] | { |
| 2915 | prefix: (.ip_prefix // .ipv6_prefix), |
| 2916 | tenant: "amazon", |
| 2917 | region: .region, |
| 2918 | site: .network_border_group, |
| 2919 | role: (.service | ascii_downcase) |
| 2920 | } |
| 2921 | troubleshooting: |
| 2922 | problems: |
| 2923 | list: |
| 2924 | - name: 'Default `transform: "."` fails because output rows are missing `prefix`' |
| 2925 | description: | |
| 2926 | The default `.` returns the raw JSON object, not the per-prefix stream the |
| 2927 | plugin expects. You **must** supply a transform that yields one object per |
| 2928 | prefix (with at least a `prefix` field). Use the first example above as your |
| 2929 | starting template. |
| 2930 | - name: AWS service name precedence inside one prefix |
| 2931 | description: | |
| 2932 | AWS sometimes lists the same `ip_prefix` under multiple `service` entries |
| 2933 | (e.g. once under `AMAZON` and again under `EC2`). The plugin merges all |
| 2934 | records into a single prefix trie, and the **last write wins** for a given |
| 2935 | prefix length. If you want a specific service to take precedence, filter the |
| 2936 | transform with `select(.service != "AMAZON")` so the broader catch-all |
| 2937 | entries are dropped. |
| 2938 | - name: Static `networks:` config silently overrides AWS labels |
| 2939 | description: | |
| 2940 | When a prefix is defined in both this source and `enrichment.networks`, the |
| 2941 | static config wins on every non-empty field. This is by design (operator |
| 2942 | intent overrides imported data) but can surprise operators who expect the |
| 2943 | remote feed to be authoritative. See the cross-cutting Network Identity page |
| 2944 | for the full lookup-priority rules. |
| 2945 | - name: Empty result from the transform is treated as failure |
| 2946 | description: | |
| 2947 | If the jq filter happens to produce nothing (bad selector, wrong field name, |
| 2948 | upstream JSON shape change), the source backs off as if it errored. Check |
| 2949 | the journal for `network-sources` warnings: |
| 2950 | `journalctl --namespace netdata | grep network-sources`. |
| 2951 | - name: TLS verification cannot be disabled |
| 2952 | description: | |
| 2953 | `tls.skip_verify: true` is rejected by validation. Use `tls.ca_file` for |
| 2954 | custom-CA paths if you front AWS behind an internal TLS-terminating proxy |
| 2955 | with a private CA. |
| 2956 | alerts: [] |
| 2957 | metrics: |
| 2958 | folding: |
| 2959 | title: Metrics |
| 2960 | enabled: false |
| 2961 | description: | |
| 2962 | This integration does not produce metrics. It enriches flow records with the |
| 2963 | following journal fields when a flow's source or destination IP matches an AWS |
| 2964 | prefix: |
| 2965 | |
| 2966 | - `SRC_NET_NAME` / `DST_NET_NAME` |
| 2967 | - `SRC_NET_ROLE` / `DST_NET_ROLE` |
| 2968 | - `SRC_NET_SITE` / `DST_NET_SITE` |
| 2969 | - `SRC_NET_REGION` / `DST_NET_REGION` |
| 2970 | - `SRC_NET_TENANT` / `DST_NET_TENANT` |
| 2971 | |
| 2972 | Plus, when the transform sets them, the country / state / city / ASN overrides |
| 2973 | documented on the Network Identity page. Verify on the Network Flows view via the |
| 2974 | `*_NET_*` columns or by filtering on `tenant=amazon`. |
| 2975 | availability: [] |
| 2976 | scopes: [] |
| 2977 | - meta: |
| 2978 | plugin_name: netflow-plugin |
| 2979 | module_name: gcp-ip-ranges |
| 2980 | monitored_instance: |
| 2981 | name: GCP IP Ranges |
| 2982 | link: https://cloud.google.com/compute/docs/faq#find_ip_range |
| 2983 | categories: |
| 2984 | - flows.enrichment-methods |
| 2985 | icon_filename: network-wired.svg |
| 2986 | keywords: |
| 2987 | - gcp |
| 2988 | - google cloud |
| 2989 | |
| 2990 | - cloud |
| 2991 | - ip ranges |
| 2992 | - cloud.json |
| 2993 | - goog.json |
| 2994 | - prefix list |
| 2995 | - network identity |
| 2996 | related_resources: |
| 2997 | integrations: |
| 2998 | list: |
| 2999 | - plugin_name: netflow-plugin |
| 3000 | module_name: aws-ip-ranges |
| 3001 | - plugin_name: netflow-plugin |
| 3002 | module_name: azure-ip-ranges |
| 3003 | - plugin_name: netflow-plugin |
| 3004 | module_name: netbox |
| 3005 | - plugin_name: netflow-plugin |
| 3006 | module_name: generic-ipam |
| 3007 | info_provided_to_referring_integrations: |
| 3008 | description: "" |
| 3009 | overview: |
| 3010 | data_collection: |
| 3011 | metrics_description: | |
| 3012 | Annotate network flows with Google ownership, service, and scope labels from |
| 3013 | Google public IP ranges. Google publishes its public IP ranges as two static JSON |
| 3014 | files served from `gstatic.com`. Wire one of them as a |
| 3015 | `network_source` and the netflow plugin fetches it periodically, transforms each |
| 3016 | prefix entry through your jq filter, and merges the resulting CIDR-keyed records |
| 3017 | into the network-attributes trie. Flows whose source or destination IP falls |
| 3018 | inside a Google-owned prefix are then labeled with `*_NET_TENANT`, |
| 3019 | `*_NET_REGION`, `*_NET_ROLE`, etc. according to the labels your transform emits. |
| 3020 | |
| 3021 | Two upstream files exist and they are NOT interchangeable: |
| 3022 | |
| 3023 | - **`https://www.gstatic.com/ipranges/cloud.json`** -- the prefixes used by |
| 3024 | Google Cloud (Compute Engine, GKE, Cloud Run, Cloud Functions, Cloud SQL, |
| 3025 | and the rest of Google Cloud Platform). This is what you almost always want |
| 3026 | for cloud-traffic attribution. |
| 3027 | - **`https://www.gstatic.com/ipranges/goog.json`** -- the broader Google IP |
| 3028 | space, including consumer-facing properties (Search, Gmail, YouTube, Google |
| 3029 | Workspace, Maps, Ads, etc.). It is a strict superset of `cloud.json` but |
| 3030 | **carries no service or region metadata** -- only the bare CIDRs. |
| 3031 | |
| 3032 | For the cross-cutting network-identity behavior (merge order with GeoIP and |
| 3033 | static config, jq output contract, TLS rules, single-page fetch, failure / |
| 3034 | backoff handling), see |
| 3035 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 3036 | method_description: | |
| 3037 | Periodic HTTPS GET against the chosen `gstatic.com` URL. The body is parsed |
| 3038 | as JSON and the configured `transform` (a [jaq](https://github.com/01mf02/jaq) |
| 3039 | jq-equivalent expression) runs over the parsed value. Each output object is |
| 3040 | merged into the network-attributes trie keyed on its `prefix` field; remaining |
| 3041 | fields (`name`, `role`, `site`, `region`, `country`, `state`, `city`, `tenant`, |
| 3042 | `asn`, `asn_name`) populate the corresponding `*_NET_*` flow-record columns |
| 3043 | when a flow's IP matches the prefix. |
| 3044 | |
| 3045 | The upstream files are static JSON snapshots, not pageable APIs -- one fetch |
| 3046 | per cycle is the whole dataset. Each contains a `syncToken` and a |
| 3047 | `creationTime` you can inspect to see when Google last regenerated them. |
| 3048 | The plugin fetches the full document on each refresh cycle. |
| 3049 | supported_platforms: |
| 3050 | include: |
| 3051 | - Linux |
| 3052 | exclude: [] |
| 3053 | multi_instance: true |
| 3054 | additional_permissions: |
| 3055 | description: "" |
| 3056 | default_behavior: |
| 3057 | auto_detection: |
| 3058 | description: "Disabled by default. Add an entry under enrichment.network_sources to enable." |
| 3059 | limits: |
| 3060 | description: "One full GCP prefix document is fetched per refresh. Resource use scales with the number of GCP prefixes selected by your transform and the refresh interval." |
| 3061 | performance_impact: |
| 3062 | description: | |
| 3063 | One short HTTPS GET per `interval`, plus a jq pass over the response. |
| 3064 | `cloud.json` is on the order of ~1000 prefix entries and the trie merge |
| 3065 | is bounded by that count; runtime cost is negligible against typical flow |
| 3066 | ingest. `goog.json` is similar in size. |
| 3067 | setup: |
| 3068 | prerequisites: |
| 3069 | list: |
| 3070 | - title: Outbound HTTPS to gstatic.com |
| 3071 | description: | |
| 3072 | The agent host must be able to reach `https://www.gstatic.com/ipranges/cloud.json` |
| 3073 | (or `goog.json`). The files are public CDN-served static JSON; no Google |
| 3074 | Cloud project, no service account, no API key, and no IAM permission is |
| 3075 | required. |
| 3076 | - title: Pick the right file for your use case |
| 3077 | description: | |
| 3078 | - Use `cloud.json` to attribute traffic to/from **Google Cloud Platform** |
| 3079 | services (Compute Engine VMs, GKE, Cloud Run, Cloud Functions, Cloud |
| 3080 | SQL, BigQuery API, Pub/Sub, GCS endpoints, etc.). Carries `service` and |
| 3081 | `scope` (region) per prefix. |
| 3082 | - Use `goog.json` to also catch **non-cloud Google traffic** (Gmail, |
| 3083 | YouTube, Google Workspace, Search, Ads, Maps, etc.). It carries no |
| 3084 | `service` or `scope` -- you only get "this is Google" attribution. |
| 3085 | - Configure both as separate sources if you want both attributions and |
| 3086 | a deterministic merge order between them. Within network-identity, more- |
| 3087 | specific prefixes win at lookup time, so a `cloud.json` match for a |
| 3088 | Compute Engine prefix will out-rank the broader `goog.json` umbrella. |
| 3089 | configuration: |
| 3090 | file: |
| 3091 | name: netflow.yaml |
| 3092 | options: |
| 3093 | description: | |
| 3094 | Add a named entry under `enrichment.network_sources`. The entry name is |
| 3095 | cosmetic (used in logs and metrics); the labels that end up in flow records |
| 3096 | come from the fields your `transform` emits. |
| 3097 | folding: |
| 3098 | title: Config options |
| 3099 | enabled: true |
| 3100 | list: |
| 3101 | - name: url |
| 3102 | description: | |
| 3103 | The Google IP-ranges JSON URL. Use `cloud.json` for GCP services or |
| 3104 | `goog.json` for the broader Google IP space. |
| 3105 | default_value: "https://www.gstatic.com/ipranges/cloud.json" |
| 3106 | required: true |
| 3107 | - name: interval |
| 3108 | description: | |
| 3109 | How often to refetch. Google does not publish a fixed cadence; their |
| 3110 | documentation states the lists are "published and updated frequently". |
| 3111 | Daily is plenty for most deployments and avoids hammering the CDN. |
| 3112 | default_value: "60s (loop floor)" |
| 3113 | required: false |
| 3114 | - name: timeout |
| 3115 | description: Per-request timeout. |
| 3116 | default_value: "60s" |
| 3117 | required: false |
| 3118 | - name: transform |
| 3119 | description: | |
| 3120 | jq expression that converts each entry in `prefixes[]` into an object |
| 3121 | with a `prefix` field (CIDR string) plus any of the optional label |
| 3122 | fields (`name`, `role`, `site`, `region`, `country`, `state`, `city`, |
| 3123 | `tenant`, `asn`, `asn_name`). Required. |
| 3124 | default_value: "." |
| 3125 | required: true |
| 3126 | examples: |
| 3127 | folding: |
| 3128 | title: Config |
| 3129 | enabled: true |
| 3130 | list: |
| 3131 | - name: GCP cloud.json -- tag Google Cloud prefixes by region |
| 3132 | folding: |
| 3133 | enabled: false |
| 3134 | description: | |
| 3135 | Tags every flow whose IP matches a Google Cloud prefix with |
| 3136 | `*_NET_TENANT="gcp"`, `*_NET_ROLE="google-cloud"`, and |
| 3137 | `*_NET_REGION` set to the GCP scope (`us-central1`, `europe-west1`, |
| 3138 | `global`, etc.). Both `ipv4Prefix` and `ipv6Prefix` entries are |
| 3139 | covered by the `// (slash-or)` fallback. |
| 3140 | config: | |
| 3141 | enrichment: |
| 3142 | network_sources: |
| 3143 | gcp: |
| 3144 | url: "https://www.gstatic.com/ipranges/cloud.json" |
| 3145 | interval: 24h |
| 3146 | timeout: 60s |
| 3147 | transform: | |
| 3148 | .prefixes[] | { |
| 3149 | prefix: (.ipv4Prefix // .ipv6Prefix), |
| 3150 | tenant: "gcp", |
| 3151 | role: "google-cloud", |
| 3152 | region: .scope |
| 3153 | } |
| 3154 | - name: GCP cloud.json -- IPv4 only |
| 3155 | folding: |
| 3156 | enabled: true |
| 3157 | description: | |
| 3158 | Same as above but skips the IPv6 prefixes (some operators only need |
| 3159 | IPv4 attribution and prefer to keep the trie smaller). |
| 3160 | config: | |
| 3161 | enrichment: |
| 3162 | network_sources: |
| 3163 | gcp-v4: |
| 3164 | url: "https://www.gstatic.com/ipranges/cloud.json" |
| 3165 | interval: 24h |
| 3166 | transform: | |
| 3167 | .prefixes[] |
| 3168 | | select(.ipv4Prefix) |
| 3169 | | { |
| 3170 | prefix: .ipv4Prefix, |
| 3171 | tenant: "gcp", |
| 3172 | role: "google-cloud", |
| 3173 | region: .scope |
| 3174 | } |
| 3175 | - name: Tag a single GCP region |
| 3176 | folding: |
| 3177 | enabled: true |
| 3178 | description: | |
| 3179 | Narrow the source to one region (here `us-central1`) so you can build |
| 3180 | dashboards that distinguish that region from the rest of GCP. |
| 3181 | config: | |
| 3182 | enrichment: |
| 3183 | network_sources: |
| 3184 | gcp-us-central1: |
| 3185 | url: "https://www.gstatic.com/ipranges/cloud.json" |
| 3186 | interval: 24h |
| 3187 | transform: | |
| 3188 | .prefixes[] |
| 3189 | | select(.scope == "us-central1") |
| 3190 | | { |
| 3191 | prefix: (.ipv4Prefix // .ipv6Prefix), |
| 3192 | tenant: "gcp", |
| 3193 | role: "google-cloud", |
| 3194 | region: "us-central1", |
| 3195 | site: "us-central1" |
| 3196 | } |
| 3197 | - name: Broader Google IP space (goog.json) |
| 3198 | folding: |
| 3199 | enabled: true |
| 3200 | description: | |
| 3201 | Use `goog.json` to also attribute traffic to non-Cloud Google services |
| 3202 | (Gmail, YouTube, Workspace, Search, Ads, Maps). The file has no |
| 3203 | `service` or `scope` fields, so only `tenant` / `role` are set. |
| 3204 | Configure this **alongside** `cloud.json` if you want both -- the more |
| 3205 | specific GCP prefixes from `cloud.json` will still win at lookup time. |
| 3206 | config: | |
| 3207 | enrichment: |
| 3208 | network_sources: |
| 3209 | google-all: |
| 3210 | url: "https://www.gstatic.com/ipranges/goog.json" |
| 3211 | interval: 24h |
| 3212 | transform: | |
| 3213 | .prefixes[] | { |
| 3214 | prefix: (.ipv4Prefix // .ipv6Prefix), |
| 3215 | tenant: "google", |
| 3216 | role: "google" |
| 3217 | } |
| 3218 | troubleshooting: |
| 3219 | problems: |
| 3220 | list: |
| 3221 | - name: cloud.json vs goog.json -- pick the right one |
| 3222 | description: | |
| 3223 | `cloud.json` is the Google Cloud Platform list (Compute Engine, GKE, |
| 3224 | Cloud Run, Cloud SQL, BigQuery, GCS, etc.) and carries `service` plus |
| 3225 | `scope` (region) per entry. `goog.json` is the broader Google list |
| 3226 | (consumer products: Gmail, YouTube, Workspace, Search, Ads, Maps) and |
| 3227 | carries **only the bare CIDRs** -- no service, no scope, no region. |
| 3228 | Using `goog.json` as if it were `cloud.json` will give you "this is |
| 3229 | Google" attribution but no per-region or per-service breakdown. Most |
| 3230 | operators want `cloud.json`; some configure both as separate sources. |
| 3231 | - name: No per-service breakdown from cloud.json |
| 3232 | description: | |
| 3233 | Today every entry in `cloud.json` reports `service: "Google Cloud"` -- |
| 3234 | the file does not split prefixes by individual GCP service (Compute |
| 3235 | Engine vs GKE vs Cloud Run, etc.). Region (`scope`) is the dimension |
| 3236 | you can actually pivot on. Per-service attribution requires a different |
| 3237 | data source. |
| 3238 | - name: Empty result from the transform is treated as failure |
| 3239 | description: | |
| 3240 | If the jq expression yields zero objects (for example, an over-narrow |
| 3241 | `select()` that no entry passes), the source backs off as if the fetch |
| 3242 | had errored. Check the journal for `network-sources` warnings and verify |
| 3243 | your filter against a saved copy of `cloud.json`. |
| 3244 | - name: TLS verification cannot be disabled |
| 3245 | description: | |
| 3246 | `tls.skip_verify: true` (and `tls.verify: false`) are rejected by |
| 3247 | validation. `gstatic.com` is publicly trusted so this is rarely an issue |
| 3248 | for GCP; if you front the URL through an internal proxy with a private |
| 3249 | CA, supply it via `tls.ca_file`. |
| 3250 | - name: Update cadence is not contractual |
| 3251 | description: | |
| 3252 | Google states the lists are "published and updated frequently" but does |
| 3253 | not guarantee a fixed cadence. The `syncToken` and `creationTime` keys |
| 3254 | at the top of each file let you check when Google last regenerated it. |
| 3255 | Daily polling is a safe default; sub-hourly is unnecessary. |
| 3256 | alerts: [] |
| 3257 | metrics: |
| 3258 | folding: |
| 3259 | title: Metrics |
| 3260 | enabled: false |
| 3261 | description: | |
| 3262 | Tags flow records with `SRC_NET_TENANT` / `DST_NET_TENANT`, |
| 3263 | `SRC_NET_REGION` / `DST_NET_REGION`, `SRC_NET_ROLE` / `DST_NET_ROLE`, |
| 3264 | and any other `*_NET_*` fields your transform emits, when the source or |
| 3265 | destination IP matches a Google prefix. Verify in the Network Flows view via |
| 3266 | the `*_NET_*` columns. Coordinates (`*_GEO_LATITUDE` / `*_GEO_LONGITUDE`) |
| 3267 | cannot be set from network-identity sources -- use the static `networks` |
| 3268 | block for those. |
| 3269 | availability: [] |
| 3270 | scopes: [] |
| 3271 | - meta: |
| 3272 | plugin_name: netflow-plugin |
| 3273 | module_name: azure-ip-ranges |
| 3274 | monitored_instance: |
| 3275 | name: Azure IP Ranges |
| 3276 | link: https://www.microsoft.com/en-us/download/details.aspx?id=56519 |
| 3277 | categories: |
| 3278 | - flows.enrichment-methods |
| 3279 | icon_filename: network-wired.svg |
| 3280 | keywords: |
| 3281 | - azure |
| 3282 | - microsoft |
| 3283 | - cloud |
| 3284 | - ip ranges |
| 3285 | - service tags |
| 3286 | related_resources: |
| 3287 | integrations: |
| 3288 | list: |
| 3289 | - plugin_name: netflow-plugin |
| 3290 | module_name: aws-ip-ranges |
| 3291 | - plugin_name: netflow-plugin |
| 3292 | module_name: gcp-ip-ranges |
| 3293 | - plugin_name: netflow-plugin |
| 3294 | module_name: netbox |
| 3295 | - plugin_name: netflow-plugin |
| 3296 | module_name: generic-ipam |
| 3297 | info_provided_to_referring_integrations: |
| 3298 | description: "" |
| 3299 | overview: |
| 3300 | data_collection: |
| 3301 | metrics_description: | |
| 3302 | Annotate network flows with Azure service and region labels from Azure Service |
| 3303 | Tags. Microsoft publishes Azure's public IP prefixes as **Service Tags JSON |
| 3304 | files**. A service tag is Microsoft's name for a group of IP prefixes belonging |
| 3305 | to one Azure service (e.g., `Storage`, `Sql`, `AzureFrontDoor.Backend`); the |
| 3306 | JSON enumerates every tag together with the prefixes, the platform, and -- for |
| 3307 | tags that support regional scope -- the Azure region. |
| 3308 | |
| 3309 | Microsoft publishes four separate JSON files, one per Azure cloud: |
| 3310 | |
| 3311 | - **Azure Public** -- commercial Azure, the file most users want. |
| 3312 | - **Azure US Government** -- separate cloud for US public-sector workloads. |
| 3313 | - **Microsoft Azure operated by 21Vianet** -- the China cloud. |
| 3314 | - **Azure Germany** -- legacy German cloud. |
| 3315 | |
| 3316 | This integration tags flow records to/from Azure-hosted services with |
| 3317 | `*_NET_TENANT="azure"` plus per-region and per-service labels. Use it to |
| 3318 | identify traffic to Azure SQL, Azure Storage, Azure Front Door, App Service, |
| 3319 | and the rest -- without maintaining the prefix list yourself. |
| 3320 | |
| 3321 | For the full network-identity concept (merge order, jq output shape, TLS |
| 3322 | verification, failure modes), see |
| 3323 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 3324 | method_description: | |
| 3325 | Periodic HTTPS GET against the Service Tags JSON, jq transform via the |
| 3326 | [jaq](https://github.com/01mf02/jaq) library, merge into the network-attributes |
| 3327 | trie. Same mechanism as AWS / GCP IP Ranges -- different URL and JSON shape, |
| 3328 | plus one important operational caveat documented below. |
| 3329 | supported_platforms: |
| 3330 | include: |
| 3331 | - Linux |
| 3332 | exclude: [] |
| 3333 | multi_instance: true |
| 3334 | additional_permissions: |
| 3335 | description: "" |
| 3336 | default_behavior: |
| 3337 | auto_detection: |
| 3338 | description: "Disabled by default. Add an entry under enrichment.network_sources to enable." |
| 3339 | limits: |
| 3340 | description: "One full Azure Service Tags document is fetched per refresh. Resource use scales with the number of Azure prefixes selected by your transform and the refresh interval." |
| 3341 | performance_impact: |
| 3342 | description: "One HTTPS request per refresh interval plus a jq transform over the Azure Service Tags document. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records." |
| 3343 | setup: |
| 3344 | prerequisites: |
| 3345 | list: |
| 3346 | - title: Azure publishes weekly, but the URL rotates |
| 3347 | description: | |
| 3348 | Microsoft updates the Service Tags JSON **weekly** (per |
| 3349 | <https://learn.microsoft.com/en-us/azure/virtual-network/service-tags-overview#discover-service-tags-by-using-downloadable-json-files>: |
| 3350 | "These lists are updated and published weekly"), and Microsoft asks you |
| 3351 | to delay rolling out new prefixes for at least one week after publication |
| 3352 | ("When new IP addresses are added to service tags, they aren't used in |
| 3353 | Azure for at least one week"). |
| 3354 | |
| 3355 | The catch: the actual download URL is **not stable**. The download portal |
| 3356 | at <https://www.microsoft.com/en-us/download/details.aspx?id=56519> serves |
| 3357 | a `ServiceTags_Public_<YYYYMMDD>.json` filename whose date stamp changes |
| 3358 | every week. A naive `interval`-based fetch against last week's URL will |
| 3359 | start 404'ing as soon as a new file is published. |
| 3360 | |
| 3361 | You have three reasonable options, in increasing order of effort: |
| 3362 | |
| 3363 | 1. **Skip Azure IP Ranges entirely** and rely on GeoIP / ASN data |
| 3364 | (Microsoft's main ASN is 8075). Coarse but zero-maintenance. |
| 3365 | 2. **Use a stable redirect** if your environment provides one (some |
| 3366 | enterprise mirrors do). |
| 3367 | 3. **Run a small mirror script** -- a cron job that fetches the latest |
| 3368 | JSON via the [Service Tag Discovery REST API](https://learn.microsoft.com/en-us/rest/api/virtualnetwork/servicetags/list) |
| 3369 | (or `Get-AzNetworkServiceTag` in PowerShell) and writes the result to |
| 3370 | a stable path on an internal HTTP server. Point the plugin at that |
| 3371 | stable URL. |
| 3372 | |
| 3373 | Option 3 is what most operators end up with. The Service Tag Discovery |
| 3374 | API is authoritative and authenticated (Azure subscription with read |
| 3375 | role); the downloadable JSON is unauthenticated but URL-rotated. Pick |
| 3376 | whichever fits your operational model. |
| 3377 | - title: Outbound HTTPS to your mirror (or to Microsoft, if you script the URL discovery) |
| 3378 | description: | |
| 3379 | The plugin only needs to reach the configured `url`. Azure credentials |
| 3380 | are needed only on the side that resolves "what's this week's URL?" -- |
| 3381 | not by the plugin itself. |
| 3382 | - title: A jq transform that matches the Service Tags schema |
| 3383 | description: | |
| 3384 | The Service Tags JSON has a nested shape: |
| 3385 | |
| 3386 | ```text |
| 3387 | { |
| 3388 | "changeNumber": 123, |
| 3389 | "cloud": "Public", |
| 3390 | "values": [ |
| 3391 | { |
| 3392 | "name": "Storage.WestUS", |
| 3393 | "id": "Storage.WestUS", |
| 3394 | "properties": { |
| 3395 | "changeNumber": 456, |
| 3396 | "region": "westus", |
| 3397 | "regionId": 30, |
| 3398 | "platform": "Azure", |
| 3399 | "systemService": "AzureStorage", |
| 3400 | "addressPrefixes": ["13.105.16.4/30", "20.150.0.0/17", ...], |
| 3401 | "networkFeatures": ["API", "NSG"] |
| 3402 | } |
| 3403 | }, |
| 3404 | ... |
| 3405 | ] |
| 3406 | } |
| 3407 | ``` |
| 3408 | |
| 3409 | The transform must unwrap `values[]` -> `properties.addressPrefixes[]` |
| 3410 | and emit one object per prefix. Schema reference: `values[].name`, |
| 3411 | `values[].id`, |
| 3412 | `values[].properties.region` (region tag, can be empty for |
| 3413 | cloud-wide tags like the bare `Storage`), `values[].properties.platform`, |
| 3414 | `values[].properties.systemService` (e.g. `AzureStorage`, `AzureSql`), |
| 3415 | `values[].properties.addressPrefixes[]` (CIDR strings, IPv4 and IPv6 |
| 3416 | mixed). Top-level `changeNumber` increments on every publication. |
| 3417 | configuration: |
| 3418 | file: |
| 3419 | name: netflow.yaml |
| 3420 | options: |
| 3421 | description: | |
| 3422 | Add a named entry under `enrichment.network_sources` pointing at your |
| 3423 | stable mirror URL (or, with the limitations above, at this week's |
| 3424 | `ServiceTags_Public_<date>.json` URL). |
| 3425 | folding: |
| 3426 | title: Config options |
| 3427 | enabled: true |
| 3428 | list: |
| 3429 | - name: url |
| 3430 | description: Stable URL to your locally-mirrored Azure Service Tags JSON, or this week's date-stamped Microsoft URL. |
| 3431 | default_value: "" |
| 3432 | required: true |
| 3433 | - name: interval |
| 3434 | description: How often to fetch. Microsoft publishes weekly; daily is plenty. |
| 3435 | default_value: "60s (loop floor)" |
| 3436 | required: false |
| 3437 | - name: timeout |
| 3438 | description: Per-request timeout. Service Tags JSON files are large (tens of MB) -- give it room. |
| 3439 | default_value: "60s" |
| 3440 | required: false |
| 3441 | - name: transform |
| 3442 | description: jq expression that maps `values[].properties.addressPrefixes[]` into per-prefix objects. |
| 3443 | default_value: "." |
| 3444 | required: true |
| 3445 | examples: |
| 3446 | folding: |
| 3447 | title: Config |
| 3448 | enabled: true |
| 3449 | list: |
| 3450 | - name: Azure Public via internal mirror -- tag by region and service |
| 3451 | folding: |
| 3452 | enabled: false |
| 3453 | description: | |
| 3454 | Internal mirror serving the latest Public-cloud Service Tags JSON. |
| 3455 | Tags every Azure prefix with `tenant=azure`, the region, and the |
| 3456 | lowercase systemService name as the role. |
| 3457 | config: | |
| 3458 | enrichment: |
| 3459 | network_sources: |
| 3460 | azure: |
| 3461 | url: "https://internal.example/azure-service-tags.json" |
| 3462 | interval: 24h |
| 3463 | timeout: 60s |
| 3464 | transform: | |
| 3465 | .values[] |
| 3466 | | .properties as $p |
| 3467 | | $p.addressPrefixes[] |
| 3468 | | { |
| 3469 | prefix: ., |
| 3470 | tenant: "azure", |
| 3471 | region: ($p.region // ""), |
| 3472 | role: (($p.systemService // "") | ascii_downcase) |
| 3473 | } |
| 3474 | - name: Azure Storage only |
| 3475 | folding: |
| 3476 | enabled: true |
| 3477 | description: Filter to a single Azure systemService for narrower tagging. |
| 3478 | config: | |
| 3479 | enrichment: |
| 3480 | network_sources: |
| 3481 | azure-storage: |
| 3482 | url: "https://internal.example/azure-service-tags.json" |
| 3483 | interval: 24h |
| 3484 | transform: | |
| 3485 | .values[] |
| 3486 | | select(.properties.systemService == "AzureStorage") |
| 3487 | | .properties as $p |
| 3488 | | $p.addressPrefixes[] |
| 3489 | | { |
| 3490 | prefix: ., |
| 3491 | tenant: "azure", |
| 3492 | role: "storage", |
| 3493 | region: ($p.region // "") |
| 3494 | } |
| 3495 | - name: Azure US Government cloud |
| 3496 | folding: |
| 3497 | enabled: true |
| 3498 | description: | |
| 3499 | The US Government cloud is published as a separate JSON file with its |
| 3500 | own download ID (57063). Same schema as the Public cloud; tag tenant |
| 3501 | differently so you can tell them apart in dashboards. |
| 3502 | config: | |
| 3503 | enrichment: |
| 3504 | network_sources: |
| 3505 | azure-usgov: |
| 3506 | url: "https://internal.example/azure-service-tags-usgov.json" |
| 3507 | interval: 24h |
| 3508 | transform: | |
| 3509 | .values[] |
| 3510 | | .properties as $p |
| 3511 | | $p.addressPrefixes[] |
| 3512 | | { |
| 3513 | prefix: ., |
| 3514 | tenant: "azure-usgov", |
| 3515 | region: ($p.region // ""), |
| 3516 | role: (($p.systemService // "") | ascii_downcase) |
| 3517 | } |
| 3518 | troubleshooting: |
| 3519 | problems: |
| 3520 | list: |
| 3521 | - name: 404 a week after deploy |
| 3522 | description: | |
| 3523 | You configured `url` against `ServiceTags_Public_<date>.json` directly. |
| 3524 | Microsoft rotates the date stamp weekly, so the URL stops resolving |
| 3525 | within a week. Move to an internal mirror that resolves the latest URL |
| 3526 | each cycle, or to the Service Tag Discovery REST API |
| 3527 | (<https://learn.microsoft.com/en-us/rest/api/virtualnetwork/servicetags/list>) |
| 3528 | fed into a static file. |
| 3529 | - name: Empty results from the transform |
| 3530 | description: | |
| 3531 | The Service Tags JSON is nested two levels deep |
| 3532 | (`values[].properties.addressPrefixes[]`). If the jq doesn't unwrap |
| 3533 | both levels, every fetch yields zero rows and the source backs off as if |
| 3534 | it errored. Test the jq locally with `jq < azure-service-tags.json` and |
| 3535 | confirm at least one `{prefix, tenant, ...}` object comes out. |
| 3536 | - name: Service Tag API data lags the JSON file |
| 3537 | description: | |
| 3538 | Per Microsoft's note on the Service Tags overview page, "It takes up to |
| 3539 | four weeks for new Service Tag data to propagate in the API results |
| 3540 | across all Azure regions". If you mirror via the REST API, expect newly |
| 3541 | added tags to appear in the JSON download a few weeks before the API, |
| 3542 | not the other way around. For prefix tagging this rarely matters; for |
| 3543 | firewall rules it can. |
| 3544 | - name: TLS verification cannot be disabled |
| 3545 | description: | |
| 3546 | `tls.skip_verify: true` is rejected by validation. Use `tls.ca_file` |
| 3547 | for custom-CA paths (e.g., on an internal mirror with a private CA). |
| 3548 | alerts: [] |
| 3549 | metrics: |
| 3550 | folding: |
| 3551 | title: Metrics |
| 3552 | enabled: false |
| 3553 | description: | |
| 3554 | Tags flow records with `*_NET_TENANT`, `*_NET_REGION`, `*_NET_ROLE` when the |
| 3555 | source or destination IP matches an Azure prefix. Verify on the Network Flows |
| 3556 | tab via the `*_NET_*` columns. Country / state / city overrides |
| 3557 | (`*_COUNTRY`, `*_GEO_STATE`, `*_GEO_CITY`) are also available if your |
| 3558 | transform sets them, but Service Tags don't carry country/state/city data -- |
| 3559 | only `region` codes like `westus` or `northeurope`. |
| 3560 | availability: [] |
| 3561 | scopes: [] |
| 3562 | - meta: |
| 3563 | plugin_name: netflow-plugin |
| 3564 | module_name: netbox |
| 3565 | monitored_instance: |
| 3566 | name: NetBox |
| 3567 | link: https://github.com/netbox-community/netbox |
| 3568 | categories: |
| 3569 | - flows.enrichment-methods |
| 3570 | icon_filename: network-wired.svg |
| 3571 | keywords: |
| 3572 | - netbox |
| 3573 | - ipam |
| 3574 | - dcim |
| 3575 | - source of truth |
| 3576 | - prefix list |
| 3577 | - network identity |
| 3578 | related_resources: |
| 3579 | integrations: |
| 3580 | list: |
| 3581 | - plugin_name: netflow-plugin |
| 3582 | module_name: aws-ip-ranges |
| 3583 | - plugin_name: netflow-plugin |
| 3584 | module_name: gcp-ip-ranges |
| 3585 | - plugin_name: netflow-plugin |
| 3586 | module_name: azure-ip-ranges |
| 3587 | - plugin_name: netflow-plugin |
| 3588 | module_name: generic-ipam |
| 3589 | info_provided_to_referring_integrations: |
| 3590 | description: "" |
| 3591 | overview: |
| 3592 | data_collection: |
| 3593 | metrics_description: | |
| 3594 | Annotate network flows with tenant, site, role, VRF, and description metadata |
| 3595 | from NetBox. [NetBox](https://github.com/netbox-community/netbox) is the most |
| 3596 | widely deployed open-source IPAM / DCIM. It is Apache-2.0 licensed and maintained |
| 3597 | by the netbox-community organization (commercial variants -- NetBox Cloud and |
| 3598 | NetBox Enterprise -- are offered by NetBox Labs on top of the same upstream code). |
| 3599 | Operators already curate prefix metadata in NetBox and want flow records to |
| 3600 | inherit those labels automatically rather than maintaining a parallel list in |
| 3601 | `netflow.yaml`. |
| 3602 | |
| 3603 | This integration polls NetBox's Prefixes REST API at a configurable interval, |
| 3604 | transforms the response with jq, and labels matching flow prefixes with whatever |
| 3605 | fields you map. NetBox is the authoritative source -- when the same prefix is |
| 3606 | tagged in NetBox and in the static `networks:` block, the static block wins |
| 3607 | (operator override is intentional, see the |
| 3608 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment) |
| 3609 | concept page). |
| 3610 | |
| 3611 | Use cases: applying your organisation's data-centre rack labels, tenant names, |
| 3612 | environment tags (prod / staging / dev), and VRF / site identifiers to flows so |
| 3613 | dashboards and queries surface "this is the staging tier in fra1" instead of a |
| 3614 | raw CIDR. |
| 3615 | method_description: | |
| 3616 | Periodic HTTPS GET to `/api/ipam/prefixes/` on your NetBox host. Authentication is |
| 3617 | a NetBox API token in the `Authorization` header. NetBox supports two token |
| 3618 | formats: |
| 3619 | |
| 3620 | - **Legacy v1 tokens** (`Authorization: Token <token>`) -- accepted by all NetBox |
| 3621 | versions, simplest to wire up. |
| 3622 | - **v2 tokens** (NetBox 4.5+, `Authorization: Bearer nbt_<12-char-key>.<40-char-token>`) -- |
| 3623 | the prefix `nbt_`, the token key, a dot, and the token secret are all part of |
| 3624 | the header value. |
| 3625 | |
| 3626 | The plugin transports either format -- the value is whatever NetBox issued for the |
| 3627 | service account. |
| 3628 | |
| 3629 | The plugin runs the configured `transform` (jaq -- a jq-equivalent) over the |
| 3630 | parsed JSON body and produces per-prefix objects. NetBox's response is paginated; |
| 3631 | the plugin does **not** follow `next` links. Pass `?limit=0` (NetBox default |
| 3632 | `MAX_PAGE_SIZE` is 1000; setting `0` removes the cap when the server config |
| 3633 | allows) or an explicit `?limit=N` greater than your prefix count, or expose a |
| 3634 | server-side aggregator that returns the full list at one URL. |
| 3635 | supported_platforms: |
| 3636 | include: |
| 3637 | - Linux |
| 3638 | exclude: [] |
| 3639 | multi_instance: true |
| 3640 | additional_permissions: |
| 3641 | description: "" |
| 3642 | default_behavior: |
| 3643 | auto_detection: |
| 3644 | description: "Disabled by default. Add an entry under `enrichment.network_sources` with your NetBox URL and API token." |
| 3645 | limits: |
| 3646 | description: "Resource use scales with the number of NetBox prefixes returned by the URL and transform. Use NetBox filtering when you only need a subset of VRFs, sites, tenants, or roles." |
| 3647 | performance_impact: |
| 3648 | description: "One HTTP request per refresh interval plus a jq transform over the response. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records." |
| 3649 | setup: |
| 3650 | prerequisites: |
| 3651 | list: |
| 3652 | - title: NetBox API token with read scope on Prefixes |
| 3653 | description: | |
| 3654 | In NetBox, create or reuse a service-account user, then generate an API token |
| 3655 | under "Admin > Users > Tokens". Restrict the token to read-only and (for |
| 3656 | NetBox 4.x) limit the scope to `ipam.view_prefix`. The token value goes in the |
| 3657 | `Authorization` header. |
| 3658 | |
| 3659 | The plugin only reads -- never writes -- so a read-only token is sufficient |
| 3660 | and recommended. Token format depends on the NetBox version: |
| 3661 | |
| 3662 | - NetBox 3.x or earlier: `Token <40-char-hex>` (legacy). |
| 3663 | - NetBox 4.0 through 4.4: legacy `Token <hex>`. |
| 3664 | - NetBox 4.5+: legacy `Token <hex>` or new |
| 3665 | `Bearer nbt_<12-char-key>.<40-char-token>` (v2 tokens). |
| 3666 | - title: Bulk endpoint (`?limit=0` or aggregator) |
| 3667 | description: | |
| 3668 | The plugin fetches a single page. NetBox's default `PAGINATE_COUNT` is 50 |
| 3669 | and `MAX_PAGE_SIZE` is 1000. For inventories above 1000 prefixes, either |
| 3670 | raise `MAX_PAGE_SIZE` server-side and pass `?limit=0`, or expose an internal |
| 3671 | aggregator endpoint that walks pagination and returns the full list at one URL. |
| 3672 | - title: NetBox version-aware field mapping |
| 3673 | description: | |
| 3674 | NetBox **4.2** replaced the per-prefix `site` foreign key with a generic |
| 3675 | `scope` field (a prefix can now be scoped to a region, site group, site, or |
| 3676 | location -- not just a site). API responses on 4.2+ omit `site` and expose |
| 3677 | `scope` (read-only) plus `scope_type` / `scope_id`. Adjust your jq accordingly |
| 3678 | -- on 4.2+ use `(.scope.name // "")`, on 4.1 and earlier use |
| 3679 | `(.site.name // "")`. |
| 3680 | configuration: |
| 3681 | file: |
| 3682 | name: netflow.yaml |
| 3683 | options: |
| 3684 | description: | |
| 3685 | Add a named entry under `enrichment.network_sources` pointing at your NetBox. |
| 3686 | The network source configuration accepts `url`, `method`, |
| 3687 | `headers`, `proxy`, `tls`, `timeout`, `interval`, and `transform`. Authentication |
| 3688 | is configured via the generic `headers:` map -- there is no NetBox-specific |
| 3689 | auth helper. |
| 3690 | folding: |
| 3691 | title: Config options |
| 3692 | enabled: true |
| 3693 | list: |
| 3694 | - name: url |
| 3695 | description: NetBox prefixes API endpoint, including `?limit=` (recommend `?limit=0` for full inventory in one shot when the server's `MAX_PAGE_SIZE` allows it). |
| 3696 | default_value: "" |
| 3697 | required: true |
| 3698 | - name: headers.Authorization |
| 3699 | description: 'NetBox API token. Use `Token <hex>` for legacy v1 or `Bearer nbt_<12-char-key>.<40-char-token>` for v4.5+ v2 tokens.' |
| 3700 | default_value: "" |
| 3701 | required: true |
| 3702 | - name: interval |
| 3703 | description: How often to refresh. NetBox is your source of truth; 5 minutes is typical for IPAMs that change frequently, 1 hour is fine for static inventories. |
| 3704 | default_value: "60s" |
| 3705 | required: false |
| 3706 | - name: timeout |
| 3707 | description: HTTP request timeout. Bump to 30-60s if your NetBox returns thousands of prefixes in one shot. |
| 3708 | default_value: "10s" |
| 3709 | required: false |
| 3710 | - name: transform |
| 3711 | description: jq expression mapping NetBox's `.results[]` to per-prefix objects with `prefix` and any of `name`, `role`, `site`, `region`, `country`, `state`, `city`, `tenant`, `asn`, `asn_name`. |
| 3712 | default_value: "." |
| 3713 | required: true |
| 3714 | - name: tls.ca_file |
| 3715 | description: Path to your internal CA bundle when NetBox runs behind internal PKI. Disabling verification is rejected by the validator -- use this instead. |
| 3716 | default_value: "" |
| 3717 | required: false |
| 3718 | examples: |
| 3719 | folding: |
| 3720 | title: Config |
| 3721 | enabled: true |
| 3722 | list: |
| 3723 | - name: NetBox 4.5+ with v2 token, scope-aware mapping |
| 3724 | folding: |
| 3725 | enabled: false |
| 3726 | description: | |
| 3727 | Standard NetBox 4.2+ wiring. Maps tenant, role, the new `scope` field |
| 3728 | (covers site / region / site-group / location), and the human-readable |
| 3729 | description. `?limit=0` returns all prefixes in one call when |
| 3730 | `MAX_PAGE_SIZE` is set to `0` or `None` on the NetBox server. |
| 3731 | config: | |
| 3732 | enrichment: |
| 3733 | network_sources: |
| 3734 | netbox: |
| 3735 | url: "https://netbox.example.internal/api/ipam/prefixes/?limit=0" |
| 3736 | headers: |
| 3737 | Authorization: "Bearer nbt_<12-char-key>.<40-char-token>" |
| 3738 | interval: 5m |
| 3739 | timeout: 30s |
| 3740 | transform: | |
| 3741 | .results[] | { |
| 3742 | prefix: .prefix, |
| 3743 | tenant: (.tenant.name // ""), |
| 3744 | role: (.role.name // ""), |
| 3745 | site: (.scope.name // ""), |
| 3746 | name: (.description // "") |
| 3747 | } |
| 3748 | - name: NetBox 3.x / 4.0 / 4.1 with legacy site field |
| 3749 | folding: |
| 3750 | enabled: true |
| 3751 | description: | |
| 3752 | Pre-4.2 NetBox still exposes `site` directly on the Prefix object. Use a |
| 3753 | legacy `Token` header and read `.site.name` instead of `.scope.name`. |
| 3754 | config: | |
| 3755 | enrichment: |
| 3756 | network_sources: |
| 3757 | netbox: |
| 3758 | url: "https://netbox.example.internal/api/ipam/prefixes/?limit=10000" |
| 3759 | headers: |
| 3760 | Authorization: "Token abcdef0123456789abcdef0123456789abcdef01" |
| 3761 | interval: 15m |
| 3762 | transform: | |
| 3763 | .results[] | { |
| 3764 | prefix: .prefix, |
| 3765 | tenant: (.tenant.name // ""), |
| 3766 | role: (.role.name // ""), |
| 3767 | site: (.site.name // ""), |
| 3768 | name: (.description // "") |
| 3769 | } |
| 3770 | - name: NetBox behind internal PKI (mTLS) |
| 3771 | folding: |
| 3772 | enabled: true |
| 3773 | description: | |
| 3774 | When NetBox is fronted by internal PKI. `tls.verify: false` is rejected at |
| 3775 | config-load -- supply your CA explicitly and (optionally) a client |
| 3776 | certificate. |
| 3777 | config: | |
| 3778 | enrichment: |
| 3779 | network_sources: |
| 3780 | netbox: |
| 3781 | url: "https://netbox.example.internal/api/ipam/prefixes/?limit=0" |
| 3782 | headers: |
| 3783 | Authorization: "Token abcdef0123456789abcdef0123456789abcdef01" |
| 3784 | interval: 5m |
| 3785 | tls: |
| 3786 | enable: true |
| 3787 | ca_file: /etc/netdata/ssl/internal-ca.pem |
| 3788 | cert_file: /etc/netdata/ssl/netdata.crt |
| 3789 | key_file: /etc/netdata/ssl/netdata.key |
| 3790 | transform: | |
| 3791 | .results[] | { |
| 3792 | prefix: .prefix, |
| 3793 | tenant: (.tenant.name // ""), |
| 3794 | role: (.role.name // ""), |
| 3795 | site: (.scope.name // ""), |
| 3796 | name: (.description // "") |
| 3797 | } |
| 3798 | - name: VRF-aware mapping (multi-tenant networks) |
| 3799 | folding: |
| 3800 | enabled: true |
| 3801 | description: | |
| 3802 | When the same RFC1918 prefix appears in multiple VRFs, fold the VRF name |
| 3803 | into the friendly `name` so dashboards disambiguate. Note: the plugin keys |
| 3804 | on `prefix` only -- if two rows share a CIDR, the last one wins. Filter the |
| 3805 | NetBox query (`?vrf_id=N`) to scope the export. |
| 3806 | config: | |
| 3807 | enrichment: |
| 3808 | network_sources: |
| 3809 | netbox-prod-vrf: |
| 3810 | url: "https://netbox.example.internal/api/ipam/prefixes/?limit=0&vrf_id=12" |
| 3811 | headers: |
| 3812 | Authorization: "Token abcdef0123456789abcdef0123456789abcdef01" |
| 3813 | interval: 5m |
| 3814 | transform: | |
| 3815 | .results[] | { |
| 3816 | prefix: .prefix, |
| 3817 | tenant: (.tenant.name // ""), |
| 3818 | role: (.role.name // ""), |
| 3819 | site: (.scope.name // ""), |
| 3820 | name: ((.vrf.name // "default") + ": " + (.description // "")) |
| 3821 | } |
| 3822 | troubleshooting: |
| 3823 | problems: |
| 3824 | list: |
| 3825 | - name: Only first 50 prefixes loaded |
| 3826 | description: | |
| 3827 | NetBox's default `PAGINATE_COUNT` is 50 and the plugin does not follow `next` |
| 3828 | links. Pass `?limit=0` (NetBox removes the cap when `MAX_PAGE_SIZE` is |
| 3829 | `0` server-side) or `?limit=N` larger than your inventory. For inventories |
| 3830 | above the server's `MAX_PAGE_SIZE` (default 1000), expose a server-side |
| 3831 | aggregator endpoint. |
| 3832 | - name: 401 / 403 from NetBox |
| 3833 | description: | |
| 3834 | Token missing, expired, or wrong format. Verify with one of: |
| 3835 | `curl -H "Authorization: Token <legacy-token>" https://netbox/api/ipam/prefixes/` |
| 3836 | or |
| 3837 | `curl -H "Authorization: Bearer nbt_<12-char-key>.<40-char-token>" https://netbox/api/ipam/prefixes/`. |
| 3838 | On NetBox 4.5+ check whether the token is v1 (`Token <hex>`) or v2 |
| 3839 | (`Bearer nbt_<12-char-key>.<40-char-token>`) and use the matching header. |
| 3840 | A v2 token sent with the legacy `Token` prefix is rejected as an invalid v1 |
| 3841 | token. Watch the journal |
| 3842 | for `network-sources` warnings -- HTTP errors are logged there as |
| 3843 | refresh-failed warnings. |
| 3844 | - name: site is empty after upgrade to NetBox 4.2 |
| 3845 | description: | |
| 3846 | NetBox 4.2 removed `site` from the Prefix model and replaced it with `scope` |
| 3847 | (a generic foreign key that can reference region / site group / site / |
| 3848 | location). Update the jq from `(.site.name // "")` to `(.scope.name // "")`. |
| 3849 | Alternatively, key on `(.scope.name // .site.name // "")` so the same config |
| 3850 | works across the upgrade boundary. |
| 3851 | - name: VRF collisions (same CIDR in multiple VRFs) |
| 3852 | description: | |
| 3853 | The plugin's network-attributes trie keys on `prefix` only -- there is no VRF |
| 3854 | dimension on the flow side. If NetBox lists `10.0.0.0/24` in three VRFs, only |
| 3855 | the last row wins. Filter the export with `?vrf_id=N` per data-plane and run |
| 3856 | one named source per VRF, or fold the VRF name into the friendly `name` so |
| 3857 | dashboards disambiguate. |
| 3858 | - name: Empty result triggers backoff |
| 3859 | description: | |
| 3860 | An empty `results` array (legitimate state for a freshly-installed NetBox) |
| 3861 | is treated as a fetch failure by the cross-cutting source loop and triggers |
| 3862 | exponential backoff. Add at least one synthetic prefix (e.g. a `RFC1918` |
| 3863 | container) so the response is never empty. |
| 3864 | alerts: [] |
| 3865 | metrics: |
| 3866 | folding: |
| 3867 | title: Metrics |
| 3868 | enabled: false |
| 3869 | description: | |
| 3870 | Tags flow records with the labels you mapped from NetBox -- typically |
| 3871 | `*_NET_TENANT`, `*_NET_SITE`, `*_NET_ROLE`, `*_NET_NAME`. Verify on the Network |
| 3872 | Flows tab. |
| 3873 | availability: [] |
| 3874 | scopes: [] |
| 3875 | - meta: |
| 3876 | plugin_name: netflow-plugin |
| 3877 | module_name: generic-ipam |
| 3878 | monitored_instance: |
| 3879 | name: Generic JSON-over-HTTP IPAM |
| 3880 | link: https://learn.netdata.cloud/docs/network-flows/enrichment |
| 3881 | categories: |
| 3882 | - flows.enrichment-methods |
| 3883 | icon_filename: network-wired.svg |
| 3884 | keywords: |
| 3885 | - ipam |
| 3886 | - cmdb |
| 3887 | - infoblox |
| 3888 | - bluecat |
| 3889 | - phpipam |
| 3890 | - servicenow |
| 3891 | - custom |
| 3892 | - prefix list |
| 3893 | - network identity |
| 3894 | related_resources: |
| 3895 | integrations: |
| 3896 | list: |
| 3897 | - plugin_name: netflow-plugin |
| 3898 | module_name: aws-ip-ranges |
| 3899 | - plugin_name: netflow-plugin |
| 3900 | module_name: gcp-ip-ranges |
| 3901 | - plugin_name: netflow-plugin |
| 3902 | module_name: azure-ip-ranges |
| 3903 | - plugin_name: netflow-plugin |
| 3904 | module_name: netbox |
| 3905 | info_provided_to_referring_integrations: |
| 3906 | description: "" |
| 3907 | overview: |
| 3908 | data_collection: |
| 3909 | metrics_description: | |
| 3910 | Annotate network flows with network-identity labels from any JSON-over-HTTP IPAM |
| 3911 | or CMDB endpoint. The plugin does not care WHO produces the JSON -- it cares |
| 3912 | about the JSON shape after your `transform` (a jq expression) runs over it. |
| 3913 | Examples: Infoblox WAPI, BlueCat REST API, phpIPAM, ServiceNow CMDB queries, an |
| 3914 | internal aggregator, a Lambda function, a static file served from S3, a |
| 3915 | hand-rolled CMDB. |
| 3916 | |
| 3917 | This card is also the **reference description** of how all the other Network |
| 3918 | Identity cards work under the hood. AWS IP Ranges, GCP IP Ranges, Azure IP Ranges |
| 3919 | and NetBox are all special-cased instances of this same mechanism: each is just |
| 3920 | a known URL plus a recommended jq `transform`. Read this card to understand the |
| 3921 | generic shape, then read the per-source cards for the URL and transform that fit |
| 3922 | that specific provider. |
| 3923 | |
| 3924 | For the full network-identity concept (merge order vs GeoIP and static, what |
| 3925 | fields you can populate, why TLS verification cannot be disabled), see |
| 3926 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 3927 | method_description: | |
| 3928 | The plugin issues a periodic HTTP request (GET by default, POST optionally) to |
| 3929 | your configured URL with the headers you specify, parses the response as JSON, |
| 3930 | runs your `transform` jq expression (compiled by the |
| 3931 | [jaq](https://github.com/01mf02/jaq) library) over the parsed body, and merges |
| 3932 | the resulting per-prefix rows into the network-attributes trie. |
| 3933 | |
| 3934 | The same flow applies to every JSON-over-HTTP source; the integration-specific |
| 3935 | part is the URL and the `transform` expression. |
| 3936 | supported_platforms: |
| 3937 | include: |
| 3938 | - Linux |
| 3939 | exclude: [] |
| 3940 | multi_instance: true |
| 3941 | additional_permissions: |
| 3942 | description: "" |
| 3943 | default_behavior: |
| 3944 | auto_detection: |
| 3945 | description: "Disabled by default. Add a named entry under `enrichment.network_sources` for each IPAM/CMDB/endpoint you want to import." |
| 3946 | limits: |
| 3947 | description: "Resource use scales with response size, transform complexity, refresh interval, and the number of emitted prefixes. Empty transform output is treated as a fetch failure." |
| 3948 | performance_impact: |
| 3949 | description: "One HTTP request per refresh interval plus a jq transform over the response. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records." |
| 3950 | setup: |
| 3951 | prerequisites: |
| 3952 | list: |
| 3953 | - title: An HTTP/HTTPS endpoint returning JSON |
| 3954 | description: | |
| 3955 | The endpoint must respond with a parseable JSON document (the plugin sets |
| 3956 | `Accept: application/json`). Only GET and POST are accepted as request |
| 3957 | methods. There is no pagination, no |
| 3958 | cursor following, no `Link: rel=next` handling -- the fetch is one-shot |
| 3959 | per cycle. If your IPAM paginates, expose a bulk endpoint or wrap it in a |
| 3960 | server-side aggregator that returns the full list at one URL. |
| 3961 | - title: Authentication via headers |
| 3962 | description: | |
| 3963 | The plugin has no built-in OAuth flow, basic-auth helper, or token |
| 3964 | refresh. Whatever the API needs (bearer tokens, API keys, custom header |
| 3965 | names, basic-auth realms) goes into the `headers:` map. The header map is |
| 3966 | an arbitrary string-to-string mapping, so any single-shot scheme works. |
| 3967 | For short-lived tokens, refresh them outside Netdata and reload the |
| 3968 | plugin config. |
| 3969 | - title: A POST endpoint must accept an empty body |
| 3970 | description: | |
| 3971 | When `method: POST` is configured, the plugin sends the request with the |
| 3972 | configured headers but **no request body**. If your CMDB requires a JSON query body to return prefixes, |
| 3973 | wrap it server-side with an endpoint that accepts GET (or POST with no |
| 3974 | body) and returns the full prefix set. |
| 3975 | configuration: |
| 3976 | file: |
| 3977 | name: netflow.yaml |
| 3978 | options: |
| 3979 | description: | |
| 3980 | Add a named entry under `enrichment.network_sources`. Unknown keys cause a |
| 3981 | config error. |
| 3982 | folding: |
| 3983 | title: Config options |
| 3984 | enabled: true |
| 3985 | list: |
| 3986 | - name: url |
| 3987 | description: HTTP/HTTPS endpoint. Required (a non-empty URL is enforced by validation). |
| 3988 | default_value: "" |
| 3989 | required: true |
| 3990 | - name: method |
| 3991 | description: HTTP method. Only `GET` and `POST` are accepted; anything else fails validation. Note that `POST` is sent with no request body. |
| 3992 | default_value: "GET" |
| 3993 | required: false |
| 3994 | - name: headers |
| 3995 | description: | |
| 3996 | Map of additional HTTP request headers. Use this for any authentication |
| 3997 | scheme (`Authorization: Bearer ...`, `token: ...`, custom API-key |
| 3998 | headers, basic-auth realms encoded explicitly, etc.). Values are |
| 3999 | passed through verbatim. |
| 4000 | default_value: "{}" |
| 4001 | required: false |
| 4002 | - name: interval |
| 4003 | description: Refresh cadence. Values below 60 seconds are accepted, but refreshes still run no faster than once per minute. Pick the value that matches how often your IPAM actually changes (5-15 minutes for a curated CMDB; daily for slow-moving prefix lists). |
| 4004 | default_value: "60s" |
| 4005 | required: false |
| 4006 | - name: timeout |
| 4007 | description: Per-request timeout. Must be greater than 0. |
| 4008 | default_value: "60s" |
| 4009 | required: false |
| 4010 | - name: proxy |
| 4011 | description: Whether to honor the system HTTP/HTTPS proxy environment variables. Set to `false` to bypass the proxy for this source (useful when the IPAM is on the internal network and the proxy is for outbound traffic only). |
| 4012 | default_value: "true" |
| 4013 | required: false |
| 4014 | - name: tls.enable |
| 4015 | description: Enables custom TLS settings (custom CA bundle, mTLS client certificate). Must be `true` whenever any of `tls.ca_file`, `tls.cert_file`, `tls.key_file` is set. |
| 4016 | default_value: "false" |
| 4017 | required: false |
| 4018 | - name: tls.verify |
| 4019 | description: Certificate verification toggle. **Setting this to `false` is rejected by validation** -- network-identity data flows directly into security-relevant enrichment, so MITM-able responses are not allowed. Use `tls.ca_file` to trust internal CAs instead. |
| 4020 | default_value: "true" |
| 4021 | required: false |
| 4022 | - name: tls.skip_verify |
| 4023 | description: | |
| 4024 | Legacy alias for the same intent as `tls.verify: false`. **Setting this |
| 4025 | to `true` is rejected by validation.** Use `tls.ca_file`. |
| 4026 | default_value: "false" |
| 4027 | required: false |
| 4028 | - name: tls.ca_file |
| 4029 | description: PEM file with the CA bundle to trust for this endpoint (instead of, or in addition to, the system roots). The recommended way to talk to an internal IPAM behind your own PKI. |
| 4030 | default_value: "" |
| 4031 | required: false |
| 4032 | - name: tls.cert_file |
| 4033 | description: PEM file with the client certificate, for mTLS-protected endpoints. |
| 4034 | default_value: "" |
| 4035 | required: false |
| 4036 | - name: tls.key_file |
| 4037 | description: PEM file with the client private key. Required only when the key is in a separate file from the certificate; if omitted, the cert file is reused. `tls.cert_file` must also be set whenever `tls.key_file` is set. |
| 4038 | default_value: "" |
| 4039 | required: false |
| 4040 | - name: transform |
| 4041 | description: | |
| 4042 | jq expression compiled by [jaq](https://github.com/01mf02/jaq). Receives the entire parsed JSON body once and must produce a **stream of objects**, where each object has at minimum a `prefix` field (a CIDR string -- `IPv4/len` or `IPv6/len`) plus any of the optional attribute fields: `name`, `role`, `site`, `region`, `country`, `state`, `city`, `tenant`, `asn`, `asn_name`. Any field not produced is treated as empty. The transform compiles at startup -- a syntax error fails the config load. An empty stream at runtime is treated as a fetch failure and triggers backoff. |
| 4043 | default_value: "." |
| 4044 | required: true |
| 4045 | examples: |
| 4046 | folding: |
| 4047 | title: Config |
| 4048 | enabled: true |
| 4049 | list: |
| 4050 | - name: Flat list with `{prefix, name, env}` |
| 4051 | folding: |
| 4052 | enabled: false |
| 4053 | description: | |
| 4054 | The simplest IPAM shape: a top-level array of subnets, each with a CIDR |
| 4055 | string and a few labels. Maps `env` -> tenant and the `name` -> NET_NAME. |
| 4056 | config: | |
| 4057 | enrichment: |
| 4058 | network_sources: |
| 4059 | custom_ipam: |
| 4060 | url: "https://ipam.corp.example/api/v1/prefixes" |
| 4061 | headers: |
| 4062 | Authorization: "Bearer abcdef0123456789" |
| 4063 | interval: 5m |
| 4064 | transform: | |
| 4065 | .[] | { |
| 4066 | prefix: .prefix, |
| 4067 | name: .name, |
| 4068 | tenant: .env |
| 4069 | } |
| 4070 | - name: phpIPAM with API token |
| 4071 | folding: |
| 4072 | enabled: true |
| 4073 | description: | |
| 4074 | phpIPAM exposes `/api/<APP>/subnets/`. Replace `<APP>` with your phpIPAM |
| 4075 | application name. phpIPAM splits the address into a base and a mask -- |
| 4076 | the transform reassembles them into a CIDR string. |
| 4077 | config: | |
| 4078 | enrichment: |
| 4079 | network_sources: |
| 4080 | phpipam: |
| 4081 | url: "https://ipam.example/api/netdata/subnets/" |
| 4082 | headers: |
| 4083 | token: "abcdef..." |
| 4084 | interval: 10m |
| 4085 | transform: | |
| 4086 | .data[] | { |
| 4087 | prefix: (.subnet + "/" + (.mask|tostring)), |
| 4088 | name: .description, |
| 4089 | tenant: (.custom_tenant // ""), |
| 4090 | site: (.location.name // "") |
| 4091 | } |
| 4092 | - name: Hierarchical CMDB export |
| 4093 | folding: |
| 4094 | enabled: true |
| 4095 | description: | |
| 4096 | A CMDB that returns nested objects -- top-level sites, each with a list |
| 4097 | of vlans, each with a list of prefixes. Flattens to one row per prefix, |
| 4098 | inheriting site and role labels from the parent. |
| 4099 | config: | |
| 4100 | enrichment: |
| 4101 | network_sources: |
| 4102 | cmdb: |
| 4103 | url: "https://cmdb.example/export/networks.json" |
| 4104 | headers: |
| 4105 | Authorization: "Bearer <CMDB_TOKEN>" |
| 4106 | interval: 15m |
| 4107 | transform: | |
| 4108 | .sites[] as $site |
| 4109 | | $site.vlans[] as $vlan |
| 4110 | | $vlan.prefixes[] |
| 4111 | | { |
| 4112 | prefix: .cidr, |
| 4113 | name: .label, |
| 4114 | site: $site.name, |
| 4115 | region: $site.region, |
| 4116 | role: $vlan.role, |
| 4117 | tenant: $vlan.tenant |
| 4118 | } |
| 4119 | - name: Internal IPAM behind mTLS |
| 4120 | folding: |
| 4121 | enabled: true |
| 4122 | description: | |
| 4123 | When the IPAM is fronted by your internal PKI and requires a client |
| 4124 | certificate. `tls.enable: true` activates the custom TLS path; `tls.verify` |
| 4125 | must remain true (cannot be disabled). |
| 4126 | config: | |
| 4127 | enrichment: |
| 4128 | network_sources: |
| 4129 | corp_ipam: |
| 4130 | url: "https://ipam.corp/api/networks" |
| 4131 | tls: |
| 4132 | enable: true |
| 4133 | ca_file: /etc/netdata/ssl/corp-ca.pem |
| 4134 | cert_file: /etc/netdata/ssl/netdata.crt |
| 4135 | key_file: /etc/netdata/ssl/netdata.key |
| 4136 | interval: 10m |
| 4137 | transform: | |
| 4138 | .[] | { |
| 4139 | prefix: .cidr, |
| 4140 | name: .label, |
| 4141 | tenant: .tenant |
| 4142 | } |
| 4143 | - name: Source with the system proxy bypassed |
| 4144 | folding: |
| 4145 | enabled: true |
| 4146 | description: | |
| 4147 | When the agent host has an outbound HTTP proxy configured but the IPAM |
| 4148 | lives on the internal network and should be reached directly. |
| 4149 | config: | |
| 4150 | enrichment: |
| 4151 | network_sources: |
| 4152 | internal_ipam: |
| 4153 | url: "https://ipam.internal/api/networks" |
| 4154 | proxy: false |
| 4155 | headers: |
| 4156 | Authorization: "Bearer ..." |
| 4157 | interval: 5m |
| 4158 | transform: | |
| 4159 | .[] | { prefix: .cidr, name: .label } |
| 4160 | troubleshooting: |
| 4161 | problems: |
| 4162 | list: |
| 4163 | - name: Endpoint requires pagination |
| 4164 | description: | |
| 4165 | The plugin does not paginate. Either raise the page size in the URL to |
| 4166 | cover your full inventory, or wrap the endpoint with a server-side |
| 4167 | aggregator that returns all results at one URL. There is no built-in |
| 4168 | `Link: rel=next` follower. |
| 4169 | - name: POST endpoint requires a request body |
| 4170 | description: | |
| 4171 | The plugin's POST request is sent with no body. If your CMDB requires a |
| 4172 | JSON query body to return prefixes, wrap it server-side with an endpoint |
| 4173 | that accepts GET (or accepts POST with no body) and returns the full prefix |
| 4174 | set. |
| 4175 | - name: TLS verification cannot be disabled |
| 4176 | description: | |
| 4177 | `tls.verify: false` and `tls.skip_verify: true` are both rejected during |
| 4178 | configuration validation. Use `tls.ca_file` to trust internal CAs. This is |
| 4179 | deliberate -- network-identity rows feed enrichment used in security |
| 4180 | investigations and capacity decisions, where silently accepting MITM-able |
| 4181 | responses would corrupt every downstream analysis. |
| 4182 | - name: Empty result back-off |
| 4183 | description: | |
| 4184 | An empty stream from the jq transform is treated as a fetch failure. |
| 4185 | The source then backs |
| 4186 | off exponentially -- starting at `interval / 10` (floor 1s), doubling on |
| 4187 | each consecutive failure, and capped at the regular `interval`. On the |
| 4188 | next successful non-empty fetch the cadence resets to `interval`. If your |
| 4189 | IPAM legitimately has no prefixes (a quiet state), have the upstream |
| 4190 | return at least one synthetic prefix so the source does not back off. |
| 4191 | - name: Refresh appears slower than configured |
| 4192 | description: | |
| 4193 | The fetch loop floors the configured `interval` at 60 seconds. Configuring |
| 4194 | `interval: 5s` does not produce a 5-second loop -- it produces a 60-second |
| 4195 | loop. Pick a value at or above 60s that matches how often your IPAM |
| 4196 | actually changes (5-15 minutes is typical for a curated CMDB). |
| 4197 | - name: Unknown config keys cause errors |
| 4198 | description: | |
| 4199 | The config struct uses `deny_unknown_fields`. Typos like `headres:` or a |
| 4200 | non-existent option fail config load with a parse error rather than being |
| 4201 | silently ignored. Check the `enrichment.network_sources.<name>` schema |
| 4202 | listed under "Config options". |
| 4203 | - name: JSON parse errors are silent in the dashboard |
| 4204 | description: | |
| 4205 | Decode failures (HTTP error, JSON parse error, jq runtime error, schema |
| 4206 | mismatch on `prefix`) are logged but do not surface in the dashboard. |
| 4207 | Watch the Netdata journal for warnings: |
| 4208 | `journalctl --namespace netdata | grep network_sources`. |
| 4209 | - name: Prefer explicit authorization headers over URL credentials |
| 4210 | description: | |
| 4211 | URLs with embedded credentials (`https://user:pass@host`) are converted to |
| 4212 | HTTP Basic authentication by the HTTP client. Prefer `headers:` for clarity |
| 4213 | and to avoid storing credentials in URLs -- e.g. |
| 4214 | `headers: { Authorization: "Basic dXNlcjpwYXNz" }` for HTTP basic-auth. |
| 4215 | alerts: [] |
| 4216 | metrics: |
| 4217 | folding: |
| 4218 | title: Metrics |
| 4219 | enabled: false |
| 4220 | description: | |
| 4221 | Tags flow records with whatever labels your jq transform emits -- typically |
| 4222 | `*_NET_NAME`, `*_NET_TENANT`, `*_NET_SITE`, `*_NET_ROLE`, `*_NET_REGION`. Verify |
| 4223 | on the Network Flows view via the `*_NET_*` columns. |
| 4224 | availability: [] |
| 4225 | scopes: [] |
| 4226 | - meta: |
| 4227 | plugin_name: netflow-plugin |
| 4228 | module_name: static-metadata |
| 4229 | monitored_instance: |
| 4230 | name: Static Metadata |
| 4231 | link: https://learn.netdata.cloud/docs/network-flows/enrichment |
| 4232 | categories: |
| 4233 | - flows.enrichment-methods |
| 4234 | icon_filename: network-wired.svg |
| 4235 | keywords: |
| 4236 | - static metadata |
| 4237 | - exporters |
| 4238 | - interfaces |
| 4239 | - networks |
| 4240 | - cidr |
| 4241 | - sampling rate override |
| 4242 | - asn override |
| 4243 | - country override |
| 4244 | - tenant |
| 4245 | - site |
| 4246 | - region |
| 4247 | - role |
| 4248 | related_resources: |
| 4249 | integrations: |
| 4250 | list: |
| 4251 | - plugin_name: netflow-plugin |
| 4252 | module_name: generic-ipam |
| 4253 | - plugin_name: netflow-plugin |
| 4254 | module_name: netbox |
| 4255 | - plugin_name: netflow-plugin |
| 4256 | module_name: dbip |
| 4257 | info_provided_to_referring_integrations: |
| 4258 | description: "" |
| 4259 | overview: |
| 4260 | data_collection: |
| 4261 | metrics_description: | |
| 4262 | Annotate network flows with operator-defined labels for exporters, interfaces, |
| 4263 | networks, and sampling rates. Static Metadata is a YAML-defined enrichment source |
| 4264 | that lives entirely in `netflow.yaml`. Use it when there is no IPAM, no CMDB, and |
| 4265 | no MMDB to query -- you simply know your own networks, your own routers, and what |
| 4266 | the interfaces on those routers do, and you want those labels visible on the |
| 4267 | Network Flows view. |
| 4268 | |
| 4269 | Three independent surfaces are configured under this card, each populating a |
| 4270 | different set of flow-record fields: |
| 4271 | |
| 4272 | - `enrichment.metadata_static.exporters.<ip-or-cidr>` -- per-exporter labels |
| 4273 | (matched against the source IP of the UDP datagram) and per-interface labels |
| 4274 | (matched against the ifIndex from the flow record). |
| 4275 | - `enrichment.networks.<cidr>` -- labels for arbitrary CIDR blocks (your own |
| 4276 | corp ranges, customer ranges, public blocks you operate). Matched against the |
| 4277 | flow's source and destination IPs. |
| 4278 | - `enrichment.override_sampling_rate.<cidr>` -- per-prefix sampling-rate |
| 4279 | substitution for exporters that do not communicate their rate (e.g. NetFlow |
| 4280 | v7, which has no sampling field, or a v9 exporter that never sends a Sampling |
| 4281 | Options Template). |
| 4282 | |
| 4283 | Fields populated: |
| 4284 | |
| 4285 | - From `metadata_static.exporters` (per-exporter): |
| 4286 | `EXPORTER_NAME`, `EXPORTER_GROUP`, `EXPORTER_ROLE`, `EXPORTER_SITE`, |
| 4287 | `EXPORTER_REGION`, `EXPORTER_TENANT`. |
| 4288 | - From `metadata_static.exporters.if_indexes` (per-interface): |
| 4289 | `IN_IF_NAME` / `OUT_IF_NAME`, `IN_IF_DESCRIPTION` / `OUT_IF_DESCRIPTION`, |
| 4290 | `IN_IF_SPEED` / `OUT_IF_SPEED` (in **bits per second**), `IN_IF_PROVIDER` / |
| 4291 | `OUT_IF_PROVIDER`, `IN_IF_CONNECTIVITY` / `OUT_IF_CONNECTIVITY`, |
| 4292 | `IN_IF_BOUNDARY` / `OUT_IF_BOUNDARY` (`1` = external, `2` = internal, |
| 4293 | `0`/omitted = removed from output). |
| 4294 | - From `enrichment.networks` (per-CIDR): |
| 4295 | `SRC_NET_NAME` / `DST_NET_NAME`, `SRC_NET_ROLE` / `DST_NET_ROLE`, |
| 4296 | `SRC_NET_SITE` / `DST_NET_SITE`, `SRC_NET_REGION` / `DST_NET_REGION`, |
| 4297 | `SRC_NET_TENANT` / `DST_NET_TENANT`, plus overrides for the GeoIP-derived |
| 4298 | `SRC_COUNTRY` / `DST_COUNTRY`, `SRC_GEO_STATE` / `DST_GEO_STATE`, |
| 4299 | `SRC_GEO_CITY` / `DST_GEO_CITY`, `SRC_GEO_LATITUDE` / `DST_GEO_LATITUDE`, |
| 4300 | `SRC_GEO_LONGITUDE` / `DST_GEO_LONGITUDE`. Setting `asn:` on a `networks` |
| 4301 | entry overrides the AS *number* via the same merge path; the AS *name* |
| 4302 | still comes from the ASN database (see "Composition with dynamic sources" |
| 4303 | below). |
| 4304 | - From `override_sampling_rate` (longest-prefix match against the exporter IP): |
| 4305 | `SAMPLING_RATE` is set unconditionally when an override prefix matches. This |
| 4306 | is different from `default_sampling_rate`, which only applies when the flow |
| 4307 | does not already carry a rate. |
| 4308 | |
| 4309 | For the cross-cutting Enrichment concept (provider chains, merge order rules, |
| 4310 | how static composes with dynamic sources, the static-blocks-classifiers |
| 4311 | interaction, and shared failure modes), see |
| 4312 | [Enrichment](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 4313 | method_description: | |
| 4314 | Edit `netflow.yaml`, restart the plugin, and the YAML-defined data is loaded |
| 4315 | into the same in-memory tries the GeoIP, IPAM, and BGP enrichment paths read |
| 4316 | from. There is no network access, no file watching, and no hot reload -- |
| 4317 | changes take effect on the next plugin startup. |
| 4318 | supported_platforms: |
| 4319 | include: |
| 4320 | - Linux |
| 4321 | exclude: [] |
| 4322 | multi_instance: false |
| 4323 | additional_permissions: |
| 4324 | description: "" |
| 4325 | default_behavior: |
| 4326 | auto_detection: |
| 4327 | description: "Disabled by default. Add entries under `enrichment.metadata_static`, `enrichment.networks`, and / or `enrichment.override_sampling_rate` to populate it." |
| 4328 | limits: |
| 4329 | description: "Resource use scales with the number of exporter, interface, network, and sampling-override entries. Keep CIDR ranges and interface maps specific enough to match the routers you actually export." |
| 4330 | performance_impact: |
| 4331 | description: "Static lookups are local map/trie lookups during enrichment. Cost is usually small compared with flow decode and journal writes, but very large maps add memory and lookup work." |
| 4332 | setup: |
| 4333 | prerequisites: |
| 4334 | list: |
| 4335 | - title: An understanding of your own topology |
| 4336 | description: | |
| 4337 | You need to know which IP addresses your routers / switches / firewalls |
| 4338 | export from, the integer ifIndex values they use for each interface, and |
| 4339 | the CIDR ranges that belong to each of your sites / tenants / roles. None |
| 4340 | of this is auto-discovered -- this card is the "I will tell you" path. |
| 4341 | For sources that auto-discover the same data, see Generic IPAM, NetBox, |
| 4342 | and the cloud IP-range cards. |
| 4343 | - title: Persistent ifIndex values |
| 4344 | description: | |
| 4345 | `if_indexes` keys are the integer ifIndex the router puts in the flow |
| 4346 | record. Some platforms reassign ifIndex on line-card reseat or stack |
| 4347 | rebuild. After hardware changes, audit the labels -- a stale ifIndex |
| 4348 | entry silently no longer applies. The `default` interface block is used |
| 4349 | instead, or no labels are written when `skip_missing_interfaces: true` is set. |
| 4350 | configuration: |
| 4351 | file: |
| 4352 | name: netflow.yaml |
| 4353 | options: |
| 4354 | description: | |
| 4355 | All keys live under `enrichment:` in `netflow.yaml`. Unknown keys fail config |
| 4356 | load with a parse error. |
| 4357 | folding: |
| 4358 | title: Config options |
| 4359 | enabled: true |
| 4360 | list: |
| 4361 | - name: enrichment.metadata_static.exporters |
| 4362 | description: | |
| 4363 | Map keyed by exporter IP or CIDR. Longest-prefix match wins. A `/32` |
| 4364 | key and a bare IP are equivalent. Each entry holds exporter-level |
| 4365 | labels (`name`, `group`, `role`, `site`, `region`, `tenant`), an |
| 4366 | optional `default` interface block, an `if_indexes` map keyed by |
| 4367 | integer ifIndex, and `skip_missing_interfaces`. |
| 4368 | default_value: "{}" |
| 4369 | required: false |
| 4370 | - name: enrichment.metadata_static.exporters.<key>.if_indexes |
| 4371 | description: | |
| 4372 | Map keyed by integer ifIndex (the value the router puts in the flow |
| 4373 | record). Each entry holds `name`, `description`, `speed` (bits per |
| 4374 | second), `provider`, `connectivity`, and `boundary`. |
| 4375 | default_value: "{}" |
| 4376 | required: false |
| 4377 | - name: enrichment.metadata_static.exporters.<key>.default |
| 4378 | description: | |
| 4379 | Interface block applied to flow records whose ifIndex is not present |
| 4380 | in `if_indexes`. Same shape as an `if_indexes` entry. Ignored when |
| 4381 | `skip_missing_interfaces: true` is set. |
| 4382 | default_value: "{}" |
| 4383 | required: false |
| 4384 | - name: enrichment.metadata_static.exporters.<key>.skip_missing_interfaces |
| 4385 | description: | |
| 4386 | When `true`, flow records whose ifIndex is not in `if_indexes` get no |
| 4387 | interface labels at all (not even from `default`). Useful when you only |
| 4388 | care about a known set of WAN interfaces and want unknown ones to stay |
| 4389 | blank instead of inheriting a placeholder. |
| 4390 | default_value: "false" |
| 4391 | required: false |
| 4392 | - name: enrichment.metadata_static.exporters.<key>.if_indexes.<n>.boundary |
| 4393 | description: | |
| 4394 | Interface boundary marker. Accepts the integers `0` (undefined), |
| 4395 | `1` (external -- faces the outside world: Internet, peer, transit), |
| 4396 | `2` (internal -- faces your own infrastructure), or the case-insensitive |
| 4397 | strings `"undefined"`, `"external"`, `"internal"`. Any other value fails config load. Filtering for `IN_IF_BOUNDARY=1` |
| 4398 | cleanly gives you "traffic that arrived from outside". |
| 4399 | default_value: "0" |
| 4400 | required: false |
| 4401 | - name: enrichment.metadata_static.exporters.<key>.if_indexes.<n>.speed |
| 4402 | description: | |
| 4403 | Interface speed in **bits per second**. A 1 Gbps interface is |
| 4404 | `1000000000`, not `1000` and not `1000000`. Operators thinking in |
| 4405 | megabits get the value wrong by a factor of 1000 to 1000000. |
| 4406 | A `0` value means "not set" and removes the field from the output. |
| 4407 | default_value: "0" |
| 4408 | required: false |
| 4409 | - name: enrichment.networks |
| 4410 | description: | |
| 4411 | Map keyed by CIDR. Longest-prefix match contributes the most-specific |
| 4412 | fields; less-specific containing prefixes contribute their non-empty |
| 4413 | fields too. The same merge rule is used by `network_sources`. Each value is either a string |
| 4414 | (shorthand for `name:`) or a map with `name`, `role`, `site`, `region`, |
| 4415 | `country`, `state`, `city`, `latitude`, `longitude`, `tenant`, `asn`. |
| 4416 | default_value: "{}" |
| 4417 | required: false |
| 4418 | - name: enrichment.networks.<cidr>.asn |
| 4419 | description: | |
| 4420 | Forces the AS *number* for traffic in this prefix, overriding whatever |
| 4421 | the `asn_providers` chain computed. The AS *name* is still resolved |
| 4422 | from the ASN MMDB -- there is no `asn_name` config field. See the ASN |
| 4423 | section of [Enrichment](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 4424 | default_value: "0" |
| 4425 | required: false |
| 4426 | - name: enrichment.networks.<cidr>.latitude / longitude |
| 4427 | description: | |
| 4428 | Per-CIDR coordinate override. Out-of-range values (latitude not in |
| 4429 | [-90, 90] or longitude not in [-180, 180]) and non-finite values are |
| 4430 | silently coerced to empty strings -- the field is dropped, no error. |
| 4431 | Validate input out of band if your data matters. |
| 4432 | default_value: "null" |
| 4433 | required: false |
| 4434 | - name: enrichment.default_sampling_rate |
| 4435 | description: | |
| 4436 | Global fallback applied only when the flow record does not already |
| 4437 | carry a sampling rate. Either a single integer or a CIDR-keyed map |
| 4438 | (longest-prefix match against the exporter IP). |
| 4439 | default_value: "0" |
| 4440 | required: false |
| 4441 | - name: enrichment.override_sampling_rate |
| 4442 | description: | |
| 4443 | Per-exporter substitution that **always** wins when its prefix matches |
| 4444 | the exporter IP, regardless of what the flow record carried. Either a |
| 4445 | single integer or a CIDR-keyed map. |
| 4446 | default_value: "{}" |
| 4447 | required: false |
| 4448 | examples: |
| 4449 | folding: |
| 4450 | title: Config |
| 4451 | enabled: true |
| 4452 | list: |
| 4453 | - name: Naming exporters and their interfaces |
| 4454 | folding: |
| 4455 | enabled: false |
| 4456 | description: | |
| 4457 | One edge router and one spine. The `default` block applies to any |
| 4458 | ifIndex not listed under `if_indexes` -- handy when you only care about |
| 4459 | a few WAN ports and want everything else to inherit a placeholder. |
| 4460 | config: | |
| 4461 | enrichment: |
| 4462 | metadata_static: |
| 4463 | exporters: |
| 4464 | 192.0.2.10: # bare IP == /32 |
| 4465 | name: edge-router-1 |
| 4466 | site: par1 |
| 4467 | region: eu-west |
| 4468 | role: edge |
| 4469 | tenant: tenant-a |
| 4470 | default: |
| 4471 | description: unclassified port |
| 4472 | if_indexes: |
| 4473 | 1: |
| 4474 | name: Gi0/0/1 |
| 4475 | description: uplink to ISP-A |
| 4476 | speed: 10000000000 # 10 Gbps in bits per second |
| 4477 | provider: isp-a |
| 4478 | connectivity: transit |
| 4479 | boundary: external |
| 4480 | 2: |
| 4481 | name: Gi0/0/2 |
| 4482 | description: LAN core |
| 4483 | speed: 1000000000 |
| 4484 | connectivity: lan |
| 4485 | boundary: internal |
| 4486 | 198.51.100.0/24: # all routers in this subnet |
| 4487 | site: dc-fra1 |
| 4488 | region: eu-central |
| 4489 | role: spine |
| 4490 | default: |
| 4491 | connectivity: lan |
| 4492 | boundary: internal |
| 4493 | - name: Tagging your own networks |
| 4494 | folding: |
| 4495 | enabled: true |
| 4496 | description: | |
| 4497 | The shorthand form (`203.0.113.0/24: transit-a`) sets only the `name`. |
| 4498 | The map form takes any subset of the schema fields. Less-specific |
| 4499 | supernets contribute non-empty fields to more-specific subnets via |
| 4500 | ascending-prefix-length merge -- declare RFC1918 once at `/8` and |
| 4501 | a `/24` inherits everything you did not override. |
| 4502 | config: | |
| 4503 | enrichment: |
| 4504 | networks: |
| 4505 | 10.0.0.0/8: |
| 4506 | name: corp-internal |
| 4507 | role: internal |
| 4508 | tenant: tenant-a |
| 4509 | 198.51.100.0/24: # a public block you operate |
| 4510 | name: customer-acme |
| 4511 | role: customer |
| 4512 | site: par1 |
| 4513 | country: FR |
| 4514 | city: Paris |
| 4515 | latitude: 48.8566 |
| 4516 | longitude: 2.3522 |
| 4517 | asn: 64500 |
| 4518 | 203.0.113.0/24: transit-a # shorthand: name only |
| 4519 | - name: Skipping interfaces you have not labelled |
| 4520 | folding: |
| 4521 | enabled: true |
| 4522 | description: | |
| 4523 | With `skip_missing_interfaces: true`, an ifIndex that is not in |
| 4524 | `if_indexes` produces no interface labels at all -- the `default` |
| 4525 | block is ignored. Useful when you want clearly-blank values for |
| 4526 | unmodelled interfaces instead of a placeholder description. |
| 4527 | config: | |
| 4528 | enrichment: |
| 4529 | metadata_static: |
| 4530 | exporters: |
| 4531 | 192.0.2.10: |
| 4532 | name: edge-router-1 |
| 4533 | skip_missing_interfaces: true |
| 4534 | if_indexes: |
| 4535 | 1: |
| 4536 | name: Gi0/0/1 |
| 4537 | description: uplink to ISP-A |
| 4538 | connectivity: transit |
| 4539 | boundary: external |
| 4540 | - name: Sampling rate override for old or silent exporters |
| 4541 | folding: |
| 4542 | enabled: true |
| 4543 | description: | |
| 4544 | NetFlow v7 has no sampling field; some v9 exporters never send a |
| 4545 | Sampling Options Template. Without an override, the bytes/packets |
| 4546 | stored in the journal are exporter-side **sampled** counters with |
| 4547 | no scaling factor. `override_sampling_rate` always wins; |
| 4548 | `default_sampling_rate` is the fallback used only when the flow |
| 4549 | did not carry a rate. |
| 4550 | config: | |
| 4551 | enrichment: |
| 4552 | default_sampling_rate: 1 # global fallback |
| 4553 | override_sampling_rate: |
| 4554 | 10.1.0.0/16: 1024 # an old NetFlow v7 stack |
| 4555 | 192.0.2.10/32: 4096 # one specific exporter |
| 4556 | troubleshooting: |
| 4557 | problems: |
| 4558 | list: |
| 4559 | - name: A more-specific entry inherits the wrong field from its supernet |
| 4560 | description: | |
| 4561 | `enrichment.networks` merges containing prefixes from least-specific to |
| 4562 | most-specific. Leaving a field blank on a `/24` does **not** clear the `/16`'s value |
| 4563 | for that field -- you must explicitly set the field on the more-specific |
| 4564 | entry to overwrite. The same merge rule applies to entries from |
| 4565 | `network_sources`, which interleave at the same prefix lengths. |
| 4566 | - name: GeoIP returns spurious data for an internal range |
| 4567 | description: | |
| 4568 | Until you declare your RFC1918 / RFC6598 / link-local ranges as |
| 4569 | `enrichment.networks` entries, the GeoIP-derived country / city / coord |
| 4570 | fields can pick up junk for those addresses. Adding a `networks` entry |
| 4571 | for the range overrides the GeoIP layer at the merge step. |
| 4572 | - name: ifIndex changed after a hardware swap |
| 4573 | description: | |
| 4574 | `if_indexes` keys are the numeric ifIndex sent in the flow record. A |
| 4575 | line-card reseat or stack rebuild can renumber the interfaces; the old |
| 4576 | ifIndex no longer matches and the per-interface block silently no longer |
| 4577 | applies. Audit after hardware changes. |
| 4578 | - name: speed shows up wrong by a factor of 1000 |
| 4579 | description: | |
| 4580 | `speed:` is in **bits per second**. `speed: 1000` means 1 kbps, not |
| 4581 | 1 Mbps. A 1 Gbps interface is `1000000000`. `speed: 0` means "not set" |
| 4582 | and removes the field from the output. |
| 4583 | - name: An interface block silently no longer applies |
| 4584 | description: | |
| 4585 | When the configured ifIndex is not present in the flow record, the |
| 4586 | `default` block is used instead -- unless `skip_missing_interfaces: true` |
| 4587 | is set, in which case no interface labels are written at all. If you expected your block to apply but the labels are blank, the |
| 4588 | router is sending a different ifIndex. |
| 4589 | - name: Coordinates dropped silently |
| 4590 | description: | |
| 4591 | Out-of-range latitude / longitude (`latitude: 91.5`) and non-finite |
| 4592 | values become empty strings without an error. The map quietly stops drawing the marker. Validate input externally if |
| 4593 | the data matters. |
| 4594 | - name: Static labels block the classifiers |
| 4595 | description: | |
| 4596 | When static metadata sets **any** of `group`, `role`, `site`, `region`, |
| 4597 | `tenant` for an exporter, the `exporter_classifiers` rule chain does not |
| 4598 | run for that exporter at all. The same is true for an interface: any of `provider`, `connectivity`, |
| 4599 | `boundary` set by static metadata short-circuits |
| 4600 | `interface_classifiers`. If |
| 4601 | you want classifiers to run on top of static metadata, drop the static |
| 4602 | fields they are supposed to set. |
| 4603 | - name: A typo fails config load |
| 4604 | description: | |
| 4605 | The schema is `deny_unknown_fields` at every level. A typo such as |
| 4606 | `if_index` (the canonical key is `if_indexes`; aliases `ifindexes`, |
| 4607 | `if-indexes` are accepted) or a misspelt attribute (`teannt:`) fails plugin |
| 4608 | start with a YAML parse error rather than being silently ignored. |
| 4609 | - name: Sampling override looks ignored |
| 4610 | description: | |
| 4611 | `override_sampling_rate` always wins when its prefix matches; if the |
| 4612 | field still looks unset, check that the **exporter IP** -- not the |
| 4613 | flow's source / destination IP -- falls under the configured prefix. |
| 4614 | `default_sampling_rate` is only consulted when the flow did not already |
| 4615 | carry a rate. |
| 4616 | - name: Changes do not take effect |
| 4617 | description: | |
| 4618 | Static metadata is loaded at plugin startup and there is no file-change |
| 4619 | watcher. Restart the plugin (or the agent) after editing |
| 4620 | `netflow.yaml`. |
| 4621 | alerts: [] |
| 4622 | metrics: |
| 4623 | folding: |
| 4624 | title: Metrics |
| 4625 | enabled: false |
| 4626 | description: | |
| 4627 | Tags flow records with operator-defined labels and (optionally) substitutes |
| 4628 | the sampling rate; produces no metrics of its own. Verify on the Network |
| 4629 | Flows tab via the `EXPORTER_*`, `IN_IF_*`, `OUT_IF_*`, `*_NET_*`, |
| 4630 | `*_COUNTRY`, `*_GEO_*`, and `SAMPLING_RATE` columns. |
| 4631 | availability: [] |
| 4632 | scopes: [] |
| 4633 | - meta: |
| 4634 | plugin_name: netflow-plugin |
| 4635 | module_name: classifiers |
| 4636 | monitored_instance: |
| 4637 | name: Classifiers |
| 4638 | link: https://learn.netdata.cloud/docs/network-flows/enrichment |
| 4639 | categories: |
| 4640 | - flows.enrichment-methods |
| 4641 | icon_filename: network-wired.svg |
| 4642 | keywords: |
| 4643 | - classifiers |
| 4644 | - rules |
| 4645 | - akvorado |
| 4646 | - expression |
| 4647 | - exporter |
| 4648 | - interface |
| 4649 | - boundary |
| 4650 | - connectivity |
| 4651 | - provider |
| 4652 | - tagging |
| 4653 | related_resources: |
| 4654 | integrations: |
| 4655 | list: |
| 4656 | - plugin_name: netflow-plugin |
| 4657 | module_name: static-metadata |
| 4658 | - plugin_name: netflow-plugin |
| 4659 | module_name: generic-ipam |
| 4660 | - plugin_name: netflow-plugin |
| 4661 | module_name: netbox |
| 4662 | info_provided_to_referring_integrations: |
| 4663 | description: "" |
| 4664 | overview: |
| 4665 | data_collection: |
| 4666 | metrics_description: | |
| 4667 | Annotate network flows with exporter and interface labels derived from reusable |
| 4668 | classification rules. Where [static metadata](https://learn.netdata.cloud/docs/network-flows/enrichment) |
| 4669 | forces you to enumerate every exporter and every ifIndex by hand, classifiers |
| 4670 | let you express the network design once -- "anything matching `^edge-` is the |
| 4671 | edge tier", "any interface with `BACKBONE-LUMEN` in its description is on |
| 4672 | Lumen", "any interface at 100Gbps is a core uplink" -- and apply that labelling |
| 4673 | across the whole flow stream. |
| 4674 | |
| 4675 | The plugin ships two rule lists, evaluated in YAML order: |
| 4676 | |
| 4677 | - `enrichment.exporter_classifiers` -- runs once per exporter (cached). Sees the |
| 4678 | exporter's IP and friendly name, and any classification slots already filled |
| 4679 | by static metadata or by earlier rules. Can set |
| 4680 | `EXPORTER_GROUP / ROLE / SITE / REGION / TENANT`. |
| 4681 | - `enrichment.interface_classifiers` -- runs once per `(exporter, interface)` |
| 4682 | pair, applied **twice per flow record** (once for the input interface, once |
| 4683 | for the output). Sees |
| 4684 | everything an exporter rule sees plus `Interface.Index / Name / Description / |
| 4685 | Speed / VLAN`. Can set `IN_IF_PROVIDER / OUT_IF_PROVIDER`, |
| 4686 | `IN_IF_CONNECTIVITY / OUT_IF_CONNECTIVITY`, `IN_IF_BOUNDARY / OUT_IF_BOUNDARY` |
| 4687 | (1=external, 2=internal), and override `IN_IF_NAME / DESCRIPTION` / |
| 4688 | `OUT_IF_NAME / DESCRIPTION`. |
| 4689 | |
| 4690 | The expression language is **Akvorado-compatible for the documented operators |
| 4691 | and actions**. It implements a subset of Akvorado's `expr-lang`-derived grammar. Akvorado rules |
| 4692 | using only equality, comparison, `in`, `contains`, `startsWith`, `endsWith`, |
| 4693 | `matches`, `&&`, `||`, `!`, parentheses, and the documented `Classify*` / |
| 4694 | `Reject` / `Format` actions will work; arithmetic, ternaries, lambdas, and |
| 4695 | arbitrary `expr-lang` features are not supported. |
| 4696 | |
| 4697 | Output values written by `Classify*` actions are **lowercased and stripped to |
| 4698 | ASCII alphanumerics + `.` + `+` + `-`** before they reach the flow record. So |
| 4699 | `ClassifyRegion("EU West")` becomes `euwest`. Use `SetName` / `SetDescription` |
| 4700 | when you want to preserve case and whitespace -- those write directly without |
| 4701 | normalisation. |
| 4702 | |
| 4703 | For the cross-cutting Enrichment concept (where classifiers sit in the merge |
| 4704 | order vs static metadata, GeoIP, IPAM, BGP routing), see |
| 4705 | [Enrichment](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 4706 | method_description: | |
| 4707 | Each rule is a single boolean expression; an action with no condition (e.g. |
| 4708 | `Classify("edge")` at top level) is treated as always-true and always fires. |
| 4709 | Rules are AND/OR-composed, so the typical shape is `condition && Classify*(...)`. |
| 4710 | The plugin evaluates the list top to bottom, **first-write-wins per slot**: |
| 4711 | once `EXPORTER_GROUP` is set, no later rule can change it. Order rules from |
| 4712 | most-specific to least-specific. |
| 4713 | |
| 4714 | Two short-circuit rules end the loop early. For exporter rules, the loop stops |
| 4715 | when `group + role + site + region + tenant` are all non-empty. For |
| 4716 | interface rules, the loop stops when `connectivity + provider + boundary` are |
| 4717 | all set. `SetName` / `SetDescription` / |
| 4718 | `Reject` do not contribute to short-circuit. |
| 4719 | |
| 4720 | A rule that throws at runtime (e.g. comparing a string with `>`) breaks out of |
| 4721 | the loop for that record and keeps whatever was set so far. Use `matches`, `startsWith`, or `contains` |
| 4722 | on string fields instead of `>` / `<` to avoid this. |
| 4723 | |
| 4724 | **Akvorado parity**: if `metadata_static` already filled any classification |
| 4725 | slot for the target, the matching classifier list does **not run** for that |
| 4726 | target -- operator-provided classification has priority and the rules cannot |
| 4727 | override it. Don't try to mix static and rule-based labelling on the same |
| 4728 | exporter or interface; pick one tool per target. |
| 4729 | |
| 4730 | Results are cached. The exporter cache keys on `ExporterInfo (ip + name)`. The |
| 4731 | interface cache keys on `(exporter, exporter_classification, interface)` -- so |
| 4732 | when the exporter's classification changes (for example after you push new |
| 4733 | static metadata and restart) the interface caches naturally invalidate. The |
| 4734 | cache TTL is `enrichment.classifier_cache_duration` (default 5 minutes). It is |
| 4735 | a last-access TTL so entries live as long as they're queried. |
| 4736 | supported_platforms: |
| 4737 | include: |
| 4738 | - Linux |
| 4739 | exclude: [] |
| 4740 | multi_instance: false |
| 4741 | additional_permissions: |
| 4742 | description: "" |
| 4743 | default_behavior: |
| 4744 | auto_detection: |
| 4745 | description: "Disabled by default. Both rule lists are empty; populate `enrichment.exporter_classifiers` and / or `enrichment.interface_classifiers` to enable." |
| 4746 | limits: |
| 4747 | description: "Resource use scales with rule count and the number of distinct exporters and interfaces. The classifier cache limits repeat evaluation for stable exporter/interface inventories." |
| 4748 | performance_impact: |
| 4749 | description: | |
| 4750 | Rules run at decode time, in the flow-pipeline hot path, so cost matters. |
| 4751 | The cache absorbs nearly all of it: per (exporter, interface) the rule list |
| 4752 | evaluates only on cache miss. Tune |
| 4753 | `enrichment.classifier_cache_duration` upwards (15-60 minutes) for very |
| 4754 | high-cardinality exporter / interface pools where the default 5 minutes |
| 4755 | still yields visible misses; tune downwards (30-60 seconds) when iterating |
| 4756 | on rule changes during a config session. |
| 4757 | setup: |
| 4758 | prerequisites: |
| 4759 | list: |
| 4760 | - title: Know what to classify |
| 4761 | description: | |
| 4762 | Classifiers shine when there is a pattern to match -- exporter naming |
| 4763 | conventions (`edge-...`, `core-...`), management-IP subnets per site, |
| 4764 | SNMP interface descriptions that follow a template (`BACKBONE-<carrier>`, |
| 4765 | `TRANSIT-...`, `IX-...`), or 100Gbps-equals-core conventions. If your |
| 4766 | fleet has no such pattern, [static metadata](https://learn.netdata.cloud/docs/network-flows/enrichment) |
| 4767 | is the better fit -- it lets you list each exporter and ifIndex by hand. |
| 4768 | - title: Configure interface metadata first if you want interface rules |
| 4769 | description: | |
| 4770 | The plugin does **not** poll SNMP itself, so `Interface.Name`, |
| 4771 | `Interface.Description`, and `Interface.Speed` are populated only from |
| 4772 | `enrichment.metadata_static` (the static-metadata integration card). If |
| 4773 | you have not configured `interfaces:` under `metadata_static.exporters`, |
| 4774 | those identifiers will be empty strings / zero, and any rule that |
| 4775 | matches against them will never fire. `Interface.Index` and |
| 4776 | `Interface.VLAN` come from the flow record itself and are always available. |
| 4777 | configuration: |
| 4778 | file: |
| 4779 | name: netflow.yaml |
| 4780 | options: |
| 4781 | description: | |
| 4782 | Both lists live under `enrichment:`. Each entry is a free-form string |
| 4783 | containing a single rule expression. The cache TTL is one global setting. |
| 4784 | folding: |
| 4785 | title: Config options |
| 4786 | enabled: true |
| 4787 | list: |
| 4788 | - name: enrichment.exporter_classifiers |
| 4789 | description: | |
| 4790 | Ordered list of rules applied per exporter. Each rule is a string |
| 4791 | expression. Available identifiers: `Exporter.IP`, `Exporter.Name`, |
| 4792 | `CurrentClassification.Group / .Role / .Site / .Region / .Tenant`. |
| 4793 | Available actions: `Classify` / `ClassifyGroup`, `ClassifyRole`, |
| 4794 | `ClassifySite`, `ClassifyRegion`, `ClassifyTenant`, plus the |
| 4795 | `*Regex(input, pattern, template)` variants of each, plus `Reject()`. |
| 4796 | Interface-only actions (`ClassifyProvider`, `ClassifyConnectivity`, |
| 4797 | `ClassifyExternal` / `ClassifyInternal`, `SetName`, `SetDescription`) |
| 4798 | fail at runtime if used here. |
| 4799 | default_value: "[]" |
| 4800 | required: false |
| 4801 | - name: enrichment.interface_classifiers |
| 4802 | description: | |
| 4803 | Ordered list of rules applied per `(exporter, interface)` pair. Sees |
| 4804 | everything an exporter rule sees, plus `Interface.Index`, |
| 4805 | `Interface.Name`, `Interface.Description`, `Interface.Speed` (bits per |
| 4806 | second), `Interface.VLAN`, and the per-interface |
| 4807 | `CurrentClassification.Connectivity / .Provider / .Boundary / .Name / |
| 4808 | .Description`. Available actions: `ClassifyProvider`, |
| 4809 | `ClassifyConnectivity`, `ClassifyExternal()`, `ClassifyInternal()`, |
| 4810 | `SetName`, `SetDescription`, `Reject()`, plus the `*Regex` variants of |
| 4811 | provider / connectivity. Exporter-only `Classify*` actions fail at |
| 4812 | runtime if used here. |
| 4813 | default_value: "[]" |
| 4814 | required: false |
| 4815 | - name: enrichment.classifier_cache_duration |
| 4816 | description: | |
| 4817 | Last-access TTL for both classifier caches (exporter and interface). |
| 4818 | Values below 1 second are rejected. The cache prunes opportunistically |
| 4819 | -- entries idle longer than the TTL are dropped on the next prune pass, |
| 4820 | capped at one prune every TTL or 30 seconds, whichever is smaller. |
| 4821 | Restart the plugin to clear caches outright when you change rules. |
| 4822 | default_value: "5m" |
| 4823 | required: false |
| 4824 | examples: |
| 4825 | folding: |
| 4826 | title: Config |
| 4827 | enabled: true |
| 4828 | list: |
| 4829 | - name: Exporter classification by name pattern |
| 4830 | folding: |
| 4831 | enabled: false |
| 4832 | description: | |
| 4833 | Tag exporters by the prefix of their friendly name -- the simplest and |
| 4834 | most common pattern. Falls back to a regex capture for the region code |
| 4835 | when the name encodes one. The final `Reject()` rule drops a test |
| 4836 | exporter from collection entirely. |
| 4837 | config: | |
| 4838 | enrichment: |
| 4839 | exporter_classifiers: |
| 4840 | # Group by name prefix. |
| 4841 | - 'Exporter.Name startsWith "edge-" && Classify("edge")' |
| 4842 | - 'Exporter.Name startsWith "core-" && Classify("core")' |
| 4843 | - 'Exporter.Name startsWith "agg-" && Classify("aggregation")' |
| 4844 | |
| 4845 | # Site by management-IP subnet. |
| 4846 | - 'Exporter.IP startsWith "10.1." && ClassifySite("ny-dc1")' |
| 4847 | - 'Exporter.IP startsWith "10.2." && ClassifySite("par-dc1")' |
| 4848 | |
| 4849 | # Region from a name suffix like "edge-fra-01" -> "fra". |
| 4850 | - 'ClassifyRegionRegex(Exporter.Name, "-([a-z]{3})-[0-9]+$", "$1")' |
| 4851 | |
| 4852 | # Drop a lab exporter entirely. |
| 4853 | - 'Exporter.IP startsWith "192.0.2." && Reject()' |
| 4854 | - name: Interface classification from SNMP descriptions |
| 4855 | folding: |
| 4856 | enabled: true |
| 4857 | description: | |
| 4858 | Encode the boundary, the provider, and the connectivity tier from the |
| 4859 | interface description that your network team already maintains. The |
| 4860 | `(?i)` regex flag is the Rust regex inline-case-insensitive prefix. |
| 4861 | config: | |
| 4862 | enrichment: |
| 4863 | interface_classifiers: |
| 4864 | # Provider tag from a description prefix. |
| 4865 | - 'Interface.Description startsWith "BACKBONE-LUMEN" && ClassifyProvider("Lumen")' |
| 4866 | - 'Interface.Description startsWith "BACKBONE-COGENT" && ClassifyProvider("Cogent")' |
| 4867 | - 'Interface.Description startsWith "BACKBONE-NTT" && ClassifyProvider("NTT")' |
| 4868 | |
| 4869 | # Transit links: external boundary + connectivity tag. |
| 4870 | - 'Interface.Description contains "TRANSIT" && ClassifyConnectivity("transit") && ClassifyExternal()' |
| 4871 | |
| 4872 | # Peering and IX -- case-insensitive regex. |
| 4873 | - 'Interface.Description matches "(?i)^(IX|peering)-.*" && ClassifyConnectivity("peering") && ClassifyExternal()' |
| 4874 | |
| 4875 | # Internal customer-facing access ports. |
| 4876 | - 'Interface.Description startsWith "CUSTOMER-" && ClassifyConnectivity("customer") && ClassifyInternal()' |
| 4877 | - name: Boundary inferred from interface speed |
| 4878 | folding: |
| 4879 | enabled: true |
| 4880 | description: | |
| 4881 | A pragmatic shorthand when descriptions are unreliable but speed is |
| 4882 | consistent. 100Gbps and faster interfaces are core, 10Gbps are |
| 4883 | aggregation, 1Gbps and slower are access. `Interface.Speed` is in bits |
| 4884 | per second -- numeric comparisons are safe. |
| 4885 | config: | |
| 4886 | enrichment: |
| 4887 | interface_classifiers: |
| 4888 | - 'Interface.Speed >= 100000000000 && ClassifyConnectivity("core")' |
| 4889 | - 'Interface.Speed >= 10000000000 && ClassifyConnectivity("aggregation")' |
| 4890 | - 'Interface.Speed > 0 && ClassifyConnectivity("access")' |
| 4891 | - name: Combining exporter context with interface rules |
| 4892 | folding: |
| 4893 | enabled: true |
| 4894 | description: | |
| 4895 | Interface rules see the exporter's already-resolved classification |
| 4896 | via `CurrentClassification.*`. Use it to scope interface rules to |
| 4897 | specific tiers -- for example: every interface on an edge exporter |
| 4898 | without a more-specific match falls back to "external". |
| 4899 | config: | |
| 4900 | enrichment: |
| 4901 | exporter_classifiers: |
| 4902 | - 'Exporter.Name startsWith "edge-" && Classify("edge") && ClassifyRole("border")' |
| 4903 | - 'Exporter.Name startsWith "core-" && Classify("core") && ClassifyRole("backbone")' |
| 4904 | interface_classifiers: |
| 4905 | # Specific provider rules first (most-specific to least-specific). |
| 4906 | - 'Interface.Description startsWith "BACKBONE-LUMEN" && ClassifyProvider("Lumen")' |
| 4907 | - 'Interface.Description startsWith "BACKBONE-COGENT" && ClassifyProvider("Cogent")' |
| 4908 | |
| 4909 | # Generic transit rule. |
| 4910 | - 'Interface.Description contains "TRANSIT" && ClassifyConnectivity("transit") && ClassifyExternal()' |
| 4911 | |
| 4912 | # Fallback: any unclassified interface on an edge box is external. |
| 4913 | - 'CurrentClassification.Role == "border" && CurrentClassification.Boundary == 0 && ClassifyExternal()' |
| 4914 | - name: Building values with Format and human-readable names |
| 4915 | folding: |
| 4916 | enabled: true |
| 4917 | description: | |
| 4918 | `Format(pattern, args...)` mimics Go's `fmt.Sprintf` for `%s`, `%v`, |
| 4919 | `%d`, `%%`. `Classify*` |
| 4920 | normalises (lowercase + strip non-alphanumeric); `SetName` and |
| 4921 | `SetDescription` do not, so they preserve the case and spaces of the |
| 4922 | computed value. |
| 4923 | config: | |
| 4924 | enrichment: |
| 4925 | exporter_classifiers: |
| 4926 | # Tenant computed from name, normalised on write -> "tenant-edge01". |
| 4927 | - 'ClassifyTenant(Format("tenant-%s", Exporter.Name))' |
| 4928 | interface_classifiers: |
| 4929 | # Human-readable name = "<exporter>:if<index>". Preserved verbatim. |
| 4930 | - 'SetName(Format("%s:if%d", Exporter.Name, Interface.Index))' |
| 4931 | - name: Tuning the cache for a large fleet |
| 4932 | folding: |
| 4933 | enabled: true |
| 4934 | description: | |
| 4935 | The default 5-minute last-access TTL is right for steady-state. Raise |
| 4936 | it when the (exporter, interface) population is large enough that |
| 4937 | evicted entries are quickly re-queried. Lower it when |
| 4938 | actively iterating on rule changes so misses pick up the new rules |
| 4939 | quickly. |
| 4940 | config: | |
| 4941 | enrichment: |
| 4942 | classifier_cache_duration: 30m |
| 4943 | exporter_classifiers: |
| 4944 | - 'Exporter.Name startsWith "edge-" && Classify("edge")' |
| 4945 | interface_classifiers: |
| 4946 | - 'Interface.Speed >= 100000000000 && ClassifyConnectivity("core")' |
| 4947 | troubleshooting: |
| 4948 | problems: |
| 4949 | list: |
| 4950 | - name: Plugin fails to start with a parser error |
| 4951 | description: | |
| 4952 | A rule failed to parse. The journal log includes the index in the list |
| 4953 | and a parser context (`unsupported rule term`, `unsupported value |
| 4954 | expression`, `Reject() does not accept arguments`, etc.). Common causes: |
| 4955 | missing `&&` between condition and action; an action used in the wrong |
| 4956 | list (`ClassifyExternal` in an exporter rule); strings written with |
| 4957 | single quotes (only JSON-style double quotes are accepted); regex literals |
| 4958 | that fail to compile. |
| 4959 | - name: Classifier rules never run for an exporter or interface |
| 4960 | description: | |
| 4961 | Likely cause: `metadata_static` already set **any** classification field |
| 4962 | on that target. By design, the matching list is suppressed entirely when |
| 4963 | the classification is non-empty. Either remove the static-metadata entry for that target, or |
| 4964 | keep static-metadata as the sole source for it. |
| 4965 | - name: A value appears differently in the dashboard than in the rule |
| 4966 | description: | |
| 4967 | `Classify*` actions normalise output to `[a-z0-9.+-]` only -- so |
| 4968 | `ClassifyRegion("EU West")` lands as `euwest`, and |
| 4969 | `Classify("Edge_Tier_1")` lands as `edgetier1`. Use `SetName` / |
| 4970 | `SetDescription` to preserve case and whitespace; those write the value |
| 4971 | verbatim. |
| 4972 | - name: First rule always wins, later rules never fire for the same slot |
| 4973 | description: | |
| 4974 | First-write-wins is by design and per slot. Order your |
| 4975 | rules from most-specific to least-specific. If you want a tiered |
| 4976 | fallback, use distinct slots (e.g. `Classify` for the broad group and |
| 4977 | `ClassifyRole` for the tier within that group). |
| 4978 | - name: A working rule stops matching some time after startup |
| 4979 | description: | |
| 4980 | Cached results expire after `classifier_cache_duration` (default 5 |
| 4981 | minutes, last-access). When you change rules, restart the plugin so the |
| 4982 | caches clear immediately -- otherwise stale cached classifications keep |
| 4983 | returning until they idle out. |
| 4984 | - name: A rule with `>` or `<` aborts the rule list |
| 4985 | description: | |
| 4986 | Comparing a string-typed identifier with `>` / `<` / `>=` / `<=` raises |
| 4987 | a runtime error, and the loop breaks out for that record. Subsequent rules in |
| 4988 | the list are skipped for that record. Use `matches`, `startsWith`, |
| 4989 | `endsWith`, `contains`, or `==` / `!=` on string fields. Keep `>` / `<` |
| 4990 | for `Interface.Index`, `Interface.Speed`, and `Interface.VLAN` (the |
| 4991 | numeric identifiers). |
| 4992 | - name: ClassifyExternal fires only on one side |
| 4993 | description: | |
| 4994 | Interface classifiers run twice per flow record -- once for the input |
| 4995 | interface, once for the output. Both invocations see the same rule list. If your rule conditions on |
| 4996 | `Interface.Index == 42` and that ifIndex appears in `IN_IF` of one flow |
| 4997 | and `OUT_IF` of another, the rule fires correctly in both places. But |
| 4998 | the `IN_IF_BOUNDARY` / `OUT_IF_BOUNDARY` columns are independent -- a |
| 4999 | rule firing on the output side of a flow only sets the output side's |
| 5000 | boundary, and vice versa. |
Showing first 5,000 of 5,277 lines.
View raw