@cryptotaxi247 / netdata
Code Commits Issues Pulls Sessions CI/CD
master
master
md 16.3 KB
View source Raw

Static Metadata

Plugin: netflow-plugin Module: static-metadata

Overview

Annotate network flows with operator-defined labels for exporters, interfaces, networks, and sampling rates. Static Metadata is a YAML-defined enrichment source that lives entirely in netflow.yaml. Use it when there is no IPAM, no CMDB, and no MMDB to query -- you simply know your own networks, your own routers, and what the interfaces on those routers do, and you want those labels visible on the Network Flows view.

Three independent surfaces are configured under this card, each populating a different set of flow-record fields:

  • enrichment.metadata_static.exporters.<ip-or-cidr> -- per-exporter labels (matched against the source IP of the UDP datagram) and per-interface labels (matched against the ifIndex from the flow record).
  • enrichment.networks.<cidr> -- labels for arbitrary CIDR blocks (your own corp ranges, customer ranges, public blocks you operate). Matched against the flow's source and destination IPs.
  • enrichment.override_sampling_rate.<cidr> -- per-prefix sampling-rate substitution for exporters that do not communicate their rate (e.g. NetFlow v7, which has no sampling field, or a v9 exporter that never sends a Sampling Options Template).

Fields populated:

  • From metadata_static.exporters (per-exporter): EXPORTER_NAME, EXPORTER_GROUP, EXPORTER_ROLE, EXPORTER_SITE, EXPORTER_REGION, EXPORTER_TENANT.
  • From metadata_static.exporters.if_indexes (per-interface): IN_IF_NAME / OUT_IF_NAME, IN_IF_DESCRIPTION / OUT_IF_DESCRIPTION, IN_IF_SPEED / OUT_IF_SPEED (in bits per second), IN_IF_PROVIDER / OUT_IF_PROVIDER, IN_IF_CONNECTIVITY / OUT_IF_CONNECTIVITY, IN_IF_BOUNDARY / OUT_IF_BOUNDARY (1 = external, 2 = internal, 0/omitted = removed from output).
  • From enrichment.networks (per-CIDR): SRC_NET_NAME / DST_NET_NAME, SRC_NET_ROLE / DST_NET_ROLE, SRC_NET_SITE / DST_NET_SITE, SRC_NET_REGION / DST_NET_REGION, SRC_NET_TENANT / DST_NET_TENANT, plus overrides for the GeoIP-derived SRC_COUNTRY / DST_COUNTRY, SRC_GEO_STATE / DST_GEO_STATE, SRC_GEO_CITY / DST_GEO_CITY, SRC_GEO_LATITUDE / DST_GEO_LATITUDE, SRC_GEO_LONGITUDE / DST_GEO_LONGITUDE. Setting asn: on a networks entry overrides the AS number via the same merge path; the AS name still comes from the ASN database (see "Composition with dynamic sources" below).
  • From override_sampling_rate (longest-prefix match against the exporter IP): SAMPLING_RATE is set unconditionally when an override prefix matches. This is different from default_sampling_rate, which only applies when the flow does not already carry a rate.

For the cross-cutting Enrichment concept (provider chains, merge order rules, how static composes with dynamic sources, the static-blocks-classifiers interaction, and shared failure modes), see Enrichment.

Edit netflow.yaml, restart the plugin, and the YAML-defined data is loaded into the same in-memory tries the GeoIP, IPAM, and BGP enrichment paths read from. There is no network access, no file watching, and no hot reload -- changes take effect on the next plugin startup.

This integration is only supported on the following platforms:

  • Linux

This integration runs as a single instance per Netdata Agent.

Default Behavior

Auto-Detection

Disabled by default. Add entries under enrichment.metadata_static, enrichment.networks, and / or enrichment.override_sampling_rate to populate it.

Limits

Resource use scales with the number of exporter, interface, network, and sampling-override entries. Keep CIDR ranges and interface maps specific enough to match the routers you actually export.

Performance Impact

Static lookups are local map/trie lookups during enrichment. Cost is usually small compared with flow decode and journal writes, but very large maps add memory and lookup work.

Setup

Prerequisites

An understanding of your own topology

You need to know which IP addresses your routers / switches / firewalls export from, the integer ifIndex values they use for each interface, and the CIDR ranges that belong to each of your sites / tenants / roles. None of this is auto-discovered -- this card is the "I will tell you" path. For sources that auto-discover the same data, see Generic IPAM, NetBox, and the cloud IP-range cards.

Persistent ifIndex values

if_indexes keys are the integer ifIndex the router puts in the flow record. Some platforms reassign ifIndex on line-card reseat or stack rebuild. After hardware changes, audit the labels -- a stale ifIndex entry silently no longer applies. The default interface block is used instead, or no labels are written when skip_missing_interfaces: true is set.

Configuration

Options

All keys live under enrichment: in netflow.yaml. Unknown keys fail config load with a parse error.

Config options | Option | Description | Default | Required | |:-----|:------------|:--------|:---------:| | enrichment.metadata_static.exporters | Map keyed by exporter IP or CIDR. Longest-prefix match wins. A `/32` key and a bare IP are equivalent. Each entry holds exporter-level labels (`name`, `group`, `role`, `site`, `region`, `tenant`), an optional `default` interface block, an `if_indexes` map keyed by integer ifIndex, and `skip_missing_interfaces`. | {} | no | | enrichment.metadata_static.exporters.<key>.if_indexes | Map keyed by integer ifIndex (the value the router puts in the flow record). Each entry holds `name`, `description`, `speed` (bits per second), `provider`, `connectivity`, and `boundary`. | {} | no | | enrichment.metadata_static.exporters.<key>.default | Interface block applied to flow records whose ifIndex is not present in `if_indexes`. Same shape as an `if_indexes` entry. Ignored when `skip_missing_interfaces: true` is set. | {} | no | | enrichment.metadata_static.exporters.<key>.skip_missing_interfaces | When `true`, flow records whose ifIndex is not in `if_indexes` get no interface labels at all (not even from `default`). Useful when you only care about a known set of WAN interfaces and want unknown ones to stay blank instead of inheriting a placeholder. | false | no | | enrichment.metadata_static.exporters.<key>.if_indexes.<n>.boundary | Interface boundary marker. Accepts the integers `0` (undefined), `1` (external -- faces the outside world: Internet, peer, transit), `2` (internal -- faces your own infrastructure), or the case-insensitive strings `"undefined"`, `"external"`, `"internal"`. Any other value fails config load. Filtering for `IN_IF_BOUNDARY=1` cleanly gives you "traffic that arrived from outside". | 0 | no | | enrichment.metadata_static.exporters.<key>.if_indexes.<n>.speed | Interface speed in **bits per second**. A 1 Gbps interface is `1000000000`, not `1000` and not `1000000`. Operators thinking in megabits get the value wrong by a factor of 1000 to 1000000. A `0` value means "not set" and removes the field from the output. | 0 | no | | enrichment.networks | Map keyed by CIDR. Longest-prefix match contributes the most-specific fields; less-specific containing prefixes contribute their non-empty fields too. The same merge rule is used by `network_sources`. Each value is either a string (shorthand for `name:`) or a map with `name`, `role`, `site`, `region`, `country`, `state`, `city`, `latitude`, `longitude`, `tenant`, `asn`. | {} | no | | enrichment.networks.<cidr>.asn | Forces the AS *number* for traffic in this prefix, overriding whatever the `asn_providers` chain computed. The AS *name* is still resolved from the ASN MMDB -- there is no `asn_name` config field. See the ASN section of [Enrichment](https://learn.netdata.cloud/docs/network-flows/enrichment). | 0 | no | | enrichment.networks.<cidr>.latitude / longitude | Per-CIDR coordinate override. Out-of-range values (latitude not in [-90, 90] or longitude not in [-180, 180]) and non-finite values are silently coerced to empty strings -- the field is dropped, no error. Validate input out of band if your data matters. | null | no | | enrichment.default_sampling_rate | Global fallback applied only when the flow record does not already carry a sampling rate. Either a single integer or a CIDR-keyed map (longest-prefix match against the exporter IP). | 0 | no | | enrichment.override_sampling_rate | Per-exporter substitution that **always** wins when its prefix matches the exporter IP, regardless of what the flow record carried. Either a single integer or a CIDR-keyed map. | {} | no |

via File

The configuration file name for this integration is netflow.yaml.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config netflow.yaml
Examples
Naming exporters and their interfaces

One edge router and one spine. The default block applies to any ifIndex not listed under if_indexes -- handy when you only care about a few WAN ports and want everything else to inherit a placeholder.

enrichment:
  metadata_static:
    exporters:
      192.0.2.10:                    # bare IP == /32
        name: edge-router-1
        site: par1
        region: eu-west
        role: edge
        tenant: tenant-a
        default:
          description: unclassified port
        if_indexes:
          1:
            name: Gi0/0/1
            description: uplink to ISP-A
            speed: 10000000000        # 10 Gbps in bits per second
            provider: isp-a
            connectivity: transit
            boundary: external
          2:
            name: Gi0/0/2
            description: LAN core
            speed: 1000000000
            connectivity: lan
            boundary: internal
      198.51.100.0/24:                # all routers in this subnet
        site: dc-fra1
        region: eu-central
        role: spine
        default:
          connectivity: lan
          boundary: internal
Tagging your own networks

The shorthand form (203.0.113.0/24: transit-a) sets only the name. The map form takes any subset of the schema fields. Less-specific supernets contribute non-empty fields to more-specific subnets via ascending-prefix-length merge -- declare RFC1918 once at /8 and a /24 inherits everything you did not override.

Config ```yaml enrichment: networks: 10.0.0.0/8: name: corp-internal role: internal tenant: tenant-a 198.51.100.0/24: # a public block you operate name: customer-acme role: customer site: par1 country: FR city: Paris latitude: 48.8566 longitude: 2.3522 asn: 64500 203.0.113.0/24: transit-a # shorthand: name only ```
Skipping interfaces you have not labelled

With skip_missing_interfaces: true, an ifIndex that is not in if_indexes produces no interface labels at all -- the default block is ignored. Useful when you want clearly-blank values for unmodelled interfaces instead of a placeholder description.

Config ```yaml enrichment: metadata_static: exporters: 192.0.2.10: name: edge-router-1 skip_missing_interfaces: true if_indexes: 1: name: Gi0/0/1 description: uplink to ISP-A connectivity: transit boundary: external ```
Sampling rate override for old or silent exporters

NetFlow v7 has no sampling field; some v9 exporters never send a Sampling Options Template. Without an override, the bytes/packets stored in the journal are exporter-side sampled counters with no scaling factor. override_sampling_rate always wins; default_sampling_rate is the fallback used only when the flow did not carry a rate.

Config ```yaml enrichment: default_sampling_rate: 1 # global fallback override_sampling_rate: 10.1.0.0/16: 1024 # an old NetFlow v7 stack 192.0.2.10/32: 4096 # one specific exporter ```

A more-specific entry inherits the wrong field from its supernet

enrichment.networks merges containing prefixes from least-specific to most-specific. Leaving a field blank on a /24 does not clear the /16's value for that field -- you must explicitly set the field on the more-specific entry to overwrite. The same merge rule applies to entries from network_sources, which interleave at the same prefix lengths.

GeoIP returns spurious data for an internal range

Until you declare your RFC1918 / RFC6598 / link-local ranges as enrichment.networks entries, the GeoIP-derived country / city / coord fields can pick up junk for those addresses. Adding a networks entry for the range overrides the GeoIP layer at the merge step.

ifIndex changed after a hardware swap

if_indexes keys are the numeric ifIndex sent in the flow record. A line-card reseat or stack rebuild can renumber the interfaces; the old ifIndex no longer matches and the per-interface block silently no longer applies. Audit after hardware changes.

speed shows up wrong by a factor of 1000

speed: is in bits per second. speed: 1000 means 1 kbps, not 1 Mbps. A 1 Gbps interface is 1000000000. speed: 0 means "not set" and removes the field from the output.

An interface block silently no longer applies

When the configured ifIndex is not present in the flow record, the default block is used instead -- unless skip_missing_interfaces: true is set, in which case no interface labels are written at all. If you expected your block to apply but the labels are blank, the router is sending a different ifIndex.

Coordinates dropped silently

Out-of-range latitude / longitude (latitude: 91.5) and non-finite values become empty strings without an error. The map quietly stops drawing the marker. Validate input externally if the data matters.

Static labels block the classifiers

When static metadata sets any of group, role, site, region, tenant for an exporter, the exporter_classifiers rule chain does not run for that exporter at all. The same is true for an interface: any of provider, connectivity, boundary set by static metadata short-circuits interface_classifiers. If you want classifiers to run on top of static metadata, drop the static fields they are supposed to set.

A typo fails config load

The schema is deny_unknown_fields at every level. A typo such as if_index (the canonical key is if_indexes; aliases ifindexes, if-indexes are accepted) or a misspelt attribute (teannt:) fails plugin start with a YAML parse error rather than being silently ignored.

Sampling override looks ignored

override_sampling_rate always wins when its prefix matches; if the field still looks unset, check that the exporter IP -- not the flow's source / destination IP -- falls under the configured prefix. default_sampling_rate is only consulted when the flow did not already carry a rate.

Changes do not take effect

Static metadata is loaded at plugin startup and there is no file-change watcher. Restart the plugin (or the agent) after editing netflow.yaml.