| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/static_metadata.md" |
| 3 | meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml" |
| 4 | sidebar_label: "Static Metadata" |
| 5 | learn_status: "Published" |
| 6 | learn_rel_path: "Network Flows/Enrichment Methods" |
| 7 | keywords: ['static metadata', 'exporters', 'interfaces', 'networks', 'cidr', 'sampling rate override', 'asn override', 'country override', 'tenant', 'site', 'region', 'role'] |
| 8 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE" |
| 9 | endmeta--> |
| 10 | |
| 11 | <!-- markdownlint-disable-file --> |
| 12 | |
| 13 | # Static Metadata |
| 14 | |
| 15 | |
| 16 | <img src="https://netdata.cloud/img/network-wired.svg" width="150"/> |
| 17 | |
| 18 | |
| 19 | Plugin: netflow-plugin |
| 20 | Module: static-metadata |
| 21 | |
| 22 | <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> |
| 23 | |
| 24 | ## Overview |
| 25 | |
| 26 | Annotate network flows with operator-defined labels for exporters, interfaces, |
| 27 | networks, and sampling rates. Static Metadata is a YAML-defined enrichment source |
| 28 | that lives entirely in `netflow.yaml`. Use it when there is no IPAM, no CMDB, and |
| 29 | no MMDB to query -- you simply know your own networks, your own routers, and what |
| 30 | the interfaces on those routers do, and you want those labels visible on the |
| 31 | Network Flows view. |
| 32 | |
| 33 | Three independent surfaces are configured under this card, each populating a |
| 34 | different set of flow-record fields: |
| 35 | |
| 36 | - `enrichment.metadata_static.exporters.<ip-or-cidr>` -- per-exporter labels |
| 37 | (matched against the source IP of the UDP datagram) and per-interface labels |
| 38 | (matched against the ifIndex from the flow record). |
| 39 | - `enrichment.networks.<cidr>` -- labels for arbitrary CIDR blocks (your own |
| 40 | corp ranges, customer ranges, public blocks you operate). Matched against the |
| 41 | flow's source and destination IPs. |
| 42 | - `enrichment.override_sampling_rate.<cidr>` -- per-prefix sampling-rate |
| 43 | substitution for exporters that do not communicate their rate (e.g. NetFlow |
| 44 | v7, which has no sampling field, or a v9 exporter that never sends a Sampling |
| 45 | Options Template). |
| 46 | |
| 47 | Fields populated: |
| 48 | |
| 49 | - From `metadata_static.exporters` (per-exporter): |
| 50 | `EXPORTER_NAME`, `EXPORTER_GROUP`, `EXPORTER_ROLE`, `EXPORTER_SITE`, |
| 51 | `EXPORTER_REGION`, `EXPORTER_TENANT`. |
| 52 | - From `metadata_static.exporters.if_indexes` (per-interface): |
| 53 | `IN_IF_NAME` / `OUT_IF_NAME`, `IN_IF_DESCRIPTION` / `OUT_IF_DESCRIPTION`, |
| 54 | `IN_IF_SPEED` / `OUT_IF_SPEED` (in **bits per second**), `IN_IF_PROVIDER` / |
| 55 | `OUT_IF_PROVIDER`, `IN_IF_CONNECTIVITY` / `OUT_IF_CONNECTIVITY`, |
| 56 | `IN_IF_BOUNDARY` / `OUT_IF_BOUNDARY` (`1` = external, `2` = internal, |
| 57 | `0`/omitted = removed from output). |
| 58 | - From `enrichment.networks` (per-CIDR): |
| 59 | `SRC_NET_NAME` / `DST_NET_NAME`, `SRC_NET_ROLE` / `DST_NET_ROLE`, |
| 60 | `SRC_NET_SITE` / `DST_NET_SITE`, `SRC_NET_REGION` / `DST_NET_REGION`, |
| 61 | `SRC_NET_TENANT` / `DST_NET_TENANT`, plus overrides for the GeoIP-derived |
| 62 | `SRC_COUNTRY` / `DST_COUNTRY`, `SRC_GEO_STATE` / `DST_GEO_STATE`, |
| 63 | `SRC_GEO_CITY` / `DST_GEO_CITY`, `SRC_GEO_LATITUDE` / `DST_GEO_LATITUDE`, |
| 64 | `SRC_GEO_LONGITUDE` / `DST_GEO_LONGITUDE`. Setting `asn:` on a `networks` |
| 65 | entry overrides the AS *number* via the same merge path; the AS *name* |
| 66 | still comes from the ASN database (see "Composition with dynamic sources" |
| 67 | below). |
| 68 | - From `override_sampling_rate` (longest-prefix match against the exporter IP): |
| 69 | `SAMPLING_RATE` is set unconditionally when an override prefix matches. This |
| 70 | is different from `default_sampling_rate`, which only applies when the flow |
| 71 | does not already carry a rate. |
| 72 | |
| 73 | For the cross-cutting Enrichment concept (provider chains, merge order rules, |
| 74 | how static composes with dynamic sources, the static-blocks-classifiers |
| 75 | interaction, and shared failure modes), see |
| 76 | [Enrichment](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 77 | |
| 78 | |
| 79 | Edit `netflow.yaml`, restart the plugin, and the YAML-defined data is loaded |
| 80 | into the same in-memory tries the GeoIP, IPAM, and BGP enrichment paths read |
| 81 | from. There is no network access, no file watching, and no hot reload -- |
| 82 | changes take effect on the next plugin startup. |
| 83 | |
| 84 | |
| 85 | This integration is only supported on the following platforms: |
| 86 | |
| 87 | - Linux |
| 88 | |
| 89 | This integration runs as a single instance per Netdata Agent. |
| 90 | |
| 91 | |
| 92 | ### Default Behavior |
| 93 | |
| 94 | #### Auto-Detection |
| 95 | |
| 96 | Disabled by default. Add entries under `enrichment.metadata_static`, `enrichment.networks`, and / or `enrichment.override_sampling_rate` to populate it. |
| 97 | |
| 98 | #### Limits |
| 99 | |
| 100 | Resource use scales with the number of exporter, interface, network, and sampling-override entries. Keep CIDR ranges and interface maps specific enough to match the routers you actually export. |
| 101 | |
| 102 | #### Performance Impact |
| 103 | |
| 104 | Static lookups are local map/trie lookups during enrichment. Cost is usually small compared with flow decode and journal writes, but very large maps add memory and lookup work. |
| 105 | |
| 106 | ## Setup |
| 107 | |
| 108 | |
| 109 | ### Prerequisites |
| 110 | |
| 111 | #### An understanding of your own topology |
| 112 | |
| 113 | You need to know which IP addresses your routers / switches / firewalls |
| 114 | export from, the integer ifIndex values they use for each interface, and |
| 115 | the CIDR ranges that belong to each of your sites / tenants / roles. None |
| 116 | of this is auto-discovered -- this card is the "I will tell you" path. |
| 117 | For sources that auto-discover the same data, see Generic IPAM, NetBox, |
| 118 | and the cloud IP-range cards. |
| 119 | |
| 120 | |
| 121 | #### Persistent ifIndex values |
| 122 | |
| 123 | `if_indexes` keys are the integer ifIndex the router puts in the flow |
| 124 | record. Some platforms reassign ifIndex on line-card reseat or stack |
| 125 | rebuild. After hardware changes, audit the labels -- a stale ifIndex |
| 126 | entry silently no longer applies. The `default` interface block is used |
| 127 | instead, or no labels are written when `skip_missing_interfaces: true` is set. |
| 128 | |
| 129 | |
| 130 | |
| 131 | ### Configuration |
| 132 | |
| 133 | #### Options |
| 134 | |
| 135 | All keys live under `enrichment:` in `netflow.yaml`. Unknown keys fail config |
| 136 | load with a parse error. |
| 137 | |
| 138 | |
| 139 | <details open><summary>Config options</summary> |
| 140 | |
| 141 | |
| 142 | |
| 143 | | Option | Description | Default | Required | |
| 144 | |:-----|:------------|:--------|:---------:| |
| 145 | | enrichment.metadata_static.exporters | Map keyed by exporter IP or CIDR. Longest-prefix match wins. A `/32` key and a bare IP are equivalent. Each entry holds exporter-level labels (`name`, `group`, `role`, `site`, `region`, `tenant`), an optional `default` interface block, an `if_indexes` map keyed by integer ifIndex, and `skip_missing_interfaces`. | {} | no | |
| 146 | | enrichment.metadata_static.exporters.<key>.if_indexes | Map keyed by integer ifIndex (the value the router puts in the flow record). Each entry holds `name`, `description`, `speed` (bits per second), `provider`, `connectivity`, and `boundary`. | {} | no | |
| 147 | | enrichment.metadata_static.exporters.<key>.default | Interface block applied to flow records whose ifIndex is not present in `if_indexes`. Same shape as an `if_indexes` entry. Ignored when `skip_missing_interfaces: true` is set. | {} | no | |
| 148 | | enrichment.metadata_static.exporters.<key>.skip_missing_interfaces | When `true`, flow records whose ifIndex is not in `if_indexes` get no interface labels at all (not even from `default`). Useful when you only care about a known set of WAN interfaces and want unknown ones to stay blank instead of inheriting a placeholder. | false | no | |
| 149 | | enrichment.metadata_static.exporters.<key>.if_indexes.<n>.boundary | Interface boundary marker. Accepts the integers `0` (undefined), `1` (external -- faces the outside world: Internet, peer, transit), `2` (internal -- faces your own infrastructure), or the case-insensitive strings `"undefined"`, `"external"`, `"internal"`. Any other value fails config load. Filtering for `IN_IF_BOUNDARY=1` cleanly gives you "traffic that arrived from outside". | 0 | no | |
| 150 | | enrichment.metadata_static.exporters.<key>.if_indexes.<n>.speed | Interface speed in **bits per second**. A 1 Gbps interface is `1000000000`, not `1000` and not `1000000`. Operators thinking in megabits get the value wrong by a factor of 1000 to 1000000. A `0` value means "not set" and removes the field from the output. | 0 | no | |
| 151 | | enrichment.networks | Map keyed by CIDR. Longest-prefix match contributes the most-specific fields; less-specific containing prefixes contribute their non-empty fields too. The same merge rule is used by `network_sources`. Each value is either a string (shorthand for `name:`) or a map with `name`, `role`, `site`, `region`, `country`, `state`, `city`, `latitude`, `longitude`, `tenant`, `asn`. | {} | no | |
| 152 | | enrichment.networks.<cidr>.asn | Forces the AS *number* for traffic in this prefix, overriding whatever the `asn_providers` chain computed. The AS *name* is still resolved from the ASN MMDB -- there is no `asn_name` config field. See the ASN section of [Enrichment](https://learn.netdata.cloud/docs/network-flows/enrichment). | 0 | no | |
| 153 | | enrichment.networks.<cidr>.latitude / longitude | Per-CIDR coordinate override. Out-of-range values (latitude not in [-90, 90] or longitude not in [-180, 180]) and non-finite values are silently coerced to empty strings -- the field is dropped, no error. Validate input out of band if your data matters. | null | no | |
| 154 | | enrichment.default_sampling_rate | Global fallback applied only when the flow record does not already carry a sampling rate. Either a single integer or a CIDR-keyed map (longest-prefix match against the exporter IP). | 0 | no | |
| 155 | | enrichment.override_sampling_rate | Per-exporter substitution that **always** wins when its prefix matches the exporter IP, regardless of what the flow record carried. Either a single integer or a CIDR-keyed map. | {} | no | |
| 156 | |
| 157 | |
| 158 | </details> |
| 159 | |
| 160 | |
| 161 | |
| 162 | #### via File |
| 163 | |
| 164 | The configuration file name for this integration is `netflow.yaml`. |
| 165 | |
| 166 | |
| 167 | You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the |
| 168 | Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory). |
| 169 | |
| 170 | ```bash |
| 171 | cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata |
| 172 | sudo ./edit-config netflow.yaml |
| 173 | ``` |
| 174 | |
| 175 | ##### Examples |
| 176 | |
| 177 | ###### Naming exporters and their interfaces |
| 178 | |
| 179 | One edge router and one spine. The `default` block applies to any |
| 180 | ifIndex not listed under `if_indexes` -- handy when you only care about |
| 181 | a few WAN ports and want everything else to inherit a placeholder. |
| 182 | |
| 183 | |
| 184 | ```yaml |
| 185 | enrichment: |
| 186 | metadata_static: |
| 187 | exporters: |
| 188 | 192.0.2.10: # bare IP == /32 |
| 189 | name: edge-router-1 |
| 190 | site: par1 |
| 191 | region: eu-west |
| 192 | role: edge |
| 193 | tenant: tenant-a |
| 194 | default: |
| 195 | description: unclassified port |
| 196 | if_indexes: |
| 197 | 1: |
| 198 | name: Gi0/0/1 |
| 199 | description: uplink to ISP-A |
| 200 | speed: 10000000000 # 10 Gbps in bits per second |
| 201 | provider: isp-a |
| 202 | connectivity: transit |
| 203 | boundary: external |
| 204 | 2: |
| 205 | name: Gi0/0/2 |
| 206 | description: LAN core |
| 207 | speed: 1000000000 |
| 208 | connectivity: lan |
| 209 | boundary: internal |
| 210 | 198.51.100.0/24: # all routers in this subnet |
| 211 | site: dc-fra1 |
| 212 | region: eu-central |
| 213 | role: spine |
| 214 | default: |
| 215 | connectivity: lan |
| 216 | boundary: internal |
| 217 | |
| 218 | ``` |
| 219 | ###### Tagging your own networks |
| 220 | |
| 221 | The shorthand form (`203.0.113.0/24: transit-a`) sets only the `name`. |
| 222 | The map form takes any subset of the schema fields. Less-specific |
| 223 | supernets contribute non-empty fields to more-specific subnets via |
| 224 | ascending-prefix-length merge -- declare RFC1918 once at `/8` and |
| 225 | a `/24` inherits everything you did not override. |
| 226 | |
| 227 | |
| 228 | <details open><summary>Config</summary> |
| 229 | |
| 230 | ```yaml |
| 231 | enrichment: |
| 232 | networks: |
| 233 | 10.0.0.0/8: |
| 234 | name: corp-internal |
| 235 | role: internal |
| 236 | tenant: tenant-a |
| 237 | 198.51.100.0/24: # a public block you operate |
| 238 | name: customer-acme |
| 239 | role: customer |
| 240 | site: par1 |
| 241 | country: FR |
| 242 | city: Paris |
| 243 | latitude: 48.8566 |
| 244 | longitude: 2.3522 |
| 245 | asn: 64500 |
| 246 | 203.0.113.0/24: transit-a # shorthand: name only |
| 247 | |
| 248 | ``` |
| 249 | </details> |
| 250 | |
| 251 | ###### Skipping interfaces you have not labelled |
| 252 | |
| 253 | With `skip_missing_interfaces: true`, an ifIndex that is not in |
| 254 | `if_indexes` produces no interface labels at all -- the `default` |
| 255 | block is ignored. Useful when you want clearly-blank values for |
| 256 | unmodelled interfaces instead of a placeholder description. |
| 257 | |
| 258 | |
| 259 | <details open><summary>Config</summary> |
| 260 | |
| 261 | ```yaml |
| 262 | enrichment: |
| 263 | metadata_static: |
| 264 | exporters: |
| 265 | 192.0.2.10: |
| 266 | name: edge-router-1 |
| 267 | skip_missing_interfaces: true |
| 268 | if_indexes: |
| 269 | 1: |
| 270 | name: Gi0/0/1 |
| 271 | description: uplink to ISP-A |
| 272 | connectivity: transit |
| 273 | boundary: external |
| 274 | |
| 275 | ``` |
| 276 | </details> |
| 277 | |
| 278 | ###### Sampling rate override for old or silent exporters |
| 279 | |
| 280 | NetFlow v7 has no sampling field; some v9 exporters never send a |
| 281 | Sampling Options Template. Without an override, the bytes/packets |
| 282 | stored in the journal are exporter-side **sampled** counters with |
| 283 | no scaling factor. `override_sampling_rate` always wins; |
| 284 | `default_sampling_rate` is the fallback used only when the flow |
| 285 | did not carry a rate. |
| 286 | |
| 287 | |
| 288 | <details open><summary>Config</summary> |
| 289 | |
| 290 | ```yaml |
| 291 | enrichment: |
| 292 | default_sampling_rate: 1 # global fallback |
| 293 | override_sampling_rate: |
| 294 | 10.1.0.0/16: 1024 # an old NetFlow v7 stack |
| 295 | 192.0.2.10/32: 4096 # one specific exporter |
| 296 | |
| 297 | ``` |
| 298 | </details> |
| 299 | |
| 300 | |
| 301 | |
| 302 | ### A more-specific entry inherits the wrong field from its supernet |
| 303 | |
| 304 | `enrichment.networks` merges containing prefixes from least-specific to |
| 305 | most-specific. Leaving a field blank on a `/24` does **not** clear the `/16`'s value |
| 306 | for that field -- you must explicitly set the field on the more-specific |
| 307 | entry to overwrite. The same merge rule applies to entries from |
| 308 | `network_sources`, which interleave at the same prefix lengths. |
| 309 | |
| 310 | |
| 311 | ### GeoIP returns spurious data for an internal range |
| 312 | |
| 313 | Until you declare your RFC1918 / RFC6598 / link-local ranges as |
| 314 | `enrichment.networks` entries, the GeoIP-derived country / city / coord |
| 315 | fields can pick up junk for those addresses. Adding a `networks` entry |
| 316 | for the range overrides the GeoIP layer at the merge step. |
| 317 | |
| 318 | |
| 319 | ### ifIndex changed after a hardware swap |
| 320 | |
| 321 | `if_indexes` keys are the numeric ifIndex sent in the flow record. A |
| 322 | line-card reseat or stack rebuild can renumber the interfaces; the old |
| 323 | ifIndex no longer matches and the per-interface block silently no longer |
| 324 | applies. Audit after hardware changes. |
| 325 | |
| 326 | |
| 327 | ### speed shows up wrong by a factor of 1000 |
| 328 | |
| 329 | `speed:` is in **bits per second**. `speed: 1000` means 1 kbps, not |
| 330 | 1 Mbps. A 1 Gbps interface is `1000000000`. `speed: 0` means "not set" |
| 331 | and removes the field from the output. |
| 332 | |
| 333 | |
| 334 | ### An interface block silently no longer applies |
| 335 | |
| 336 | When the configured ifIndex is not present in the flow record, the |
| 337 | `default` block is used instead -- unless `skip_missing_interfaces: true` |
| 338 | is set, in which case no interface labels are written at all. If you expected your block to apply but the labels are blank, the |
| 339 | router is sending a different ifIndex. |
| 340 | |
| 341 | |
| 342 | ### Coordinates dropped silently |
| 343 | |
| 344 | Out-of-range latitude / longitude (`latitude: 91.5`) and non-finite |
| 345 | values become empty strings without an error. The map quietly stops drawing the marker. Validate input externally if |
| 346 | the data matters. |
| 347 | |
| 348 | |
| 349 | ### Static labels block the classifiers |
| 350 | |
| 351 | When static metadata sets **any** of `group`, `role`, `site`, `region`, |
| 352 | `tenant` for an exporter, the `exporter_classifiers` rule chain does not |
| 353 | run for that exporter at all. The same is true for an interface: any of `provider`, `connectivity`, |
| 354 | `boundary` set by static metadata short-circuits |
| 355 | `interface_classifiers`. If |
| 356 | you want classifiers to run on top of static metadata, drop the static |
| 357 | fields they are supposed to set. |
| 358 | |
| 359 | |
| 360 | ### A typo fails config load |
| 361 | |
| 362 | The schema is `deny_unknown_fields` at every level. A typo such as |
| 363 | `if_index` (the canonical key is `if_indexes`; aliases `ifindexes`, |
| 364 | `if-indexes` are accepted) or a misspelt attribute (`teannt:`) fails plugin |
| 365 | start with a YAML parse error rather than being silently ignored. |
| 366 | |
| 367 | |
| 368 | ### Sampling override looks ignored |
| 369 | |
| 370 | `override_sampling_rate` always wins when its prefix matches; if the |
| 371 | field still looks unset, check that the **exporter IP** -- not the |
| 372 | flow's source / destination IP -- falls under the configured prefix. |
| 373 | `default_sampling_rate` is only consulted when the flow did not already |
| 374 | carry a rate. |
| 375 | |
| 376 | |
| 377 | ### Changes do not take effect |
| 378 | |
| 379 | Static metadata is loaded at plugin startup and there is no file-change |
| 380 | watcher. Restart the plugin (or the agent) after editing |
| 381 | `netflow.yaml`. |
| 382 | |
| 383 | |
| 384 |