| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment.md" |
| 3 | sidebar_label: "Enrichment" |
| 4 | learn_status: "Published" |
| 5 | learn_rel_path: "Network Flows" |
| 6 | keywords: [enrichment, geoip, asn, bgp, classifiers, network labels, mmdb, ipam] |
| 7 | endmeta--> |
| 8 | |
| 9 | <!-- markdownlint-disable-file --> |
| 10 | |
| 11 | # Enrichment |
| 12 | |
| 13 | Raw flow records carry IP addresses, ports, ASNs the exporter happens to know, ifIndex numbers, and not much else. Enrichment is the post-decode pipeline that turns those into the operational labels you actually want on the dashboard: country and city, AS numbers and AS names, BGP next-hop and AS path, "this is our DMZ in Frankfurt", "this interface is the Lumen transit", "this exporter is a leaf in dc-fra1". |
| 14 | |
| 15 | This page is the cross-cutting reference for that pipeline. It documents the order of evaluation, the provider chains that decide where each field comes from, the shared mechanisms that span all enrichment methods, and the operational properties you need to know about. **Per-method specifics — installation steps, refresh cadence, expected upstream schemas, vendor-specific gotchas — live on the integration cards** linked at the bottom of this page. |
| 16 | |
| 17 | ## Order of evaluation per flow record |
| 18 | |
| 19 | Every flow record passes through the same pipeline before it is written to the journal. For one record: |
| 20 | |
| 21 | 1. **Decode** — the raw NetFlow / IPFIX / sFlow message is parsed into typed fields (`SRC_ADDR`, `DST_ADDR`, `SRC_AS`, ifIndexes, etc.). |
| 22 | 2. **Decapsulation** — when `protocols.decapsulation_mode` is `srv6` or `vxlan` and the exporter ships inner-packet bytes, the inner 5-tuple replaces the outer one. Everything below operates on the inner addresses. |
| 23 | 3. **GeoIP MMDB lookups** — the resolver runs the source and destination IPs against every configured ASN MMDB and every configured geo MMDB. Country, state, city, coordinates, AS number candidate, AS name, and the `ip_class` flag are seeded from the result. |
| 24 | 4. **Static metadata** — `enrichment.metadata_static.exporters` is matched against the exporter's IP (UDP source) and the ifIndex; `enrichment.networks` is matched per-IP, longest-prefix wins. Static `networks` entries can override country, state, city, latitude, longitude, AS number, and the `*_NET_*` labels. |
| 25 | 5. **Dynamic network sources** — every `enrichment.network_sources.<name>` source contributes CIDR records to the same network-attribute resolution path. Lookups for the source/destination IP merge matching records with static `enrichment.networks` entries. |
| 26 | 6. **Classifiers** — `exporter_classifiers` runs once per exporter; `interface_classifiers` runs twice per record (once for the input interface, once for the output). They write `EXPORTER_*` and `IN_IF_*` / `OUT_IF_*` fields. **Classifiers are skipped entirely when static metadata already set any classification field on the same target.** |
| 27 | 7. **Routing overlay** — the BGP-fed routing trie (BMP and BioRIS contribute to it) is consulted. The `asn_providers` and `net_providers` chains decide whether the AS number and the network mask come from the flow record, from BGP, or from the GeoIP MMDB. BGP-only fields (`NEXT_HOP`, `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES`) are written from BGP data. |
| 28 | 8. **Journal write** — the resulting record is written to the raw tier; rollup tiers (1 minute, 5 minutes, 1 hour) are computed from it asynchronously. |
| 29 | |
| 30 | The enricher runs when any enrichment feature is configured, including provider-chain settings. A deployment with all enrichment settings disabled writes raw decoded flow fields only. |
| 31 | |
| 32 | ## The two provider chains |
| 33 | |
| 34 | A "provider chain" is an ordered list of where to look for a given field. The plugin walks the chain left-to-right and takes the first non-zero answer. Two chains exist: |
| 35 | |
| 36 | ```yaml |
| 37 | enrichment: |
| 38 | asn_providers: [flow, routing, geoip] # default |
| 39 | net_providers: [flow, routing] # default |
| 40 | ``` |
| 41 | |
| 42 | These are the default provider chains. |
| 43 | |
| 44 | ### `asn_providers` — where the AS number comes from |
| 45 | |
| 46 | The chain decides where `SRC_AS` and `DST_AS` come from. The AS *name* (`SRC_AS_NAME`, `DST_AS_NAME`) is **always** rendered separately from the ASN MMDB lookup, regardless of the chain — see "AS numbers vs AS names" below. |
| 47 | |
| 48 | | Provider | What it returns | |
| 49 | |---|---| |
| 50 | | `flow` | The `SRC_AS` / `DST_AS` the exporter put on the record | |
| 51 | | `flow_except_private` | Same, but treats private/reserved AS numbers as zero | |
| 52 | | `flow_except_default_route` | Same, but treats `AS 0 with mask 0` as zero | |
| 53 | | `routing` | The AS from the BGP-fed routing trie (BMP, BioRIS, or static routing) | |
| 54 | | `routing_except_private` | Same as `routing` with the private filter | |
| 55 | | `geoip` | **Terminal "use 0" shortcut** — see below | |
| 56 | |
| 57 | `bmp` and `bmp-except-private` are accepted as backward-compatible aliases for `routing` and `routing_except_private`. |
| 58 | |
| 59 | The `geoip` slot is a **terminal shortcut**, not a normal provider. When the chain reaches `geoip`, the AS number is forced to `0` and the chain ends. The MMDB-derived AS number is then applied separately, outside the chain. This is intentional compatibility behaviour, not a bug. |
| 60 | |
| 61 | The implication is operational: **putting `geoip` anywhere except last truncates the chain.** With `[geoip, flow, routing]`, every AS resolves to `0` from the chain — only the GeoIP-derived AS makes it through, and `flow` and `routing` never run. This is rarely what you want. |
| 62 | |
| 63 | Common chain orderings: |
| 64 | |
| 65 | | Configuration | Behaviour | |
| 66 | |---|---| |
| 67 | | `[flow, routing, geoip]` (default) | Trust the exporter, fall back to BGP, then to GeoIP | |
| 68 | | `[flow, routing]` | No GeoIP at all | |
| 69 | | `[routing, flow, geoip]` | Trust the BGP feed first — useful when exporters have stale AS | |
| 70 | | `[flow_except_private, routing, geoip]` | Drop private AS from flow data, let routing fill in | |
| 71 | |
| 72 | `is_private_as` returns true for `0`, `23456` (RFC 4893 transition reserved), `64496..=65551` (documentation / private), and `>= 4_200_000_000` (32-bit private and high-reserved). |
| 73 | |
| 74 | ### `net_providers` — where the network mask comes from |
| 75 | |
| 76 | Same idea, smaller menu: the chain produces `SRC_MASK` and `DST_MASK` (and indirectly `NEXT_HOP` via the routing entry). Only `flow` and `routing` are valid here — there is no `geoip` slot, because the MMDB does not carry network masks. |
| 77 | |
| 78 | ### AS numbers vs AS names |
| 79 | |
| 80 | The two are resolved by different mechanisms and never share the chain. |
| 81 | |
| 82 | - **AS numbers** come from the chain above. |
| 83 | - **AS names** are always rendered from the resolved AS number plus the ASN MMDB. The format is `AS{n}` for non-zero ASNs (with `{organisation}` appended when the MMDB has it), `AS0 Unknown ASN` when the resolved AS is zero, and `AS0 Private IP Address Space` when the ASN MMDB tagged the IP as private. There is no static configuration option for AS names. |
| 84 | |
| 85 | A static `enrichment.networks.<cidr>.asn` override is applied during the per-IP network-attribute merge **before** the chain runs. The AS name still goes through the MMDB lookup. |
| 86 | |
| 87 | ## How attributes compose: GeoIP, network sources, static `networks` |
| 88 | |
| 89 | For any source or destination IP, the plugin produces network attributes by merging multiple inputs. The merge rule is "non-empty overlay overwrites; empty overlay leaves the field alone" — so the **last input with a non-empty value wins, per field.** |
| 90 | |
| 91 | The merge order, per IP, is: |
| 92 | |
| 93 | 1. **GeoIP base layer** — country, state, city, coordinates, AS number candidate, AS name, `ip_class`. |
| 94 | 2. **All matching prefixes in ascending prefix-length order** (least-specific first, most-specific last). At each prefix length: |
| 95 | - **Dynamic `network_sources` records** are merged first (priority 0). |
| 96 | - **Static `enrichment.networks` records** are merged second (priority 1). |
| 97 | |
| 98 | Two consequences worth stating explicitly: |
| 99 | |
| 100 | - **Specificity dominates.** A more-specific prefix (longer mask) always wins on its non-empty fields, regardless of whether it came from a static source or a dynamic feed. A `/24` static entry overwrites a `/16` dynamic entry; a `/24` dynamic entry overwrites a `/16` static entry. |
| 101 | - **At the same prefix length, static `networks` wins.** Because static is merged after dynamic at each level, and the merge primitive is "non-empty wins", a non-empty field in a static entry overwrites the dynamic entry's value at the same prefix length. This is intentional — explicit operator config beats imported data — but it surprises operators who expect the remote feed to be authoritative. |
| 102 | |
| 103 | A practical consequence of "merge ascending, non-empty wins": **a more-specific entry that leaves a field blank inherits the supernet's value for that field.** To clear a field on a `/24`, you must set it explicitly to a value, not just leave it out. |
| 104 | |
| 105 | Static `networks` can also override `country`, `state`, `city`, `latitude`, `longitude`, and `asn` per CIDR. These compose with the same merge rules — they are simply additional non-empty fields that overwrite the GeoIP base layer when matched. Coordinates (`latitude`, `longitude`) are static-only; network-identity sources cannot set them. |
| 106 | |
| 107 | ## The MMDB shared mechanism |
| 108 | |
| 109 | DB-IP, MaxMind GeoIP / GeoLite2, IPtoASN-converted-to-MMDB, and any custom MMDB build all use the same MMDB resolver behavior. The configuration: |
| 110 | |
| 111 | ```yaml |
| 112 | enrichment: |
| 113 | geoip: |
| 114 | asn_database: |
| 115 | - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 116 | geo_database: |
| 117 | - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 118 | optional: true |
| 119 | ``` |
| 120 | |
| 121 | | Key | Type | Notes | |
| 122 | |---|---|---| |
| 123 | | `asn_database` | list of paths | One or more ASN MMDB files. Aliases: `asn-database`. | |
| 124 | | `geo_database` | list of paths | One or more geographic MMDB files. Aliases: `geo-database`, `country-database`. | |
| 125 | | `optional` | bool, default `false` | When `false`, a missing file at startup aborts the plugin. When `true`, missing or unreadable files are tolerated. Auto-detected files are always treated as `optional: true`. | |
| 126 | |
| 127 | ### Auto-detect path order |
| 128 | |
| 129 | When neither `asn_database` nor `geo_database` is configured, the plugin searches in this order at startup: |
| 130 | |
| 131 | 1. **`<cache_dir>/topology-ip-intel/`** — where `cache_dir` is the parent of `journal.journal_dir` if that path is absolute, otherwise `NETDATA_CACHE_DIR`. Typically `/var/cache/netdata/topology-ip-intel/`. |
| 132 | 2. **`<stock_data_dir>/topology-ip-intel/`** — typically `/usr/share/netdata/topology-ip-intel/`. |
| 133 | |
| 134 | The canonical filenames are `topology-ip-asn.mmdb` and `topology-ip-geo.mmdb`. Native packages ship stock DB-IP files under the stock data directory. The [Intel Downloader](/docs/network-flows/intel-downloader.md) writes fresher copies to the cache directory, which takes precedence over stock files. Netdata does not install a downloader timer; schedule one if freshness matters. |
| 135 | |
| 136 | ### Composition: last non-empty wins, per field |
| 137 | |
| 138 | `asn_database` and `geo_database` are lists. For each lookup, every database in the list runs in order; per output field, the **last** database that produces a **non-empty** value wins. Empty / zero values returned by a later database do not overwrite an earlier match. |
| 139 | |
| 140 | This is the same merge rule as the network-attributes merge, applied to the MMDB scan. |
| 141 | |
| 142 | The practical use is stacking: list a stronger source first for one field and a backup for the rest. For example, layering MaxMind after IPtoASN in `asn_database` recovers AS *names* (which IPtoASN-converted MMDBs often lack) without losing IPtoASN's AS *number* coverage. |
| 143 | |
| 144 | ### Signature-watch reload |
| 145 | |
| 146 | The resolver checks each configured database's signature (size + mtime) every 30 seconds and reloads the readers in place when the signature changes. |
| 147 | |
| 148 | Only successful reloads swap the active readers — a transient read error keeps the previous readers serving lookups. |
| 149 | |
| 150 | This is why MMDB refresh scripts only need to atomically replace the file on disk: the plugin picks up the new file within 30 seconds, no restart needed. |
| 151 | |
| 152 | ### IPv4 / IPv6 dual-stack handling |
| 153 | |
| 154 | The resolver inspects each MMDB's metadata. An IPv6 lookup against an IPv4-only database is **silently skipped** (no warning, no error). Most current providers (DB-IP, MaxMind GeoLite2, MaxMind GeoIP2, IPtoASN combined) ship a single dual-stack MMDB that covers both families; legacy single-family builds are the only case where this matters in practice. |
| 155 | |
| 156 | ## Network sources: shared operational properties |
| 157 | |
| 158 | `enrichment.network_sources.<name>` is the dynamic counterpart to static `networks`. AWS IP Ranges, GCP IP Ranges, Azure IP Ranges, NetBox, and custom JSON-over-HTTP IPAM sources share the same operational contract. The only difference between cards is the upstream URL, the JSON shape, and the jq transform. |
| 159 | |
| 160 | ### Fetch loop and back-off |
| 161 | |
| 162 | Each source runs in its own task. Multiple sources fetch in parallel; within a source, only one fetch is in flight at a time. |
| 163 | |
| 164 | Per cycle: |
| 165 | |
| 166 | 1. HTTP GET (or POST) at `interval` cadence, with `headers:` applied. |
| 167 | 2. Response parsed as JSON. |
| 168 | 3. The configured `transform` (a [jaq](https://github.com/01mf02/jaq) expression) runs over the parsed JSON. |
| 169 | 4. Output is decoded as a stream of `RemoteRecord` objects. |
| 170 | 5. Records are merged into the shared network-attributes trie. |
| 171 | |
| 172 | On any failure (HTTP error, JSON parse error, jq runtime error, **empty result**), the source backs off exponentially starting at `interval / 10` (floor 1 second), doubling on each retry, capped at `interval`. On success it resets to the configured `interval`. |
| 173 | |
| 174 | The scheduler floors dynamic-source intervals at 60 seconds. So `interval: 5s` is effectively `60s`. There is no signature/etag change detection — the plugin re-parses and re-publishes the entire record set on every successful fetch. |
| 175 | |
| 176 | ### Expected jq output schema |
| 177 | |
| 178 | The `transform` must emit a stream of objects with this shape: |
| 179 | |
| 180 | ```json |
| 181 | { |
| 182 | "prefix": "10.0.0.0/8", |
| 183 | "name": "internal", |
| 184 | "role": "lan", |
| 185 | "site": "fra1", |
| 186 | "region": "eu-central", |
| 187 | "country": "DE", |
| 188 | "state": "HE", |
| 189 | "city": "Frankfurt", |
| 190 | "tenant": "tenant-a", |
| 191 | "asn": 64500, |
| 192 | "asn_name": "Internal AS" |
| 193 | } |
| 194 | ``` |
| 195 | |
| 196 | - **Required**: `prefix` (CIDR string). |
| 197 | - **Optional**: every other field. Defaults to empty / 0. |
| 198 | - `asn` accepts an integer (`64500`), a string (`"64500"`), or AS notation (`"AS64500"`). |
| 199 | - **Coordinates are not settable** — the deserializer has no field for `latitude` / `longitude`. Use static `networks` for coordinates. |
| 200 | |
| 201 | The default `transform` is `"."`, which returns the raw JSON object — never a per-prefix stream. **Every real source needs a custom `transform`**; the per-source integration cards each ship a working example. |
| 202 | |
| 203 | If the transform produces zero objects on a successful HTTP fetch, the cycle is treated as a failure and triggers backoff. This catches silently broken jq but punishes legitimately empty IPAMs. |
| 204 | |
| 205 | ### Strict YAML validation |
| 206 | |
| 207 | Configuration schemas use `deny_unknown_fields` at every level. A typo in `network_sources.<name>.<key>` fails at startup with a `unknown field 'X', expected one of ...` error. This applies to the entire enrichment config, not just network sources. |
| 208 | |
| 209 | ### TLS verification cannot be disabled |
| 210 | |
| 211 | Configuration validation rejects any attempt to disable TLS verification. Legacy keys are accepted for forward compatibility, but no override exists. |
| 212 | |
| 213 | Self-signed or internal CAs must be supplied via `tls.ca_file`. The rationale is deliberate: enrichment data flows directly into security investigations and capacity decisions — silently accepting MITM-able responses would corrupt every downstream analysis. |
| 214 | |
| 215 | ### Single page only — no pagination |
| 216 | |
| 217 | The fetch is one-shot per cycle. There is no pagination, no cursor handling, no `Link: rel=next` following. Operators with paginated IPAM endpoints must either expose a separate "all prefixes" bulk endpoint (most IPAMs have one) or wrap with a server-side aggregator script. |
| 218 | |
| 219 | This is irrelevant to AWS / GCP / Azure (single-file feeds) and bites NetBox / generic IPAM. The NetBox card documents `?limit=0` (NetBox 4.x) as the bypass. |
| 220 | |
| 221 | ### Authentication — bring your own headers |
| 222 | |
| 223 | The plugin has no built-in OAuth flow, basic-auth helper, or token refresh. Whatever the upstream API needs goes into `headers:`: |
| 224 | |
| 225 | ```yaml |
| 226 | headers: |
| 227 | Authorization: "Token abc123" |
| 228 | ``` |
| 229 | |
| 230 | `headers` is a free-form map, so any single-shot scheme works. URL-embedded credentials (`https://user:pass@host/...`) are converted to HTTP Basic authentication by the HTTP client, but explicit `Authorization` headers are clearer and avoid storing credentials in URLs. Short-lived tokens must be refreshed outside Netdata and reloaded into the config. |
| 231 | |
| 232 | POST is supported but **sent with no body**. POST endpoints that require a request body are not supported. |
| 233 | |
| 234 | ### Field-level merge across sources sharing prefixes |
| 235 | |
| 236 | When two sources (say `aws` and a custom internal IPAM) both define overlapping prefixes, all records are merged into the same trie. Within one source, last-write-wins for an exact prefix length; across sources, the merge is field-level — non-empty fields from a later record overwrite earlier ones. This is the same primitive described in "How attributes compose" above. |
| 237 | |
| 238 | ### Diagnostic journal output |
| 239 | |
| 240 | Failures (transform compile, HTTP, JSON parse, jq runtime, empty result) are logged in the Netdata journal namespace: |
| 241 | |
| 242 | ```bash |
| 243 | journalctl --namespace netdata | grep network-sources |
| 244 | ``` |
| 245 | |
| 246 | JSON parse errors are silent in the dashboard — the journal is the place to look. |
| 247 | |
| 248 | ## Static metadata short-circuits classifiers |
| 249 | |
| 250 | When `metadata_static.exporters` set **any** classification field (`group`, `role`, `site`, `region`, `tenant`) for an exporter, the entire `exporter_classifiers` rule chain is skipped for that exporter. |
| 251 | |
| 252 | The same rule applies to interfaces — any of `provider` / `connectivity` / `boundary` set by static metadata short-circuits `interface_classifiers` for that interface. |
| 253 | |
| 254 | This is "Akvorado parity" by design. The implication: **don't try to mix static and rule-based classification on the same target.** If you set even one field statically, all classifier rules for that target are bypassed. Decide per-target: enumerate it in static metadata, or let the classifier rules find it. |
| 255 | |
| 256 | ## Classifier evaluation surfaces and ordering |
| 257 | |
| 258 | - **Exporter classifiers** run once per exporter, with the result cached. |
| 259 | - **Interface classifiers** run once per `(exporter, interface)` pair, **twice per flow record** (once for the input interface, once for the output). |
| 260 | - **First write wins** per output slot. Once a rule sets `EXPORTER_GROUP`, no later rule can change it. Order rules from most-specific to least-specific. |
| 261 | - **Cache TTL** is `enrichment.classifier_cache_duration` (default 5 minutes, last-access). Validation rejects values shorter than 1 second. |
| 262 | - **Runtime errors stop the rule list** but keep whatever was set so far. Use string-safe operators (`matches`, `startsWith`, `contains`) rather than `>` / `<` on string fields. |
| 263 | |
| 264 | The classifier syntax is intentionally Akvorado-compatible for the documented operators and actions, so existing Akvorado rule lists usually paste in unchanged. Akvorado's full `expr-lang` features (arithmetic, ternaries, lambdas) are not implemented here. |
| 265 | |
| 266 | ## Decapsulation: the inner-packet override |
| 267 | |
| 268 | When `protocols.decapsulation_mode` is `srv6` or `vxlan` and the exporter ships inner-packet bytes (NetFlow v9 IE 104, IPFIX IE 315, sFlow `SampledHeader`), the inner 5-tuple replaces the outer one **before any enrichment runs**: |
| 269 | |
| 270 | - `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, `DST_PORT`, `PROTOCOL`, `ETYPE`, `IPTOS`, `IPTTL`, `IPV6_FLOW_LABEL`, `TCP_FLAGS`, MPLS labels, ICMP type/code, `BYTES` (inner L3 length). |
| 271 | - For VXLAN: inner `SRC_MAC` / `DST_MAC` / `SRC_VLAN` / `DST_VLAN`. **Outer MACs and VLANs are lost.** |
| 272 | |
| 273 | This means **all downstream enrichment operates on the inner addresses**: GeoIP runs against the inner IPs, network attributes match the inner prefixes, and the routing trie is consulted with the inner addresses. |
| 274 | |
| 275 | Two important rules: |
| 276 | |
| 277 | - **Decap is destructive on non-tunnel traffic.** Records arriving via the L2-section path that are not the configured tunnel are **dropped**, not falled back to outer view. Plain NetFlow / IPFIX flow records that don't go through the L2-section path are unaffected. |
| 278 | - **VXLAN VNI is parsed but not surfaced.** Bytes 4-6 of the VXLAN header are skipped. Pure VNI-based segmentation is not visible as a filter or group-by field. |
| 279 | |
| 280 | GRE, IP-in-IP, GENEVE, MPLS-over-UDP, and NVGRE are not decoded. |
| 281 | |
| 282 | See the [Decapsulation integration card](/src/crates/netflow-plugin/integrations/decapsulation.md) for exporter configuration recipes. |
| 283 | |
| 284 | ## Routing overlay (BMP and BioRIS share the trie) |
| 285 | |
| 286 | BMP and BioRIS are separate transports that feed the **same** in-memory routing trie. A deployment running BMP from internal routers and BioRIS from a separate bio-rd-compatible RIS endpoint gets unified enrichment without duplicate trie entries. |
| 287 | |
| 288 | BioRIS means Netdata consumes the `bio.ris.RoutingInformationService` gRPC API. Netdata does not connect directly to RIPE RIS Live, RIPEstat, RIS MRT dumps, or RIPE route collector sessions. If you need a RIPE-derived external view, put a converter or bio-rd-compatible service in front of that data and point Netdata at the resulting gRPC endpoint. |
| 289 | |
| 290 | Each prefix entry holds a list of routes keyed by `(peer, route_key)` so multipath BGP and multiple peers contributing the same prefix coexist. Lookups walk the trie longest-prefix-first, then refine within candidates by: |
| 291 | |
| 292 | 1. Exporter IP matches the flow's exporter AND next-hop matches the flow's next-hop; |
| 293 | 2. otherwise exporter IP matches; |
| 294 | 3. otherwise next-hop matches; |
| 295 | 4. otherwise the first route at the matched prefix. |
| 296 | |
| 297 | The fields populated are `SRC_AS` / `DST_AS` (via the `routing` provider in the chain), `SRC_MASK` / `DST_MASK` (via `net_providers`), and `NEXT_HOP`, `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES` (always written from BGP data when the lookup hits, no chain involvement). |
| 298 | |
| 299 | AS *names* still come from the ASN MMDB, not from BGP. BGP gives accurate AS numbers, paths, and communities; the names come from the database. |
| 300 | |
| 301 | ## Operational properties that span all methods |
| 302 | |
| 303 | ### Refresh windows |
| 304 | |
| 305 | | Mechanism | Window | Notes | |
| 306 | |---|---|---| |
| 307 | | GeoIP MMDB | **30 seconds** | Signature watch (size + mtime) | |
| 308 | | Dynamic network sources | per-source `interval`, **floor 60 s** | Independent per source | |
| 309 | | BMP routing | live | Routers push updates over TCP | |
| 310 | | BioRIS routing | live | gRPC stream from a bio-rd daemon | |
| 311 | | Static metadata | **plugin restart** | No hot-reload | |
| 312 | | Classifier rules | **plugin restart** | No hot-reload | |
| 313 | |
| 314 | ### Restart behaviour |
| 315 | |
| 316 | - **GeoIP databases** are reloaded automatically — no special handling needed. |
| 317 | - **The routing trie is not persisted.** A restart wipes BGP-derived data; it is re-learned as routers re-send Initiation + Update messages (BMP) or as bio-rd's next refresh cycle dumps the RIB (BioRIS). Convergence ranges from seconds (FRR) to minutes for full-table BioRIS feeds. Schedule restarts off-peak when BGP attribution matters. |
| 318 | - **Network-source records** are re-fetched on the next interval tick. There is no persistence between restarts. |
| 319 | - **Classifier caches** are wiped — they refill on first hit per target. |
| 320 | |
| 321 | ### No in-process freshness signal |
| 322 | |
| 323 | There is no metric or alert for "your MMDB is too old", "your IPAM hasn't refreshed in 6 hours", or "your BMP session is stuck". Operators must watch: |
| 324 | |
| 325 | - The MMDB file's `mtime` on disk. |
| 326 | - The Netdata journal (`journalctl --namespace netdata | grep -E 'network-sources|geoip|bmp|bioris'`) for fetch failures, parse errors, and session events. |
| 327 | - Per-source last-success timestamps (where the integration card documents them). |
| 328 | |
| 329 | ### Empty enrichment trees disable the enricher |
| 330 | |
| 331 | When every static and dynamic input is empty, no enrichment runs and the journal carries no `*_NET_*`, `*_AS_NAME`, `*_COUNTRY`, etc. Useful for verifying that adding config actually opted you in — if fields stay empty after a config change, suspect the enricher never started because the config did not enable any enrichment source. |
| 332 | |
| 333 | ### Field survival into rollup tiers |
| 334 | |
| 335 | The journal stores raw records and three rollup tiers (1 minute, 5 minutes, 1 hour). High-cardinality fields are dropped from rollups to keep them tractable. |
| 336 | |
| 337 | | Field | Raw | 1 m | 5 m | 1 h | Notes | |
| 338 | |---|---|---|---|---|---| |
| 339 | | `SRC_COUNTRY`, `DST_COUNTRY` | yes | yes | yes | yes | ~250 values | |
| 340 | | `SRC_GEO_STATE`, `DST_GEO_STATE` | yes | yes | yes | yes | Low thousands | |
| 341 | | `SRC_AS`, `DST_AS`, `SRC_AS_NAME`, `DST_AS_NAME` | yes | yes | yes | yes | Bounded by global routing table size | |
| 342 | | `SRC_NET_*`, `DST_NET_*` (NAME, ROLE, SITE, REGION, TENANT) | yes | yes | yes | yes | Operator-defined cardinality | |
| 343 | | `NEXT_HOP` | yes | yes | yes | yes | | |
| 344 | | `SRC_GEO_CITY`, `DST_GEO_CITY` | yes | — | — | — | **Raw tier only** | |
| 345 | | `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE` | yes | — | — | — | **Raw tier only**, hidden by default | |
| 346 | | `SRC_GEO_LONGITUDE`, `DST_GEO_LONGITUDE` | yes | — | — | — | **Raw tier only**, hidden by default | |
| 347 | | `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES` | yes | — | — | — | **Raw tier only** | |
| 348 | | `MPLS_LABELS`, `SRC_MAC`, `DST_MAC`, NAT fields | yes | — | — | — | **Raw tier only** | |
| 349 | |
| 350 | City / coordinate / AS-path / community queries that span longer windows than the raw retention will silently fall back to a rollup tier and return empty. Narrow the window or use country / state. |
| 351 | |
| 352 | See [Retention and querying](/docs/network-flows/retention-querying.md) and [Field reference](/docs/network-flows/field-reference.md) for the full picture. |
| 353 | |
| 354 | ### Geographic accuracy is best-effort |
| 355 | |
| 356 | City-level GeoIP is accurate for many public IPs but wrong for VPNs, mobile carriers, and cloud-provider egress IPs (which often resolve to the cloud region's central city, not the actual user). Use country and state for trends; use city only after validating it for the prefix you care about. ASN ownership data also drifts — companies merge, prefixes get reassigned. A database older than a quarter or two starts labelling reassigned prefixes with the previous owner. |
| 357 | |
| 358 | ### Sampling-rate knobs |
| 359 | |
| 360 | Two related, easily confused knobs (set under `enrichment` in `netflow.yaml`): |
| 361 | |
| 362 | - `default_sampling_rate` — consulted **only** when the flow record did not carry a rate. |
| 363 | - `override_sampling_rate` — **always** wins when its prefix matches the exporter IP (the source IP of the UDP datagram), regardless of what the flow carried. |
| 364 | |
| 365 | Both accept either an integer (uniform rate) or a CIDR-keyed map. The exporter match key is the same one used by `metadata_static.exporters` lookup — the source IP of the UDP datagram, not a router-internal management ID. |
| 366 | |
| 367 | ### Validate BGP-derived enrichment before relying on it |
| 368 | |
| 369 | BMP and BioRIS depend on router configuration, export policy, peer identity, and route-refresh behaviour. Treat configuration changes as production-impacting and validate against your specific routers before relying on BGP-derived data for capacity or security decisions. |
| 370 | |
| 371 | ## What can go wrong |
| 372 | |
| 373 | - **AS numbers all zero.** `geoip` is mid-chain. Reorder to put `geoip` last. |
| 374 | - **Map renders empty over a long window.** City / lat / lon are raw-tier only; the query auto-fell-back to a rollup tier. Narrow the window or use the country / state map. |
| 375 | - **Static-defined exporter doesn't get classifier labels.** Static metadata set at least one classification field; classifiers are skipped for that target. Decide one or the other per target. |
| 376 | - **Remote feed value not appearing — static config silently won.** Static `networks` wins ties at the same prefix length. Either use a more-specific dynamic entry or remove the conflicting static field. |
| 377 | - **Network source backs off on an empty IPAM.** Empty result is treated as failure. Workaround: have the upstream return at least one synthetic prefix. |
| 378 | - **Decap drops legitimate traffic.** When `decapsulation_mode` is set, non-tunnel traffic on the L2-section path is dropped, not fallen back. Either use a uniform encapsulation per exporter or disable decap. |
| 379 | - **`speed: 1000` means 1 kbps.** The static-metadata `speed` field is in **bits per second**. A 1 Gbps interface is `1000000000`. |
| 380 | - **MMDB updates not picked up.** Verify the file actually changed on disk (size or mtime). The 30-second poll only triggers on signature change. |
| 381 | - **Network-source TLS cannot be disabled.** Use `tls.ca_file` for internal CAs. |
| 382 | |
| 383 | ## What's next |
| 384 | |
| 385 | ### Per-method integration cards |
| 386 | |
| 387 | These cards carry the per-method specifics — installation steps, refresh cadence, expected upstream schemas, vendor-specific gotchas — that this page deliberately does not duplicate. |
| 388 | |
| 389 | **IP intelligence (MMDB)** |
| 390 | - [DB-IP IP Intelligence](/src/crates/netflow-plugin/integrations/db-ip_ip_intelligence.md) — the default that ships with Netdata |
| 391 | - [MaxMind GeoIP / GeoLite2](/src/crates/netflow-plugin/integrations/maxmind_geoip_-_geolite2.md) — commercial GeoIP2 or free GeoLite2 with attribution |
| 392 | - [IPtoASN](/src/crates/netflow-plugin/integrations/iptoasn.md) — public-domain ASN + country, hourly cadence |
| 393 | - [Custom MMDB Database](/src/crates/netflow-plugin/integrations/custom_mmdb_database.md) — your own MMDB build |
| 394 | |
| 395 | **BGP routing** |
| 396 | - [BMP (BGP Monitoring Protocol)](/src/crates/netflow-plugin/integrations/bmp_bgp_monitoring_protocol.md) — routers push BGP updates over TCP |
| 397 | - [bio-rd RIS](/src/crates/netflow-plugin/integrations/bio-rd_-_ripe_ris.md) — pull BGP data from a bio-rd-compatible RIS gRPC daemon |
| 398 | |
| 399 | **Network identity (cloud IP ranges, IPAM)** |
| 400 | - [AWS IP Ranges](/src/crates/netflow-plugin/integrations/aws_ip_ranges.md) — public AWS prefix list |
| 401 | - [GCP IP Ranges](/src/crates/netflow-plugin/integrations/gcp_ip_ranges.md) — public GCP prefix list |
| 402 | - [Azure IP Ranges](/src/crates/netflow-plugin/integrations/azure_ip_ranges.md) — Azure Service Tags (requires an internal mirror) |
| 403 | - [NetBox](/src/crates/netflow-plugin/integrations/netbox.md) — open-source IPAM / DCIM |
| 404 | - [Generic JSON-over-HTTP IPAM](/src/crates/netflow-plugin/integrations/generic_json-over-http_ipam.md) — Infoblox, BlueCat, phpIPAM, custom CMDBs |
| 405 | |
| 406 | **Static and rule-based** |
| 407 | - [Static Metadata](/src/crates/netflow-plugin/integrations/static_metadata.md) — per-exporter, per-interface, per-CIDR labels |
| 408 | - [Classifiers](/src/crates/netflow-plugin/integrations/classifiers.md) — Akvorado-compatible expression rules |
| 409 | |
| 410 | **Decapsulation** |
| 411 | - [Decapsulation](/src/crates/netflow-plugin/integrations/decapsulation.md) — SRv6 and VXLAN inner-packet extraction |
| 412 | |
| 413 | ### Related concepts |
| 414 | |
| 415 | - [Intel Downloader](/docs/network-flows/intel-downloader.md) — the bundled tool that fetches and converts MMDB files for the auto-detect path. |
| 416 | - [Configuration](/docs/network-flows/configuration.md) — the full `enrichment` section reference. |
| 417 | - [Field reference](/docs/network-flows/field-reference.md) — every flow record field, what writes it, what reads it. |
| 418 | - [Retention and querying](/docs/network-flows/retention-querying.md) — which fields survive into rollups, how queries pick a tier. |