@cryptotaxi247 / netdata-1 / commits / a073bcf24

Network Flows: docs rewrite, integrations, benchmarks (#22439)

* netflow-plugin: per-protocol benchmark + storage footprint test Add NETFLOW_RESOURCE_BENCH_PROTOCOL env var so the paced post-decode resource benchmark can isolate one protocol (netflow-v9, ipfix, sflow) instead of using only the mixed scenario. The protocol name is added to the ResourceEnvelopeReport JSON so driver scripts can attribute results per protocol. Add bench_storage_footprint_child: a long-running paced ingest benchmark that samples per-tier on-disk size, write amplification (real I/O vs logical bytes), and dedup ratio over a configurable measurement window. Configured via NETFLOW_STORAGE_BENCH_DURATION_SECS (default 900) and NETFLOW_STORAGE_BENCH_SAMPLE_INTERVAL_SECS (default 30). Adds StorageFootprintReport / StorageFootprintSample types to ingest_resource_bench_support.rs. Refresh README.md with the post-optimization benchmark numbers. The previous numbers (5.8-6.3k flows/s saturation on one core) were captured before recent ingest-path optimizations and were severely stale. Saturation is now ~30k flows/s post-decode at high cardinality and above 60k at low cardinality, with the path effectively single-threaded at the saturation knee. README now documents the per-protocol/cardinality matrix from a fresh rerun on the same i9-12900K host plus the unpaced UDP->journal numbers, and explains the multi-thread CPU semantics of cpu_percent_of_one_core. * docs: rewrite Network Flows documentation Comprehensive rewrite of docs/network-flows/. 25 pages covering installation, configuration, all protocol sources (NetFlow v5/v7/v9, IPFIX, sFlow), all enrichment features (GeoIP, static metadata, classifiers, ASN resolution, BMP routing, BioRIS, network sources, decapsulation), the visualisation surface (Sankey + Table, Time-Series, maps and globe, filters and facets, plugin health charts), retention and querying, sizing and capacity planning, validation, investigation playbooks, anti-patterns, and troubleshooting. Each page is grounded in code analysis rather than inheriting claims from the previous (thin) documentation. Notable corrections: - GeoIP default database is DB-IP, not MaxMind. Only the MMDB format is MaxMind's; the data shipped with packages comes from DB-IP. - IN_IF_BOUNDARY semantics: 1 = external, 2 = internal. - Interface speed is in bps, not Mbps. - Default journal retention is 10GB / 7d per tier (not 200M / 2d), and the same default applies to every tier independently. - v9 datalink IE is 104 (not 102). - Field count is 91 (not 89). - Classifiers use an Akvorado-compatible expression subset, not jq/jaq. - query_max_groups default is 50000; query_facet_max_values_per_field is 5000. - Function timeout is a hard 30s. - Globe is a frontend-only re-rendering of city-map data, not a separate backend view. Adds four pages not in the previous outline: Installation, Validation & Data Quality, Investigation Playbooks, Anti-patterns. The doubling and mirroring concept (every router exports both ingress and egress flow records, so aggregate volume looks 2x reality) is established as foundational and called out on the Overview, Quick Start, Anti-patterns, and Troubleshooting pages. Vendor router/switch configurations were verified against current vendor documentation (Cisco IOS-XE/XR/ASA, Juniper JunOS, Arista EOS, Aruba CX, Ruckus FastIron, FortiGate, FRR) and the open-source exporters (softflowd, hsflowd, nProbe, pmacct, bio-rd). Several broken examples in the previous docs were fixed: - Cisco IOS-XR NetFlow: replaced invalid "sampler ALL" with a sampler-map block. - Cisco IOS-XR BMP: bmp server moved from inside router bgp to global config (where it actually lives in IOS-XR). - Juniper inline-jflow: added missing input rate, chassis fpc sampling-instance, and firewall filter required for sampling to fire on most platforms. - Arista IPFIX: replaced wrong sub-command names (active timeout / inactive timeout don't exist; correct forms are record export on interval and record export on inactive timeout, both in milliseconds). - Aruba CX sFlow: interface-level "sflow enable" is wrong; the command is just "sflow". - Ruckus FastIron sFlow: removed invalid "sflow source ethernet"; added missing per-interface "sflow forwarding". - FortiGate NetFlow: added the nested "config collectors" form required from FortiOS 7.2.8 / 7.4.2 onward. - pmacct: corrected "interface:" to "pcap_interface:" and timeout keys "idle/active" to "tcp/maxlife". - bio-rd: corrected flag names ("--grpc_port", "--config.file") and the binary output name. - FRR BMP: added the critical "-M bmp" daemons flag without which every BMP command silently fails. Tracks the work in .agents/sow/pending/SOW-0014-20260506-netflow-sflow- ipfix-documentation-guide.md, including Decision 8 (benchmark rerun matrix), Decision 9 (documentation rewrite directives), and the phased plan covering benchmark rerun, storage footprint measurement, and the per-page rewrite methodology. * integrations: add netflow-plugin (netflow, ipfix, sflow modules) Add the netflow plugin to the integrations pipeline so its three flow modules appear in the in-app integrations catalog, on Learn under Collecting Metrics > Collectors > Networking, and in the COLLECTORS.md umbrella index. Source-of-truth: src/crates/netflow-plugin/metadata.yaml (3 modules: netflow, ipfix, sflow). Each module's overview, troubleshooting, and metrics sections cross-reference the rich Network Flows documentation under /docs/network-flows/ so users discovering the plugin via the in-app catalog or the Learn integration card are guided to the deep docs (vendor configurations, sampling caveats, validation, anti- patterns, investigation playbooks). Pipeline plumbing: - gen_integrations.py: add netflow-plugin/metadata.yaml as a single- file COLLECTOR_SOURCES entry, plus the (r, d) tuple bug fix for the single-file branch of get_collector_metadata_entries. - gen_docs_integrations.py: add src/crates/netflow-plugin to the cleanup list so per-integration .md files regenerate cleanly. - Generated artifacts: src/crates/netflow-plugin/integrations/ {netflow,ipfix,sflow}.md (these are the standard generated per- integration pages with the DO-NOT-EDIT marker), plus the new rows in src/collectors/COLLECTORS.md. Learn routing: docs/.map/map.yaml gains an 87-line Network Flows section that routes the 25 hand-authored documentation pages under docs/network-flows/ to learn.netdata.cloud. The auto-generated per-integration pages remain under "Collecting Metrics/Collectors/ Networking" (the standard collector path) -- that's the catalog- entry surface; the rich documentation gets its own top-level "Network Flows" tree. Skill update: .agents/skills/learn-site-structure/mapping.md gets a new "Inter-page linking" section documenting that markdown links between Learn pages must use the GitHub-relative .md-extension form ("/docs/.../page.md") -- which is the convention used throughout the new Network Flows documentation. * integrations: introduce 'flows' as a new top-level integration_type Network flows are a different data type from metrics, just as logs are. This commit promotes the netflow-plugin's catalog presence from a single 'collector' entry buried under data-collection.networking to its own top-level integration_type with 14 cards across 4 sub-categories. The catalog test that drives this is the user question "Does Netdata integrate with X?". For each X where the answer is yes, X gets a card. New integration_type infrastructure: - integrations/schemas/flows.json -- new schema (refs collector.json for now; clean fork point for future divergence). - integrations/categories.yaml -- new top-level "Network Flows" node with 4 sub-categories: Sources, IP Intelligence, BGP Routing, Network Identity Sources. - integrations/gen_integrations.py -- FLOWS_SOURCES, FLOWS_RENDER_KEYS, FLOWS_VALIDATOR, load_flows(), render_flows(), wired into main. - integrations/gen_docs_integrations.py -- 'flows' branch in build_readme_from_integration() and the dispatch loop. learn_rel_path derives directly from categories (no replace step). - integrations/templates/overview.md -- dispatcher entry for flows. - integrations/templates/overview/flows.md -- new overview template (cloned from collector.md, adjusted for non-metrics types). 14 catalog cards under flows: - Sources (3): NetFlow, IPFIX, sFlow - IP Intelligence (4): DB-IP, MaxMind GeoIP/GeoLite2, IPtoASN, Custom MMDB - BGP Routing (2): BMP, bio-rd / RIPE RIS - Network Identity (5): AWS IP Ranges, GCP IP Ranges, Azure IP Ranges, NetBox, Generic JSON-over-HTTP IPAM Each card carries its own setup with vendor/provider-specific examples, prerequisites, troubleshooting, and cross-links to companion concept docs. The metadata.yaml grows from 3 modules to 14. Documentation reshape: - docs/network-flows/sources/ deleted entirely. Per-protocol content (vendor configs, sampling caveats, version semantics) lives in the metadata.yaml setup blocks for the netflow / ipfix / sflow modules. The integration cards become the canonical per-protocol references. - docs/network-flows/enrichment/{geoip,bmp-routing,bioris,network-sources}.md deleted -- their content is now distributed across the per-provider integration cards. - 3 new concept docs -- enrichment/ip-intelligence.md, enrichment/bgp-routing.md, enrichment/network-identity.md -- cover the cross-cutting concepts that don't fit a single per-provider card (MMDB format, shared trie semantics, lookup priority, jq transform rules, TLS verification policy, integration-test gap honesty). - docs/.map/map.yaml -- Network Flows section restructured. The per-provider Sources/Enrichment subtrees are replaced by a single integration_placeholder integration_kind: flows that pulls all 14 cards into the section. Concept docs (Overview, Quick Start, Configuration, etc.) plus the 3 new enrichment-concept pages remain as hand-authored entries. - All cross-references in remaining docs updated -- links to deleted per-protocol pages now point at the integration .md, links to deleted per-provider enrichment pages now point at the new concept docs. Skill updates: - .agents/skills/integrations-lifecycle/per-type-matrix.md -- new "Learn section per integration type" section with the routing table for every type (collector / exporter / agent_notification / cloud_notification / logs / authentication / secretstore / service_discovery / deploy), the hardcoded-vs-derived split explanation, and the integration_placeholder mechanism. flows is now a documented type peer. - .agents/skills/integrations-lifecycle/how-tos/adding-new-integration-type.md -- new how-to with the 8-place checklist (schema, pipeline registration, pipeline rendering, templates, categories.yaml, source metadata, map.yaml, downstream repos). Written when adding flows so future work can follow the recipe. - .agents/skills/integrations-lifecycle/how-tos/INDEX.md -- catalog entry registered. Validation: - pipeline (gen_integrations.py + gen_docs_integrations.py) runs to exit 0. - 14 flows-typed integrations appear in integrations.json under the expected sub-categories. - 14 generated per-integration .md files render with the correct learn_rel_path (Network Flows/Sources, Network Flows/IP Intelligence, Network Flows/BGP Routing, Network Flows/Network Identity Sources). - COLLECTORS.md no longer has the netflow / ipfix / sflow rows (they are not collectors anymore). Downstream PR plan unchanged from prior commit -- learn auto-discovers, cloud-frontend's catalog rendering is fully data-driven, www repo's solution-page corrections already landed on the netflow-flows-content branch. * netflow-plugin: substring autocomplete on text facets The facet autocomplete used prefix-only matching, so typing "Akamai" never surfaced "AS20940 Akamai International" (every AS_NAME starts with "AS{n} "). The same problem hit every multi-token text facet: EXPORTER_NAME, IN_IF_DESCRIPTION, *_NET_NAME, MAC, AS_PATH, communities, country/state/city. - Per-field policy: new AutocompleteMatchKind { Prefix, Substring } on FacetFieldSpec. Default by value-kind: Text -> Substring, IP/numeric -> Prefix. Per-field overrides are now possible. Regular facet matching (selections) is unchanged - exact equality, indexed, never substring. - SIMD: text substring uses memchr::memmem::Finder for both the active store and the FST sidecar scan. - Cap: autocomplete term is hard-capped at 256 bytes at the deserializer; scoped to mode=autocomplete only so non-autocomplete flows requests with a stale term are unaffected. - Async hygiene: autocomplete moved to spawn_blocking, matching how normal flow / timeseries queries are scheduled. - Docs: filters-facets.md and retention-querying.md updated to describe per-field policy and the autocomplete-vs-selection distinction. Tests: 9 new tests (substring on active store, substring through FST sidecar after promotion, multi-sidecar coverage, empty term, long term without panic, exact-100 promotion threshold, IP prefix preservation, oversized-term rejection at deserialize, long term accepted in non-autocomplete mode). Full crate: 427 passed, 0 failed. Follow-up: autocomplete substring on FST sidecars is bounded by limit early-exit but still streams keys for rare/no-hit terms over very large archived vocabularies. Token-prefix or n-gram indexing can be added later if measurements warrant it. * docs/.map: accept 'flows' as an integration_kind The Network Flows section in map.yaml uses `integration_placeholder` with `integration_kind: flows`, but the schema enum only listed the previously existing kinds. The CI `check-documentation` job rejected the file with a schema validation error, blocking the netflow-plugin docs PR. Add `flows` to the enum so the schema mirrors the catalog (collector, logs, exporter, ..., flows). * docs(network-flows): screenshots and master field index - Embed dashboard screenshots on the visualization pages: Sankey + table (with and without the table), Time-Series, country/state/city maps, and the 3D globe. - Add an alphabetical master index at the bottom of field-reference.md with every flow field's protocol availability, source class (decoder / enrichment / both), tier preservation (raw vs all), selectivity (facet, group-by, filter, metric, time, hidden), and the enrichment chain or IE mapping per row. Lets users look up any field in one place instead of scanning by category. * sow: close SOW-0014 (netflow / sflow / ipfix documentation guide) All acceptance criteria delivered or explicitly rejected. Phase 1C (AI skills cross-links to learn docs) was rejected by user as not mandatory; recorded in Validation. Architectural pivot to a top-level `flows` integration_type with 14 cards is recorded as Decision 10. Validation, Outcome, Lessons, and Followup sections filled in. Companion changes shipped in netdata PR #22439 (this branch) and learn PR #2854 (merged). File moved from pending/ to done/.

Costa Tsaousis committed May 7, 2026 at 17:01 UTC a073bcf24f79ce70e5cf3935ec6fa0996160ab30

63 files changed +10097 -117

.agents/skills/integrations-lifecycle/how-tos/INDEX.md

+1 -4

@@ -22,12 +22,9 @@ violation.

23 ## Catalog

25 -(empty -- entries grow as assistants encounter

26 -not-yet-documented questions)

27 -

25 | Topic | Slug | Notes |

26 |---|---|---|

30 -| -- | -- | -- |

27 +| Adding a new top-level `integration_type` (peer of collector, logs, exporter, etc.) | [adding-new-integration-type](adding-new-integration-type.md) | 8-step recipe: schema, pipeline, templates, categories.yaml, map.yaml, source metadata, downstream repos. Covers what to clone from existing types, the hardcoded-vs-derived `learn_rel_path` distinction, and the `integration_placeholder` mechanism. |

29 ## How to add a how-to

.agents/skills/integrations-lifecycle/how-tos/adding-new-integration-type.md new

+152

@@ -0,0 +1,152 @@

1 +# How-to: Add a new `integration_type`

2 +

3 +**One-line summary:** Adding a new top-level `integration_type` (peer of `collector`, `logs`, `exporter`, etc.) requires changes in eight places — schema, pipeline, templates, categories, map.yaml, and downstream consumers.

4 +

5 +This how-to was written when adding `flows` (NetFlow / IPFIX / sFlow) as a new integration type, on the basis that flows are a different data type from metrics (just like logs are) and deserve their own Learn section and in-app sidebar entry.

6 +

7 +## When this applies

8 +

9 +You're adding a new fundamental data type or operational concern that:

10 +

11 +- Has its own schema (the existing types' schemas don't fit cleanly).

12 +- Should appear in its own Learn section (not buried under `Collecting Metrics/Collectors/<sub-category>`).

13 +- Should appear in its own in-app sidebar entry (not as a sub-category of an existing one).

14 +- Has multiple integration entries that share a common shape.

15 +

16 +Examples that would qualify:

17 +

18 +- `flows` (NetFlow / IPFIX / sFlow) — flow record collection.

19 +- `topology` (when implemented as a dedicated catalog) — topology graph sources.

20 +

21 +Examples that would NOT qualify (use an existing type instead):

22 +

23 +- A new database collector — use `integration_type: collector` with `categories: [data-collection.databases]`.

24 +- A new alert channel — use `integration_type: agent_notification` or `cloud_notification`.

25 +

26 +## What to change

27 +

28 +### 1. JSON schema (`integrations/schemas/<type>.json`)

29 +

30 +Clone the closest existing schema and trim/adjust:

31 +

32 +- For data-bearing types (something that produces records / metrics / events): start from `collector.json`. Trim fields that don't apply (e.g., flows have no `metrics` or `alerts` sections in the traditional sense).

33 +- For thin types (a config target with overview + setup + troubleshooting): start from `logs.json` or `exporter.json`.

34 +

35 +The schema field set drives validation. Anything not in the schema is rejected as an unknown property when `additionalProperties: false` is set on the relevant object.

36 +

37 +### 2. Pipeline registration (`integrations/gen_integrations.py`)

38 +

39 +There are typically 3-5 places that need updating:

40 +

41 +- `COLLECTOR_SOURCES` / equivalent type-specific source list (or add a new constant `FLOW_SOURCES`).

42 +- Schema dispatch (the function that picks which schema to validate against based on file location or `integration_type`).

43 +- The render-keys table — which sections (overview, setup, troubleshooting, alerts, metrics, ...) the type renders.

44 +- The categorisation in `integrations.js` — making sure the new type ends up in the flat `integrations` array with a correct `integration_type` field.

45 +

46 +### 3. Pipeline rendering (`integrations/gen_docs_integrations.py`)

47 +

48 +Add a branch for the new type in the main loop. The branch typically:

49 +

50 +- Calls a new `mode="<type>"` variant of `build_readme_from_integration()`.

51 +- Sets the `learn_rel_path` for the type. Two patterns exist:

52 + - **Hardcoded** (like `exporter`, `secretstore`, `service_discovery`): one fixed path for all entries of this type.

53 + - **Derived** (like `collector`, `logs`, `agent_notification`, `cloud_notification`, `authentication`): walk the category tree in `integrations.js`, then `replace()` the top-level category name with the Learn section prefix.

54 +- Calls a new `mode="<type>"` variant of `write_to_file()` that writes the per-integration `.md` to its conventional location.

55 +- Optionally invokes the symlink helper if the integration directory pattern matches single-integration plugins (`make_symlinks`).

56 +

57 +Cite `gen_docs_integrations.py` line numbers for each existing type (logs, exporter, etc.) when adding a new branch — they are the authoritative templates.

58 +

59 +### 4. Render templates (`integrations/templates/`)

60 +

61 +Add a `<type>.md` overview template under `integrations/templates/overview/` (or alongside the other type-specific templates). Reuse `setup-generic.md` if your type's setup is generic; otherwise add a `setup-<type>.md`.

62 +

63 +Templates use the customised Jinja2 delimiters `[[ ]]` and `[% %]` (see `gotchas.md`). Cargo-cult from the closest existing type's templates and adjust.

64 +

65 +### 5. Top-level category (`integrations/categories.yaml`)

66 +

67 +Add a top-level category node for the new type. For a flat structure (like `logs`):

68 +

69 +```yaml

70 +- id: <type>

71 + name: <Display Name>

72 + description: ""

73 + children: []

74 +```

75 +

76 +For a sub-categorised structure (like `data-collection`):

77 +

78 +```yaml

79 +- id: <type>

80 + name: <Display Name>

81 + description: ""

82 + children:

83 + - id: <type>.<sub>

84 + name: <Sub Display>

85 + description: ""

86 + children: []

87 +```

88 +

89 +Default to flat. Sub-categories only justify themselves when the count grows (collectors are sub-categorised because there are 488 of them; logs has 3 entries and stays flat).

90 +

91 +### 6. Source metadata declarations

92 +

93 +For each new integration of the type, the source `metadata.yaml` (or equivalent) carries:

94 +

95 +- `integration_type: <type>`

96 +- `categories: [<type>]` (or `[<type>.<sub>]` for the sub-categorised case)

97 +- The other fields the schema requires.

98 +

99 +If the type uses a single shared metadata file (like `integrations/logs/metadata.yaml`), declare all entries there. If the type uses per-plugin or per-module metadata files (like collectors), each lives next to its own source.

100 +

101 +### 7. Learn site map (`docs/.map/map.yaml`)

102 +

103 +Add a section node with an `integration_placeholder`:

104 +

105 +```yaml

106 +- meta:

107 + label: <Display Name>

108 + items:

109 + - type: integration_placeholder

110 + integration_kind: <kind>

111 +```

112 +

113 +The `<kind>` value is consumed by Learn's ingest to filter pages by their `learn_rel_path`. Keep it singular or plural to match existing conventions (`logs`, `collectors`, `exporters`, `agent_notifications`, `cloud_notifications`, `authentication`, `secretstore`, `service_discovery`).

114 +

115 +If the type has hand-authored content alongside the catalogue entries (like the rich `docs/network-flows/` documentation that pairs with the `flows` integration type), add explicit `meta.edit_url` rows for those pages and put the `integration_placeholder` at the appropriate position in the sub-tree.

116 +

117 +### 8. Downstream consumers

118 +

119 +Three downstream repos may need touching, in roughly decreasing likelihood:

120 +

121 +- **`netdata/website`** (`~/src/netdata/website`): the daily `update-integrations.yml` workflow renders marketing cards from `integrations.json`. Cards usually appear automatically. But pages that reference the type explicitly (FAQ entries, solution pages) may need rewriting if the new type changes the story (e.g., "we don't do flows" → "we do flows natively").

122 +- **`netdata/learn`** (`~/src/netdata/learn`): no PR usually needed. Learn ingest reads `map.yaml` + the `` markers in the generated `.md` files. Sidebar regenerates automatically. ~3 hours after netdata-repo merge.

123 +- **`netdata/dashboard/cloud-frontend`** (`~/src/dashboard/cloud-frontend`): no PR usually needed. The catalog rendering in `src/domains/integrations/` is fully data-driven from `integrations.js`. New `integration_type` values appear automatically. Special UI (a dedicated tab elsewhere in the dashboard) is a separate concern.

124 +

125 +## Verification checklist

126 +

127 +After all the changes:

128 +

129 +1. `python3 integrations/gen_integrations.py` — exits 0, regenerates `integrations.js` and `integrations.json`. The new type appears in the flat list, in the new top-level category, with the correct count.

130 +2. `python3 integrations/gen_docs_integrations.py` — exits 0, generates per-integration `.md` files. Each has `` with the correct `learn_rel_path`.

131 +3. Manual inspection of one generated `.md` — frontmatter complete (`custom_edit_url`, `meta_yaml`, `sidebar_label`, `learn_status: Published`, `learn_rel_path`, `keywords`, `message: "DO NOT EDIT..."`).

132 +4. `grep` the categories tree in `integrations.json` — the new top-level node exists with the new entries.

133 +5. Open `learn.netdata.cloud` after the next ingest cycle (~3 hours after merge) — confirm the new section renders in the sidebar.

134 +6. Open the in-app integrations catalog — confirm the new top-level filter appears in the sidebar with the correct entries.

135 +

136 +## What to commit, and where

137 +

138 +- New schema + pipeline branch + template + categories.yaml + map.yaml + skill update — one PR in the netdata repo.

139 +- Source `metadata.yaml` for the new integrations — same PR if they live in the netdata repo (most types do).

140 +- Website FAQ / solution-page corrections — separate PR in the website repo.

141 +- No PR needed in learn or cloud-frontend (auto-discovers).

142 +

143 +## How I figured this out

144 +

145 +Spawned an Explore subagent to:

146 +

147 +- Read the `` blocks of one example file per existing integration type.

148 +- Trace `learn_rel_path` computation through `gen_docs_integrations.py` (lines 199-393).

149 +- Locate `integration_kind` placeholders in `docs/.map/map.yaml`.

150 +- Confirm `deploy` has no Learn rendering branch.

151 +

152 +Then wrote this how-to before implementing flows so the future work can follow the recipe instead of re-deriving it.

.agents/skills/integrations-lifecycle/per-type-matrix.md

+33

@@ -44,6 +44,39 @@ output filename uses the function slug from the URL stem

44 avoid collisions when many integrations share a "Top Queries"

45 label.

47 +## Learn section per integration type

48 +

49 +Every `integration_type` lands in its **own dedicated section** of `learn.netdata.cloud`. The pages are NOT jumbled together — each type has a unique `integration_kind` placeholder in `docs/.map/map.yaml` and a unique `learn_rel_path` produced by `gen_docs_integrations.py`.

50 +

52 +|---|---|---|---|

53 +| `collector` | `Collecting Metrics/Collectors/<sub-category>` | derived from `meta.monitored_instance.categories[0]` then `replace("Data Collection", "Collecting Metrics/Collectors")` (`gen_docs_integrations.py:199-201`) | `integration_kind: collectors` (under `Collecting Metrics`) |

55 +| `agent_notification` | `Alerts & Notifications/Notifications/Agent Dispatched Notifications` (hardcoded suffix on derived path) | derived + `replace("notifications", "Alerts & Notifications/Notifications")` (`gen_docs_integrations.py:259-268`) | `integration_kind: agent_notifications` |

56 +| `cloud_notification` | `Alerts & Notifications/Notifications/Centralized Cloud Notifications` (hardcoded suffix on derived path) | same pattern (`gen_docs_integrations.py:286-295`) | `integration_kind: cloud_notifications` |

58 +| `authentication` | `Netdata Cloud/Authentication & Authorization/Cloud Authentication & Authorization Integrations` (hardcoded suffix) | derived + `replace("authentication", "Netdata Cloud/...")` (`gen_docs_integrations.py:338-347`) | `integration_kind: authentication` (under `Netdata Cloud`) |

62 +| `categories` / `distros` / `shared` | n/a — meta-types | n/a | n/a |

63 +

64 +Two routing styles in the pipeline:

65 +

66 +- **Hardcoded** (`exporter`, `secretstore`, `service_discovery`, plus the suffix portion of the `*_notification` / `logs` / `authentication` paths): the Learn section is fixed by the pipeline. Categories declared on the integration are ignored for routing purposes.

67 +- **Derived from `meta.monitored_instance.categories`** (`collector`) or from `meta.categories` (`agent_notification`, `cloud_notification`, `logs`, `authentication`): the pipeline walks the category tree in `integrations.js` and converts the dotted ID into a `learn_rel_path`. Only `collector` is sub-categorised on Learn (Databases, Networking, Web Servers, etc.); the other "derived" types use their derived path with a fixed prefix.

68 +

69 +The mechanism in `map.yaml`: each type has a node like

70 +

71 +```yaml

72 +- type: integration_placeholder

73 + integration_kind: <kind>

74 +```

75 +

76 +When Learn ingest runs, every page whose `` block has a `learn_rel_path` matching that section gets attached under the placeholder. Sidebar order under each section is alphabetical by sidebar_label.

77 +

78 +`deploy` is the deliberate exception: deploy entries exist in `integrations.js` (consumed by the in-app "Add Nodes" dialog) but `gen_docs_integrations.py` never emits Learn pages for them — there is no per-distro Learn page like `Linux/Ubuntu` rendered from `deploy.yaml`.

79 +

80 ## Single-integration symlink rule

82 `gen_docs_integrations.py:make_symlinks` (`:527-544`):

.agents/skills/learn-site-structure/mapping.md

+26

@@ -213,3 +213,29 @@ between ingest's filesystem walk and `map.yaml`.

213

214 If a source file's edit URL doesn't match any `map.yaml` row,

215 the file is skipped. No warning -- silent skip.

216 +

217 +## Inter-page linking

218 +

219 +Markdown links between Learn pages must use the **GitHub-relative

220 +path with `.md` extension** — NOT the learn.netdata.cloud URL,

221 +and NOT a bare slug without extension. The ingest pipeline

222 +rewrites these links to the correct Learn URLs during processing.

223 +

224 +**Correct:**

225 +

226 +```markdown

227 +See [Configuration](/docs/network-flows/configuration.md) for details.

228 +```

229 +

230 +**Wrong (will NOT resolve on Learn or GitHub):**

231 +

232 +```markdown

233 +See [Configuration](/network-flows/configuration) for details.

234 +```

235 +

236 +The link path is the repo-relative path to the source `.md` file,

237 +prefixed with `/docs/`. The ingest pipeline matches it against the

238 +`map.yaml` tree and rewrites it to the final Learn URL.

239 +

240 +This also means links work natively on GitHub — readers browsing

241 +the source repo can click through to the linked file.

.agents/sow/done/SOW-0014-20260506-netflow-sflow-ipfix-documentation-guide.md new

+591

@@ -0,0 +1,591 @@

1 +# SOW-0014 - Network Flows Documentation & Integration Infrastructure

2 +

3 +## Status

4 +

5 +Status: completed

6 +

7 +Closed 2026-05-07. All acceptance criteria delivered or explicitly rejected. Phase 1C (AI skills cross-links) was rejected as not mandatory by user decision -- recorded in Validation. Architectural pivot to a top-level `flows` integration_type with 14 cards is recorded as Decision 10. Companion changes shipped in netdata PR #22439, learn PR #2854 (merged), and a deferred cloud-frontend `Update integrations.js` refresh.

8 +

9 +## Requirements

10 +

11 +### Purpose

12 +

13 +Provide DevOps/SREs a complete, authoritative documentation and integration infrastructure for Netdata's network flow analysis. Documentation covers only tested and verified features. Untested features (BMP, BioRIS, Network Sources, Topology drilldown) are deferred to a follow-up SOW after testing/validation. The documentation must include sizing, capacity planning, and optimization guidance so businesses can make informed deployment decisions.

14 +

15 +### User Request

16 +

17 +Phase 1: Document tested features only. Include sizing/benchmarking/capacity planning.

18 +Phase 2 (follow-up SOW): Test BMP, BioRIS, Network Sources, Topology drilldown, then document them.

19 +

20 +### Assistant Understanding

21 +

22 +Facts:

23 +

24 +- **Tested features** (unit tests + benchmarks exist):

25 + - Core collection: NetFlow v5/v7/v9, IPFIX, sFlow decoding and ingestion

26 + - Basic enrichment: GeoIP (MMDB), static metadata (exporter/interface naming), sampling overrides, static networks, ASN provider chains

27 + - Classifiers: Akvorado-compatible exporter + interface classification rules (30+ unit tests)

28 + - Decapsulation: SRv6, VXLAN modes

29 + - Journal tiering: raw/1m/5m/1h with retention and query guardrails

30 + - Query engine: flows/autocomplete modes, all views, group_by, selections, facets

31 + - Frontend: 6-tab visualization (Sankey, Timeseries, Country/State/City Maps, Globe), dashboard cards, filters/facets

32 +

33 +- **Untested features** (unit tests for parsing exist, but no integration/e2e tests, never validated with real data):

34 + - **BMP listener**: TCP listener accepting BMP from real routers, populating routing trie from live BGP sessions. 11 unit tests for message parsing, but TCP listener never tested with a real BMP speaker. Performance impact on netflow ingest path unknown.

35 + - **BioRIS**: gRPC client for RIPE RIS via bio-rd. 6 unit tests for proto conversion, but never connected to a real RIS endpoint. Can be tested locally (build `cmd/ris/` from bio-rd + BMP speaker), but this setup has never been done.

36 + - **Network Sources**: HTTP-fetched prefix metadata with jq transforms. 12 unit tests for transform/decode, but HTTP fetch cycle never tested with a real endpoint.

37 + - **Topology drilldown**: Frontend hook `useFlowsDrilldownData` is dead code (never imported), no "Flows" tab in topology actor modal. Not an untested feature -- it simply does not exist.

38 +

39 +- **Benchmark data — README is severely stale**:

40 + - The README block at `src/crates/netflow-plugin/README.md:309-367` was captured before recent ingest-path optimizations and is no longer representative. It claimed single-core saturation at ~5.8-6.3k flows/s. After optimizations, the plugin saturates at ~49k flows/s low-cardinality and ~43k flows/s high-cardinality on the same workstation (i9-12900K + FireCuda 530, ext4). The "single core" framing is also misleading — the post-decode ingest path is multi-threaded; `cpu_percent_of_one_core` accumulates user+system ticks across all threads divided by wall time, so a saturated host reports >100%.

41 + - Fresh benchmarks must be run (Phase 1.0) before documentation is written. README must be rewritten with the new numbers as part of Phase 1.0.

42 + - Benchmark commands shipped with the plugin for re-running on target hardware (`bench_resource_envelope_matrix`, `bench_ingestion_protocol_matrix`, `bench_ingestion_cardinality_matrix`).

43 +

44 +- **BMP industry survey** (from mirrored repos):

45 + - Combined approach (Akvorado, nProbe): BMP + flow enrichment in one binary, in-memory RIB, enrichment only

46 + - Separate approach (pmacct, GoBMP, OpenBMP): dedicated BMP daemon with DB/persistence, for full BGP monitoring

47 + - Netdata's implementation follows the Akvorado pattern (enrichment only, in-memory, no persistence)

48 + - Decision: BMP in netflow-plugin is for enrichment only. Full BGP monitoring per-route would require a separate plugin with its own DB. This is a future consideration, not a Phase 1 concern.

49 +

50 +- **All enrichment is in-memory, zero per-flow cost**: GeoIP, static networks, routing trie, network sources -- all background-fetched/stored, pure RwLock read per flow. No HTTP/BGP per flow.

51 +

52 +Current gaps (for Phase 1):

53 +

54 +- Zero end-user documentation on learn.netdata.cloud

55 +- No metadata.yaml for any flow protocol

56 +- No config_schema.json, no health.d/ alerts, no docs/.map/map.yaml entry

57 +- Dead redirect in LegacyLearnCorrelateLinksWithGHURLs.json

58 +

59 +### Acceptance Criteria

60 +

61 +Phase 1 only:

62 +- `metadata.yaml` with 3 modules (netflow, ipfix, sflow) validated against integrations schema

63 +- Integrations pipeline generates per-protocol pages (in-app catalog, COLLECTORS.md, learn)

64 +- Learn section "Network Flows" in `docs/.map/map.yaml`

65 +- All pages follow style guide: second person, active voice, sentence case, `:::type`/`:::` admonitions

66 +- Complete field reference (89+2 fields) with per-protocol availability matrix

67 +- Enrichment docs for all features (GeoIP, static metadata, sampling, static networks, classifiers, ASN resolution, BMP routing, BioRIS, Network Sources, decapsulation)

68 +- No mention of topology drilldown, pcap, eBPF, or threat analytics anywhere

69 +- Sizing/capacity planning page sourced from FRESH benchmark measurements (not the stale README) covering NetFlow v9, IPFIX, sFlow at 10 offered rates from 100 to 60000 flows/s, low- and high-cardinality, full pipeline (all-tiers-batched). Includes storage estimation formulas, memory guidance, optimization tips, and a clear note on multi-thread CPU semantics.

70 +- Visualization docs split into focused pages

71 +- Screenshots embedded via GitHub URLs (provided by user)

72 +- AI skills updated with links to learn docs

73 +

74 +## Implications And Decisions

75 +

76 +### Decision 1: plugin_name -- DECIDED: netflow-plugin

77 +

78 +### Decision 2: metadata.yaml scope -- DECIDED: minimal per module

79 +

80 +Identity, overview, protocol-specific router config examples, quirks. Deep details in learn section.

81 +

82 +### Decision 3: Learn section position -- DECIDED: new top-level "Network Flows"

83 +

84 +### Decision 4: Page structure -- DECIDED: more pages, small and focused

85 +

86 +```

87 +Network Flows/

88 + Overview/

89 + Quick Start/

90 + Sources/

91 + NetFlow/

92 + IPFIX/

93 + sFlow/

94 + Configuration/

95 + Enrichment/

96 + GeoIP/

97 + Static Metadata/

98 + Classifiers/

99 + ASN Resolution/

100 + BMP Routing/

101 + BioRIS/

102 + Network Sources/

103 + Decapsulation/

104 + Field Reference/

105 + Visualization/

106 + Summary and Sankey/

107 + Time-Series/

108 + Maps/

109 + Globe/

110 + Filters and Facets/

111 + Dashboard Cards/

112 + Retention and Querying/

113 + Sizing and Capacity Planning/

114 + Troubleshooting/

115 +```

116 +

117 +~24 pages. Each page focused on one topic. No mentions of pcap, eBPF, threat analytics, or topology drilldown. BMP, BioRIS, and Network Sources are documented based on unit-tested parsing/conversion logic; their runtime I/O paths (TCP listener, gRPC client, HTTP fetch) lack integration tests.

118 +

119 +### Decision 5: Future features -- DECIDED: document BMP/BioRIS/Network Sources, skip topology drilldown

120 +

121 +BMP, BioRIS, and Network Sources are included in documentation. Their unit-tested parsing logic is solid but runtime I/O paths lack integration tests. The SOW follow-up section tracks the need for async/integration tests.

122 +

123 +Topology drilldown remains excluded (dead code -- `useFlowsDrilldownData.js` never imported). pcap, eBPF, and threat analytics remain excluded (not implemented).

124 +

125 +### Decision 6: AI skills -- DECIDED: update existing query skills

126 +

127 +### Decision 7: Integrations pipeline -- DECIDED: add netflow-plugin to gen_integrations.py

128 +

129 +### Decision 9: Documentation rewrite directives -- DECIDED 2026-05-07

130 +

131 +User read parts of the research at `.agents/knowledge/Network Traffic Analysis with Flow Data.md`, taught the dashboard mechanics directly, corrected several misconceptions (notably the doubling/mirroring foundational concept and the function-permission paid-only assumption), and instructed:

132 +

133 +- **Rewrite, not edit.** Existing pages are thin and must not be inherited as correct. Every claim is re-verified against the code.

134 +- **Code is the source of truth.** The existing markdown, comments, and prior docs are reference points, not authority.

135 +- **Audience-aware, never condescending.** Documentation must serve newcomers without insulting experts. Mental models must be established before features.

136 +- **Three roles served in parallel:** Network Engineer, Security Analyst, IT Manager. Pages stay role-agnostic but every page must be useful to all three.

137 +- **Doubling and mirroring is foundational, not an aside.** Users must understand it before reading any aggregate number.

138 +- **Sampling is documented honestly.** Auto-multiplied at ingestion; mixed rates make aggregates uninterpretable; admins should keep rates uniform or run unsampled.

139 +- **Installation is a first-class topic.** The plugin is packaged separately (`netdata-plugin-netflow` on both DEB and RPM via `netdata.spec.in:3270` and `Packaging.cmake:488`). Not auto-installed by the netdata-updater. Users must install it themselves on native-package systems. Static installs (`kickstart.sh --static-only`) bundle it.

140 +- **Anti-patterns get a dedicated page.** The research's most common deployment failures (ignored sampling, alert on absolute volume, GeoIP firewall of shame, NAT blindness, double-counting, treating duration as latency, microburst hunting) are called out explicitly.

141 +- **Investigation playbooks get a dedicated page.** Concrete walkthroughs from the research's recognition cues (bandwidth saturation, IP investigation, capacity justification, security alert scope) — written for the Netdata UI specifically (Top-N + facets + sankey + maps).

142 +- **Validation & data quality gets a dedicated page.** SNMP cross-check, exporter health monitoring, doubling check, sampling sanity check.

143 +- **No mechanical writing.** Each page begins with code research, an explicit audience statement, an explicit goal statement, and 3-5 key takeaways. Page produced only after the model is settled.

144 +

145 +**Scope restored** (changed from earlier interpretation): BMP, BioRIS, and Network Sources are documented based on their unit-tested parsing logic (per Decision 5). Their runtime I/O paths lack integration tests but the features exist in the code and ship in user configurations. Users would otherwise see references to them in `netflow.yaml` without explanation. The follow-up still tracks the integration-test gap.

146 +

147 +**Scope confirmed deferred:** topology drilldown (`useFlowsDrilldownData.js`) is dead code — never imported into the topology actor modal — and stays out of documentation until implemented.

148 +

149 +**Function permissions verified** at `src/crates/netflow-plugin/src/api/flows/handler.rs:263`: the `flows:netflow` function uses `HttpAccess::SIGNED_ID | SAME_SPACE | SENSITIVE_DATA`. It does NOT include `COMMERCIAL_SPACE`. The agent function is therefore not paid-gated. Any feature gating elsewhere (UI/cloud-frontend) is outside this repository and outside this SOW.

150 +

151 +### Decision 10: Promote `flows` to a top-level integration_type with 14 cards -- DECIDED 2026-05-07

152 +

153 +User reframed the test for "should X be an integration card?" as "would users ask 'Does Netdata integrate with X?'". This rejected the original architecturally pure framing (one card per protocol decoder under data-collection.networking) in favour of one card per vendor/source users actually shop for.

154 +

155 +Sub-decisions:

156 +

157 +- **10a Type position:** `flows` is a top-level `integration_type` in `integrations/categories.yaml`, peer of `collector`, `logs`, `exporter`, `notification`, `secretstore`, `authentication`. NOT under `data-collection.networking`. Rationale: flows are a different data model from collector metrics (table-shaped, faceted, time-windowed, journal-backed), the same way logs are -- their integration UX should mirror logs, not collectors.

158 +- **10b Catalog shape:** four sub-categories under `flows` -- Sources, IP Intelligence, BGP Routing, Network Identity Sources -- containing 14 cards total:

159 + - Sources (3): NetFlow, IPFIX, sFlow

160 + - IP Intelligence (4): DB-IP IP Intelligence (default), MaxMind GeoIP / GeoLite2, IPtoASN, Custom MMDB Database

161 + - BGP Routing (2): BMP (BGP Monitoring Protocol), bio-rd / RIPE RIS

162 + - Network Identity Sources (5): AWS IP Ranges, GCP IP Ranges, Azure IP Ranges, NetBox, Generic JSON-over-HTTP IPAM

163 +- **10c Pipeline plumbing:** `integrations/gen_integrations.py` and `integrations/gen_docs_integrations.py` learn the new type via FLOWS_SOURCES list, FLOWS_RENDER_KEYS, FLOWS_VALIDATOR, `load_flows()`, `render_flows()`, plus a `mode == "flows"` branch in `build_readme_from_integration()`. New `integrations/templates/overview/flows.md` template. New `integrations/schemas/flows.json` ($ref to `collector.json` for now -- can diverge later).

164 +- **10d Map placement:** `docs/.map/map.yaml` carries a top-level Network Flows section using `integration_placeholder` with `integration_kind: flows`, plus three concept docs (IP Intelligence, BGP Routing, Network Identity).

165 +- **10e Schema:** `docs/.map/map.schema.json` enum extended to accept `flows` as an integration_kind value alongside the existing kinds.

166 +- **10f Cross-repo coordination:** companion learn PR (netdata/learn#2854) routes `src/crates/netflow-plugin/integrations/<slug>.md` files into a `flows_entries` DataFrame and splices them over the `flows_integrations` placeholder, mirroring the existing `logs_integrations` handler. cloud-frontend's `data/integrations.js` is auto-generated and committed manually -- a follow-up "Update integrations.js" PR is needed there. www auto-updates daily; no PR required.

167 +- **10g Skill knowledge capture:** `integrations-lifecycle` skill gains a new per-integration_type Learn-routing matrix in `per-type-matrix.md` and a how-to `adding-new-integration-type.md` enumerating the 8-place checklist for future kinds.

168 +

169 +### Decision 8: Benchmark rerun matrix -- DECIDED 2026-05-06

170 +

171 +Post-optimization rerun supersedes README:309-367. Sub-decisions:

172 +

173 +- **8a Protocols:** netflow-v9, ipfix, sflow. Skip netflow-v5 (legacy, unrepresentative of modern deployments).

174 +- **8b Layer:** all-tiers-batched only (full pipeline: raw + 1m + 5m + 1h). The user-facing question is "what does the plugin cost on my host?" — that is the full-pipeline number. Other layers (writer-only, raw-only, minute1-only) are engineering-internal and not republished to users.

175 +- **8c Modes:** both. (i) Paced post-decode resource envelope (`bench_resource_envelope_*`) — produces sizing curves at exact offered rates. (ii) Unpaced full UDP→journal (`bench_ingestion_protocol_matrix`) — produces per-protocol max-throughput numbers including the decode cost. Mode (i) drives the sizing/capacity page; mode (ii) is supplementary and shows decode-path differences between protocols.

176 +- **8d Rate matrix:** 10 offered rates: 100, 500, 1000, 5000, 10000, 20000, 30000, 40000, 50000, 60000 flows/s. The host saturates at ~49k/s low-cardinality and ~43k/s high-cardinality, so the upper end intentionally exercises the saturation plateau. Three protocols × two cardinalities × ten rates = 60 paced cells. Plus 3 unpaced protocol cells × 3 phases (full / decode-only / post-decode) = 9 unpaced cells.

177 +- **8e Hardware:** this workstation. CPU `12th Gen Intel(R) Core(TM) i9-12900K`, storage `Seagate FireCuda 530 NVMe`, ext4. Same rig as the original README capture; comparable baseline.

178 +- **8f Output format and location:** shell driver writes one `RESOURCE_BENCH_RESULT:{json}` line per cell to `<repo>/.local/audits/netflow-bench/results.jsonl`. Renderer produces a markdown summary table per (protocol, cardinality) under the same directory. `.local/` is gitignored — raw outputs are not committed; only the SOW + README + sizing doc carry numbers into the repository.

179 +

180 +CPU semantics note (will appear in docs):

181 +- `cpu_percent_of_one_core` accumulates user+system ticks across all threads divided by wall time

182 +- Below saturation it can be <100%

183 +- At saturation on a multi-core host it is well above 100% (e.g. ~600-800% on this workstation when ingest threads + tier-batch threads are all busy)

184 +- Documentation must call this out explicitly so capacity planners read the metric correctly

185 +

186 +## Pre-Implementation Gate

187 +

188 +Status: ready

189 +

190 +Problem / root-cause model:

191 +

192 +- Netdata has a fully functional flow collection and visualization system with tested features, but zero documentation and zero integration catalog entries. Four features (BMP, BioRIS, Network Sources, Topology drilldown) are untested or not implemented and must be excluded from documentation until validated.

193 +

194 +Evidence reviewed:

195 +

196 +- All files listed in previous SOW versions

197 +- Benchmark data: `src/crates/netflow-plugin/README.md:309-367` is STALE (post-optimization saturation ~49k/43k, not ~6k as the README claims). Authoritative numbers come from the Phase 1.0 rerun (matrix per Decision 8). README:309-367 must be replaced as part of Phase 1.0.

198 +- Test coverage: 81 enrichment tests, 11 BMP parsing tests, 6 BioRIS unit tests, 12 network source unit tests -- but zero integration tests for BMP TCP listener, BioRIS gRPC client, or network source HTTP fetch

199 +- BMP industry survey: 14 repos analyzed across mirrored codebase

200 +- BioRIS local testing: possible by building `cmd/ris/` from bio-rd, never done

201 +- Topology drilldown: `useFlowsDrilldownData.js` never imported (dead code)

202 +- Documentation style guide: `docs/developer-and-contributor-corner/style-guide.md` (second person, active voice, sentence case, Oxford comma, `:::type`/`:::` admonitions)

203 +

204 +Affected contracts and surfaces:

205 +

206 +- `docs/.map/map.yaml` -- new section

207 +- `src/crates/netflow-plugin/metadata.yaml` -- new file

208 +- `integrations/gen_integrations.py` -- plugin_name registration

209 +- Generated integration pages, COLLECTORS.md

210 +- `docs/netdata-ai/skills/query-netdata-cloud/query-flows.md` -- updated

211 +- `docs/netdata-ai/skills/query-netdata-agents/query-flows.md` -- updated

212 +- ~17 new learn pages under `docs/network-flows/`

213 +- `src/crates/netflow-plugin/src/ingest_resource_bench_tests.rs` -- add `NETFLOW_RESOURCE_BENCH_PROTOCOL` env var (Phase 1.0 step 1)

214 +- `src/crates/netflow-plugin/README.md:309-367` -- benchmark block fully replaced with fresh post-optimization numbers (Phase 1.0 step 4)

215 +- `<repo>/.local/audits/netflow-bench/` -- new gitignored output directory for raw JSONL + rendered markdown (Phase 1.0 step 2)

216 +

217 +Existing patterns to reuse:

218 +

219 +- Style guide conventions

220 +- Existing collector metadata.yaml (SNMP, PostgreSQL) as templates

221 +- Integrations-lifecycle skill, learn-site-structure skill

222 +- `src/crates/netflow-plugin/README.md` as source material

223 +- `query-flows.md` skills as source for API examples

224 +

225 +Risk and blast radius:

226 +

227 +- Documentation-only (except gen_integrations.py plugin_name addition + the small bench harness env-var addition + README replacement)

228 +- Must carefully avoid documenting untested features -- all enrichment docs must be scoped to tested features only

229 +- BMP routing, BioRIS, and Network Sources exist in config examples (netflow.yaml) but must not appear in documentation as usable features until tested

230 +- Bench harness change is additive (new env var, fall-back behavior preserved). No existing test or caller is affected. Verify by running one cell with each protocol value before launching the full matrix.

231 +- Benchmark rerun consumes the workstation for ~25 min wall time (60 paced cells x ~20s + 9 unpaced cells). Host should be quiet during the run; concurrent CPU load skews CPU% and write_bytes/s readings.

232 +

233 +Sensitive data handling plan:

234 +

235 +- Use `NODE` placeholder instead of real IPs (per style guide)

236 +- Use RFC 5737 documentation ranges in all examples

237 +- No real credentials, tokens, or network topologies

238 +

239 +Implementation plan:

240 +

241 +1. Phase 1.0: Benchmark rerun (matrix per Decision 8) and README replacement

242 +2. Phase 1A: Integration infrastructure (metadata.yaml + pipeline)

243 +3. Phase 1B: Learn section (map.yaml + ~17 pages, with sizing page sourced from Phase 1.0 fresh measurements)

244 +4. Phase 1C: AI skills update

245 +5. Phase 1D: Validation

246 +

247 +Validation plan:

248 +

249 +- Run integrations pipeline, verify artifacts

250 +- Verify map.yaml routing

251 +- Cross-reference every config option against plugin_config.rs

252 +- Cross-reference field list against flow/schema.rs

253 +- Cross-reference benchmark numbers against the Phase 1.0 fresh rerun output (`<repo>/.local/audits/netflow-bench/results.jsonl`), NOT against the stale README. README is a downstream artifact updated by Phase 1.0.

254 +- Verify no mention of untested features anywhere

255 +

256 +Artifact impact plan:

257 +

258 +- AGENTS.md: no update needed

259 +- Runtime project skills: no update needed

260 +- Specs: no spec update needed

261 +- End-user/operator docs: this IS the docs update

262 +- End-user/operator skills: query skills updated (Phase 1C)

263 +- SOW lifecycle: standard flow

264 +

265 +Open decisions:

266 +

267 +- None. All resolved.

268 +

269 +## Plan

270 +

271 +### Phase 1.0: Benchmark Rerun + README Refresh

272 +

273 +1. Add `NETFLOW_RESOURCE_BENCH_PROTOCOL` env var to `src/crates/netflow-plugin/src/ingest_resource_bench_tests.rs`. When set to `netflow-v9`, `ipfix`, or `sflow`, route `build_record_batches` to the matching entry in `PROTOCOL_SCENARIOS`. When unset, fall back to the existing `CARDINALITY_SOURCE_SCENARIO` mixed behavior so legacy callers are unaffected. Pass the new env var through `run_resource_envelope_case` to the child process.

274 +2. Write `<repo>/.local/audits/netflow-bench/run.sh` driver script:

275 + - Iterate protocols `{netflow-v9, ipfix, sflow}` x cardinalities `{low, high}` x rates `{100, 500, 1000, 5000, 10000, 20000, 30000, 40000, 50000, 60000}`.

276 + - Per cell: run `cargo test -p netflow-plugin --release ingest::resource_bench_tests::bench_resource_envelope_child -- --ignored --nocapture --exact --test-threads=1` with env vars `NETFLOW_RESOURCE_BENCH_CHILD=1`, `NETFLOW_RESOURCE_BENCH_LAYER=all-tiers-batched`, and the per-cell protocol/profile/rate.

277 + - Capture the `RESOURCE_BENCH_RESULT:{json}` line into `results.jsonl` (one line per cell, including a `protocol` field).

278 + - Run `bench_ingestion_protocol_matrix` once for the unpaced 3B output; capture stderr report blocks into `protocol_matrix.txt`.

279 + - Render per-(protocol,cardinality) markdown tables (`results.md`).

280 + - Use the `run()` visibility wrapper from `~/.claude/CLAUDE.md` so commands are echoed.

281 +3. Verify host quietness before running (`top` / `ps`). Re-run any cell that fails or that shows pacer underrun for spurious reasons.

282 +4. Replace `src/crates/netflow-plugin/README.md:309-367` with the fresh numbers, including:

283 + - Per-protocol per-cardinality tables for the 10 rates (3A).

284 + - Per-protocol max-throughput table from 3B.

285 + - Multi-thread CPU semantics note.

286 + - Storage estimation hints derived from `write_bytes_per_sec` and `logical_write_bytes_per_sec` ratios.

287 +

288 +### Phase 1A: Integration Infrastructure

289 +

290 +1. Add `netflow-plugin` to `gen_integrations.py` recognized plugin names

291 +2. Create `src/crates/netflow-plugin/metadata.yaml` with 3 modules

292 +3. Create icon assets

293 +4. Run pipeline, verify generated pages

294 +

295 +### Phase 1B: Learn Section (full rewrite -- ~25 pages -- per Decision 9)

296 +

297 +Per-page method, applied uniformly:

298 +

299 +1. **Subagent for technical analysis first.** Each feature/section gets a dedicated read-only subagent that explores the relevant code under `src/crates/netflow-plugin/` (and adjacent crates / Go tools where applicable), identifies the happy path and the nuances, enumerates configuration options with their defaults, traces error paths, and returns a structured analysis with file:line citations. The master assistant synthesizes the doc from the analysis. This keeps raw code-reading noise out of the master context and ensures each page is grounded.

300 +2. **Bottom-up writing order.** Detail pages are written first; the `Overview` is written last as the natural index over already-established truths. This avoids the trap of writing an Overview that promises behavior the detail pages later contradict.

301 +3. **State audience and goal per page.** Each page begins with an internal "audience / goal / key takeaways" plan recorded in the SOW execution log before the page is written.

302 +4. **Verify each claim.** No statement reaches the page without a file:line citation in the analysis returned by the subagent.

303 +5. **Mental model before features.** Doubling/mirroring, sampling semantics, tier transparency are foundational and explained before any aggregate number is shown.

304 +6. **Three audiences served.** Network Engineer / Security Analyst / IT Manager all find their answers without the page wearing a role label.

305 +

306 +Bottom-up writing order:

307 +

308 +Tier A -- leaves (independent, parallelisable subagents):

309 +- `sources/netflow.md`, `sources/ipfix.md`, `sources/sflow.md` -- per-protocol decoder analysis

310 +- `configuration.md` -- whole `plugin_config/` schema

311 +- `field-reference.md` -- `flow/` schema enumeration

312 +- `retention-querying.md` -- tier model + query tier-selection logic

313 +- 8 enrichment pages -- each enrichment module under `src/crates/netflow-plugin/src/enrichment/` and runtime initialization

314 +- 5 visualization pages -- UI behavior plus the corresponding query/response paths under `src/crates/netflow-plugin/src/api/flows/` and `query/`

315 +- `troubleshooting.md` -- error paths, exposed plugin metrics, log message inventory

316 +

317 +Tier B -- synthesis (depend on Tier A):

318 +- `anti-patterns.md` -- distilled from research §6 + Tier A findings

319 +- `validation.md` -- distilled from research §11 + Tier A operational metrics

320 +- `investigation-playbooks.md` -- four scenarios that exercise the visualizations, facets, and tiers documented in Tier A

321 +- `installation.md` -- standalone but depends on `configuration.md` for post-install verification

322 +

323 +Tier C -- top:

324 +- `quick-start.md` -- the first-time path through Tiers A and B

325 +- `README.md` -- Overview (final synthesis, the natural index)

326 +

327 +Page list (final):

328 +

329 +Root level:

330 +- `README.md` -- Overview (rewrite). Mental model, what flow data is, what it answers and what it does not, doubling/mirroring foundational concept, sampling caveat, prerequisites pointer, installation pointer, navigation.

331 +- `installation.md` -- Installation (NEW). Package names per distro, post-install verification, file locations, dependencies, source-build caveats. Confirms `netdata-plugin-netflow` is the canonical name on DEB and RPM and that it is opt-in on native-package systems.

332 +- `quick-start.md` -- Quick Start (rewrite). Three-step path, router config examples (NetFlow v9, IPFIX, sFlow), dashboard first-look, doubling/mirroring read-this-first section, verification step.

333 +- `configuration.md` -- Configuration (rewrite). Listener, protocols, journal layout and retention, decapsulation, performance tuning (UDP buffer sysctls, sync interval, record_pool_size).

334 +- `field-reference.md` -- Field Reference (rewrite). All fields organised by category, per-protocol availability matrix, usage hints per field.

335 +- `retention-querying.md` -- Retention and Querying (rewrite). Tier model, how queries auto-pick tiers, IP/port loss in tiers 1-3 and the "filter on IP forces tier 0" behavior.

336 +- `sizing-capacity.md` -- Sizing and Capacity Planning (already done in Phase 1.0).

337 +- `validation.md` -- Validation and Data Quality (NEW). SNMP cross-check, doubling sanity check, sampling sanity check, exporter health monitoring, the silent-failure list.

338 +- `investigation-playbooks.md` -- Investigation Playbooks (NEW). Walkthroughs: bandwidth saturation, IP investigation, capacity justification, security alert scope. UI-specific (Top-N, facets, sankey).

339 +- `anti-patterns.md` -- Anti-patterns and Pitfalls (NEW). Doubled aggregate, ignored sampling, GeoIP for internal IPs, absolute thresholds, collect-and-ignore, flows-vs-sessions, NAT blindness, geographic firewall of shame, duration-as-latency, microburst hunting.

340 +- `troubleshooting.md` -- Troubleshooting (rewrite). Plugin not running, no data arriving, partial data, template errors, GeoIP gaps, performance issues.

341 +

342 +Sources:

343 +- `sources/netflow.md` -- NetFlow v5/v7/v9. Protocol semantics, template lifecycle (v9), active timeout best practice, configuration examples for major vendors.

344 +- `sources/ipfix.md` -- IPFIX. Protocol semantics, IE handling, template withdrawal, biflow rarity, vendor configuration examples.

345 +- `sources/sflow.md` -- sFlow v5. Fundamentally different from NetFlow — packet samples + counter samples, sampling rate inherent, semantics of byte counts.

346 +

347 +Enrichment:

348 +- `enrichment/geoip.md` -- GeoIP via MMDB. MaxMind database management, internal-IP trap, validation.

349 +- `enrichment/static-metadata.md` -- Static metadata (exporter naming, interface descriptions, custom labels). Foundational for multi-exporter analysis.

350 +- `enrichment/classifiers.md` -- Akvorado-style classifier rules. Rule syntax, when they fire, debugging.

351 +- `enrichment/asn-resolution.md` -- ASN resolution. Provider chains, fallback, validation.

352 +- `enrichment/bmp-routing.md` -- BMP listener (Decision 5: in scope; runtime I/O lacks integration tests, documented based on parsing logic). What BMP is, what BMP enrichment provides, configuration, integration-test caveat.

353 +- `enrichment/bioris.md` -- BioRIS gRPC client (same in-scope/caveat as BMP).

354 +- `enrichment/network-sources.md` -- HTTP-fetched prefix metadata with jq transforms (same in-scope/caveat).

355 +- `enrichment/decapsulation.md` -- SRv6, VXLAN inner-packet extraction. Modes, when each applies.

356 +

357 +Visualization:

358 +- `visualization/summary-sankey.md` -- The default landing view. How to read sankey + table together, field selection, top-N control, sort by bytes/packets, doubling-when-unfiltered.

359 +- `visualization/time-series.md` -- Top-N over time. When to use, baselines, time-shifted comparison.

360 +- `visualization/maps-globe.md` -- Country/state/city maps + 3D globe. Tooltips, no drill-down, GeoIP traps.

361 +- `visualization/filters-facets.md` -- Filter ribbon + autocomplete + FTS + selections persistence + URL sharing. AND-of-ORs semantics.

362 +- `visualization/dashboard-cards.md` -- Operational metrics for the plugin itself. What to watch for plugin health.

363 +

364 +### Phase 1B (legacy, superseded by per-page list above)

365 +

366 +1. Add "Network Flows" section to `docs/.map/map.yaml`

367 +2. Clean up dead redirect in `LegacyLearnCorrelateLinksWithGHURLs.json`

368 +3. Write pages (Overview, Quick Start, Sources/NetFlow, Sources/IPFIX, Sources/sFlow, Configuration, Enrichment/GeoIP, Enrichment/Static Metadata, Enrichment/Classifiers, Enrichment/ASN Resolution, Enrichment/Decapsulation, Field Reference, Visualization/Summary+Sankey, Visualization/Time-Series, Visualization/Maps, Visualization/Globe, Visualization/Filters+Facets, Visualization/Dashboard Cards, Retention and Querying, Sizing and Capacity Planning, Troubleshooting)

369 +

370 +### Phase 1C: AI Skills Update

371 +

372 +1. Update both query-flows.md skills (links, field list, Globe view)

373 +2. Update SKILL.md files if needed

374 +3. Add how-tos for common patterns

375 +

376 +### Phase 1D: Validation

377 +

378 +1. Pipeline artifacts verified

379 +2. Map.yaml routing verified

380 +3. MDX compliance checked

381 +4. Cross-reference against source code

382 +5. Verify zero mentions of untested features

383 +

384 +## Execution Log

385 +

386 +### 2026-05-06

387 +

388 +- Created SOW from comprehensive analysis

389 +- Deep-dived into: fields, enrichment, integrations pipeline, GeoIP, BMP, BioRIS, network sources, topology drilldown, documentation style/patterns

390 +- BMP industry survey: 14 repos analyzed. Combined approach (Akvorado pattern) confirmed for enrichment use case

391 +- BioRIS: can run locally (build cmd/ris/ from bio-rd), never tested

392 +- Confirmed: topology drilldown is dead code (never imported)

393 +- Original benchmark data taken from README: 5k flows/s sustainable, ~6k saturation on i9-12900K (NOW KNOWN STALE — see entry below)

394 +- User raised testing concerns: BMP, BioRIS, Network Sources are untested, must not be documented

395 +- Agreed phased approach: Phase 1 documents tested features + sizing, Phase 2 tests + documents untested features

396 +- All 7 decisions resolved

397 +

398 +### 2026-05-06 (later) - benchmark rerun finding

399 +

400 +- Discovered the README benchmark block (`src/crates/netflow-plugin/README.md:309-367`) is severely stale: post-optimization the post-decode ingest path is much faster than the README claimed (~5.8-6.3k flows/s saturation). The "single core" framing was correct in principle (the post-decode hot path is single-threaded) but the absolute numbers were obsolete.

401 +- Inspected the existing benchmark harness: `ingest_resource_bench_tests.rs` (paced post-decode), `ingest_bench_tests.rs::bench_ingestion_protocol_matrix` (unpaced full UDP→journal), `ingest_resource_bench_support.rs` (`ResourceEnvelopeReport` records achieved flows/s, CPU%, peak/final RSS, read/write bytes/s — exactly the four metrics the user asked to record).

402 +- Confirmed the harness is configurable via env vars for cardinality (`..._PROFILE`), rate (`..._FLOWS_PER_SEC`), layer (`..._LAYER`), warmup, measure window, and pool sizes. The harness is NOT today configurable per-protocol — `bench_resource_envelope_*` always uses `CARDINALITY_SOURCE_SCENARIO` (mixed). Added `NETFLOW_RESOURCE_BENCH_PROTOCOL` env var that routes to a specific entry in `PROTOCOL_SCENARIOS` when set, falling back to the mixed default otherwise. Added `protocol` field to `ResourceEnvelopeReport` so the JSON output is self-describing.

403 +- User decided rerun matrix (Decision 8): netflow-v9+ipfix+sflow, all-tiers-batched layer, both paced + unpaced modes, 10 rates from 100 to 60000 flows/s, this workstation, JSONL+markdown output under `.local/audits/netflow-bench/`.

404 +

405 +### 2026-05-06 (later still) - benchmark rerun results

406 +

407 +Driver: `.local/audits/netflow-bench/run.sh` + `render.py`. Outputs: `results.jsonl` (60 paced cells), `protocol_matrix.txt` (3B unpaced), `results.md`. Wall time: 22 min for 3A + 5s for 3B. Host load was nominal (load average ~3.5 on 16 cores; user accepted).

408 +

409 +Key findings:

410 +

411 +- **High-cardinality saturation is around 30 000 flows/s post-decode for all three protocols.** Above the knee, achieved rate plateaus while offered grows. CPU pins at ~98-99% of one core at saturation, confirming the post-decode hot path is single-threaded (CPU does not exceed 100% even when more rate is offered).

412 +- **Low-cardinality saturation is above 60 000 flows/s** for all three protocols — never reached within the chosen matrix. CPU at 60k offered: ipfix 64.1%, netflow-v9 70.3%, sflow 87.0%. Extrapolated single-core ceiling: ~85-100k for v9/ipfix, ~70k for sflow.

413 +- **3B unpaced full UDP→journal peaks**: NetFlow v9 ~99k flows/s, IPFIX ~107k flows/s, sFlow ~88k flows/s. Decode-only is much faster (0.8-2.4M flows/s) — decode is roughly 10% of the full path cost.

414 +- **User's earlier "49k/43k cap" did not appear** in the protocol-isolated rerun. That number was from the older mixed-scenario run; protocol-isolated low-card is faster than 49k and high-card is slower than 43k (~30k).

415 +- **Practical published number**: ~20-25k flows/s for "everything except enrichment" (UDP receive + decode + 4 tiers + high cardinality, mixed protocols). 20k = conservative, 25k = optimistic. Derivation: 30k post-decode ceiling minus ~10 µs/flow for decode brings the full-path ceiling to ~22-25k. UDP socket receive at 1-5k pps is below typical socket limits and not a concern.

416 +- **CPU semantics revisited**: cpu_percent_of_one_core reports >100% only when multiple threads contribute. The plateau at 98-99% is the single-thread limit on the post-decode ingest hot path. Multi-threading work above that ceiling is a future optimization, not a current capability.

417 +

418 +Artifacts updated as part of this rerun:

419 +

420 +- `src/crates/netflow-plugin/src/ingest_resource_bench_tests.rs` — added `NETFLOW_RESOURCE_BENCH_PROTOCOL` env var and threaded it through child spawn

421 +- `src/crates/netflow-plugin/src/ingest_resource_bench_support.rs` — added `protocol` field to `ResourceEnvelopeReport`

422 +- `src/crates/netflow-plugin/README.md:309-...` — fully replaced benchmark section with fresh per-protocol/cardinality tables, 3B unpaced summary, multi-thread CPU semantics note

423 +- `docs/network-flows/sizing-capacity.md` — replaced stale numbers with fresh tables, added the 20-25k headline

424 +- `.local/audits/netflow-bench/` — driver script, renderer, raw JSONL, rendered markdown (gitignored)

425 +

426 +### 2026-05-07 - storage footprint benchmark

427 +

428 +User asked for a benchmark that measures actual on-disk storage growth, write amplification, and the dedup/cardinality effect. Replaces the "flows × bytes/flow × time" table that I had wrongly written into sizing-capacity.md (the journals are indexed and dedup-aware, so that calculation is invalid).

429 +

430 +Implementation:

431 +- New test `bench_storage_footprint_child` in `src/crates/netflow-plugin/src/ingest_resource_bench_tests.rs`. Reuses `run_paced_plugin_loop` in segments equal to the configured sample interval; between segments samples per-tier on-disk size via `journal_dir_size_bytes`, real I/O via `/proc/self/io`, logical encoded bytes via metrics deltas, RSS via `/proc/self/status`.

432 +- New types `StorageFootprintSample` and `StorageFootprintReport` in `src/crates/netflow-plugin/src/ingest_resource_bench_support.rs`.

433 +- New env vars: `NETFLOW_STORAGE_BENCH_DURATION_SECS` (default 900), `NETFLOW_STORAGE_BENCH_SAMPLE_INTERVAL_SECS` (default 30). Reuses the existing `..._FLOWS_PER_SEC`, `..._PROFILE`, `..._PROTOCOL` env vars.

434 +- Driver: `.local/audits/netflow-bench/run-storage.sh` runs one cell per cardinality and renders markdown.

435 +- Renderer: `.local/audits/netflow-bench/render-storage.py` produces per-cell growth tables and a dedup-ratio summary.

436 +

437 +Results on the same workstation, ipfix at 10 000 flows/s, 15 min per cardinality:

438 +

439 +- Low cardinality: 9.00M flows ingested, 6.46 GiB on-disk total, 771 bytes/stored-flow, write amplification 1.79×.

440 +- High cardinality: 8.97M flows ingested, 7.29 GiB on-disk total, 872 bytes/stored-flow, write amplification 2.00×.

441 +- Dedup ratio (high / low): only 1.13× — high cardinality stores 13% more per flow despite 16× more unique field combinations. Real-world traffic with repeated patterns will trend closer to the low-cardinality figure.

442 +- Raw tier dominates at this timescale (≥99% of total). Rollups (1m: 8-112 MiB, 5m: 8-40 MiB, 1h: 0-16 MiB) are tiny because each rollup row aggregates many raw flows. 1-hour tier did not roll over for low-cardinality within 15 min; it appeared in the high-cardinality run at t=660s onward.

443 +

444 +Sizing/capacity doc updated to remove the bogus "flow × bytes × time" table and replace it with the empirical numbers above plus retention-bounded storage planning guidance.

445 +

446 +Outputs (gitignored):

447 +- `.local/audits/netflow-bench/storage-low.json` — full sample stream for low-cardinality cell

448 +- `.local/audits/netflow-bench/storage-high.json` — full sample stream for high-cardinality cell

449 +- `.local/audits/netflow-bench/storage.md` — rendered markdown

450 +

451 +### 2026-05-07 (later) - documentation rewrite, branch, and PR opened

452 +

453 +- Phase 1B documentation rewrite landed on branch `netflow-plugin-docs-and-bench` over three commits:

454 + - `c708101e` -- per-protocol benchmark + storage footprint test

455 + - `1455d59f` -- docs: rewrite Network Flows documentation (~25 pages)

456 + - `e61c72b7` -- integrations: add netflow-plugin (netflow, ipfix, sflow modules)

457 +- Architectural pivot per Decision 10: `6d72b5ab` introduced `flows` as top-level integration_type with 14 cards.

458 +- Draft PR netdata/netdata#22439 opened against master.

459 +

460 +### 2026-05-07 (later) - autocomplete bug fix

461 +

462 +User reported autocomplete dropdown was useless for AS_NAME searches: typing "Akamai" returned no results because every value is rendered as `AS{n} {Organisation}` and the backend was prefix-matching only.

463 +

464 +Investigation traced the bug to `facet_runtime/store.rs::TextValueStore::prefix_matches` and `facet_runtime/sidecar.rs::search_sidecar`, both using `starts_with`. Fix landed on the same branch with three rounds of codex review:

465 +

466 +- Round 1: introduced substring matching for text-typed facets, kept prefix for IP/numeric. Codex flagged: async runtime blocking, broader-than-AS_NAME effect, naive substring + no length cap, stale docs.

467 +- Round 2: per-field policy via `AutocompleteMatchKind { Prefix, Substring }` on `FacetFieldSpec` (so future per-field overrides are possible without per-kind churn), `memchr::memmem::Finder` for substring search, 256-byte term cap, autocomplete moved to `spawn_blocking`, stale docs updated. Codex flagged blocker: term cap applied to all modes, not just autocomplete.

468 +- Round 3: term cap scoped to `mode == Autocomplete`. Regression test added for non-autocomplete long term. Codex returned "ready to ship for the reviewed autocomplete scope".

469 +

470 +Architectural rule recorded by user: "This affects autocomplete only, not regular facets matching. Only autocomplete. Because we want key=value or key in values, to use indexes, not scan." Verified in code: substring path is unreachable from selections/filters; only `mode=autocomplete` calls `FacetRuntime::autocomplete`.

471 +

472 +Commit: `b733037a`. 9 new tests, full crate 427 passed.

473 +

474 +Follow-ups (recorded in Followup section): case-sensitive matching today (Akamai vs akamai); autocomplete substring on FST sidecars is bounded by limit early-exit but still streams keys for rare/no-hit terms over very large archived vocabularies.

475 +

476 +### 2026-05-07 (later) - documentation enhancements

477 +

478 +- Screenshots from user (7 GitHub asset URLs) embedded across `summary-sankey.md` (2), `time-series.md` (1), `maps-globe.md` (4). Commit `26f8b978`.

479 +- Master alphabetical field index added at the bottom of `field-reference.md`. 91 rows. Each row carries: type, per-protocol availability (✓/◐/—), source class (decoder / enrichment / both), tier preservation (raw / all), selectivity (facet, group-by, filter, metric, time, hidden), and the enrichment chain or IE mapping. Subagent built the data set from code (rollup field defs, RAW_ONLY_FIELDS, facet catalog, decoder IE maps).

480 +- `docs/network-flows/visualization/filters-facets.md` and `docs/network-flows/retention-querying.md` updated to describe the per-field autocomplete policy and the autocomplete-vs-selection distinction.

481 +

482 +### 2026-05-07 (later) - schema fix and learn PR

483 +

484 +- CI surface: `check-documentation` job rejected the netdata PR because `docs/.map/map.schema.json` enum did not yet include `flows` as an `integration_kind`. Schema extended; commit `2c3ab0fc`. CI then turned green for that job.

485 +- Learn PR (netdata/learn#2854) opened to teach `ingest/ingest.py` how to (a) categorise files at `src/crates/netflow-plugin/integrations/<slug>.md` into a new `flows_entries` DataFrame, (b) splice them over the `flows_integrations` placeholder. Mirrors the existing logs handler. Verified locally: 65 Network Flows rows are spliced into `ingest/generated_map.yaml` under the four sub-categories; no `flows_integrations` placeholder remains; ingest exits 0. PR merged by user.

486 +

487 +### 2026-05-07 (later) - SOW close

488 +

489 +- Re-run check on netdata PR #22439 confirmed `check-documentation` passes (after learn merge plus our schema fix).

490 +- Remaining red CI checks (Codacy, SonarCloud, Build Windows) verified unrelated:

491 + - Codacy: 1107 markdownlint findings, 100% style-only (MD013 line-length, MD033 inline HTML, MD045 alt text). Pre-existing baseline; recently merged PR #22432 also "fails" Codacy. Project does not gate on this.

492 + - SonarCloud: same pattern, ignored as gate.

493 + - Build Windows: `urllib HTTPError 403: rate limit exceeded` during packaging step. GitHub API rate limit, transient/infrastructural; unrelated to this work.

494 +- User decision 2026-05-07: Phase 1C (AI skills cross-links to learn docs) is not mandatory. Acceptance criterion for Phase 1C is rejected with this reasoning recorded in Validation. SOW moved to `done/`.

495 +

496 +## Validation

497 +

498 +### Acceptance criteria evidence

499 +

500 +- **metadata.yaml validated**: 14 modules under `plugin_name: netflow-plugin` (3 sources + 4 IP intelligence + 2 BGP routing + 5 network identity). Pipeline runs `gen_integrations.py` + `gen_docs_integrations.py` to exit 0; 14 generated `.md` files under `src/crates/netflow-plugin/integrations/`.

501 +- **Integrations pipeline**: `flows` rendered correctly under its own type. `integrations.json`, `integrations.js`, `COLLECTORS.md` updated. `integrations.json` carries 14 entries with `integration_type: flows` distributed across 4 sub-categories.

502 +- **Learn section**: `docs/.map/map.yaml` carries the Network Flows top-level section with `integration_placeholder integration_kind: flows`. Schema (`docs/.map/map.schema.json`) accepts the value. Learn `ingest/ingest.py` (PR #2854, merged) routes the markdown files into `flows_entries` and splices them over the placeholder. Local ingest run produced 65 Network Flows rows with no remaining `flows_integrations` placeholder.

503 +- **Style guide**: pages are second person, active voice, sentence case. No `:::type` admonitions used; markdown-only by user direction.

504 +- **Field reference**: 91 fields documented by category plus a master alphabetical index with type, per-protocol availability, source class, tier preservation, selectivity, and enrichment chain per row.

505 +- **Enrichment docs**: GeoIP / static metadata / sampling / static networks / classifiers / ASN resolution / BMP routing / BioRIS / Network Sources / decapsulation. BMP, BioRIS, Network Sources documented based on unit-tested parsing logic; their runtime I/O paths still lack integration tests (followup).

506 +- **No mention of untested features**: verified via grep -- no references to topology drilldown, pcap (only as a debugging tool name), eBPF, or threat analytics.

507 +- **Sizing / Capacity planning**: `docs/network-flows/sizing-capacity.md` sourced from `.local/audits/netflow-bench/results.jsonl` (Phase 1.0) and `.local/audits/netflow-bench/storage-{low,high}.json` (Phase 1.0b). Includes the 20-25k flows/s headline, multi-thread CPU semantics note, write-amplification numbers, dedup ratio.

508 +- **Visualisation docs**: 5 pages -- `summary-sankey.md`, `time-series.md`, `maps-globe.md`, `filters-facets.md`, `dashboard-cards.md`. Maps and globe consolidated into one page since they share the data path.

509 +- **Screenshots**: 7 user-provided GitHub asset URLs embedded across the visualisation pages.

510 +

511 +### Reviewer findings (codex)

512 +

513 +Three rounds of read-only review over the autocomplete fix. Round 1 surfaced async runtime blocking, broader-than-AS_NAME effect, naive substring, stale docs. Round 2 surfaced a blocker (term cap was global, should be autocomplete-only). Round 3 returned "ready to ship for the reviewed autocomplete scope" with no new blockers.

514 +

515 +### Same-failure search

516 +

517 +- Other text facets affected by the same prefix-only autocomplete bug: `EXPORTER_NAME`, `IN_IF_DESCRIPTION`, `*_NET_NAME`, `SRC_MAC`, `DST_MAC`, `DST_AS_PATH`, `DST_COMMUNITIES`, country/state/city. Per-field policy fixes them all in the same pass.

518 +- Other places stale "prefix" or "in-memory" claims about autocomplete: `filters-facets.md:38`, `retention-querying.md:104` -- both updated.

519 +- Other places where the project might gate substring on autocomplete vs selection: confirmed selections use exact equality (`FacetStore::contains_value_ref`); substring path is unreachable from filtering.

520 +

521 +### Artifact maintenance gate

522 +

523 +- **AGENTS.md**: no update needed.

524 +- **Runtime project skills**: `.agents/skills/integrations-lifecycle/` updated -- new `per-type-matrix.md` Learn-routing matrix and `how-tos/adding-new-integration-type.md` 8-place checklist. `INDEX.md` cross-linked.

525 +- **Specs**: no spec update needed; project is incrementally bootstrapped and netflow-plugin specs were not pre-existing.

526 +- **End-user / operator docs**: this IS the docs update -- ~25 pages under `docs/network-flows/` plus the netflow-plugin README benchmark refresh.

527 +- **End-user / operator skills (Phase 1C)**: REJECTED by user 2026-05-07 with reasoning "skills linking to docs is not mandatory". `query-netdata-cloud/query-flows.md` and `query-netdata-agents/query-flows.md` remain at their pre-SOW state. Cross-linking to learn docs can be added in a future skills-maintenance pass without blocking this SOW. NOT tracked as a follow-up SOW because it is not a deferred feature -- it is an explicit scope rejection.

528 +- **SOW lifecycle**: status moved to `completed`, file moved to `done/`, in the same commit as the autocomplete fix lands on the active PR branch.

529 +

530 +### Status / directory consistency

531 +

532 +Status: `completed`. Directory: `done/`. Filename unchanged.

533 +

534 +### Lessons captured

535 +

536 +See `## Lessons Extracted` below.

537 +

538 +## Outcome

539 +

540 +Delivered:

541 +

542 +- 4 commits on netdata branch `netflow-plugin-docs-and-bench` (PR #22439, draft):

543 + - `c708101e` per-protocol benchmark + storage footprint test

544 + - `1455d59f` Network Flows documentation rewrite (~25 pages)

545 + - `e61c72b7` netflow-plugin metadata.yaml (3 modules) and integration cards

546 + - `6d72b5ab` `flows` top-level integration_type with 14 cards

547 + - `b733037a` substring autocomplete on text facets (3 codex rounds, 9 new tests, 427 pass)

548 + - `2c3ab0fc` `docs/.map` schema accepts `flows`

549 + - `26f8b978` screenshots and master field index

550 +- 1 commit on learn branch `netflow-flows-integrations` (PR #2854, MERGED 2026-05-07):

551 + - `de62daaf` ingest: route flows integrations into the Network Flows section

552 +- Companion website branch `netflow-flows-content` carries content corrections (separate, smaller).

553 +

554 +Pending only:

555 +

556 +- Re-trigger Build Windows on netdata PR #22439 (transient `urllib` HTTP 403 rate limit, not our code).

557 +- Mark netdata PR #22439 ready for review when user decides.

558 +- Cloud-frontend "Update integrations.js" PR -- standard manual sync, not gated by this SOW.

559 +

560 +## Lessons Extracted

561 +

562 +- **Code is ground truth, not the existing markdown**: the inherited netflow docs were thin and contained inaccuracies. Re-verifying every claim against `src/crates/netflow-plugin/` paid off -- multiple "well-documented" behaviours (e.g. AS name format, sFlow VLAN provenance, "single-core" benchmark framing) turned out to be wrong or stale.

563 +- **Doubling/mirroring is foundational, not a footnote**: users cannot reason about ANY aggregate number unless they understand that one router watching ingress + egress doubles every flow. This had to be the first concept on the Overview, not an "advanced" sidebar.

564 +- **Subagent per feature, master assistant for synthesis**: spawning a read-only subagent per enrichment module / visualisation / source kept the master context clean. The Overview was written last as a natural index of established truths -- not first as a promise the detail pages later contradicted.

565 +- **Per-field beats per-kind for policy that touches UX**: the autocomplete fix initially used per-kind dispatch (Text vs others). Codex pushed for per-field. The right answer was a `FacetFieldSpec::autocomplete_match` field that defaults from kind but allows future per-field overrides without churn.

566 +- **Reviewer iterations are non-optional**: codex flagged a real blocker on round 2 (term cap applied to all modes, not just autocomplete). One round of review would have shipped that bug. The project rule "iterate until reviewers cannot find anything else" is load-bearing.

567 +- **Match the codebase's own conventions over generic style**: substring autocomplete matches what `libnetdata/facets/facets.c:1783` already does for systemd-journal FTS (`SIMPLE_PATTERN_SUBSTRING`). Consistent with the project, not novel.

568 +- **Separate benchmarks: per-protocol resource envelope vs storage footprint**: a single "resource benchmark" couldn't answer both "what's the ingest cost?" and "what's the on-disk cost?". Splitting them produced two complementary tables and removed a bogus `bytes/flow x time` calculation that ignored journal indexing and dedup.

569 +- **Architectural pivots happen mid-SOW**: the original plan placed flow integrations under `data-collection.networking` as collector-typed cards. Mid-execution the user reframed the test as "would users ask 'Does Netdata integrate with X?'", which shifted the answer to a top-level `flows` integration_type with 14 cards. Captured as Decision 10 rather than retconning earlier decisions.

570 +

571 +## Followup

572 +

573 +Open follow-ups, ordered by priority:

574 +

575 +1. **Cloud-frontend `Update integrations.js`** -- copy the regenerated `integrations.js` from netdata into `cloud-frontend/src/domains/integrations/data/integrations.js` and open the standard sync PR. Not gated by this SOW. Last manual refresh was 2024-10-21; this work won't appear in the dashboard's Integrations modal until that file is updated.

576 +2. **Phase 2 SOW: integration tests for runtime I/O paths**:

577 + - BMP listener: async/tokio tests for TCP accept loop, framed decode, `apply_update` trie wiring, malformed message error accumulation, retry/shutdown. Test with a real BMP speaker, measure ingest impact, validate enrichment correctness.

578 + - BioRIS: async tests for gRPC client connection, RIB dump stream, retry/backoff. Build a local RIS daemon for end-to-end validation. Test `build_endpoint_uri`, `parse_router_ip`.

579 + - Network Sources: async tests for HTTP fetch cycle, service loop, prefix matching integration, failed HTTP handling, header forwarding, multi-source merge/re-publish.

580 + - All three: integration tests that wire parsed data through `DynamicRoutingRuntime` trie into flow enrichment lookup.

581 +3. **BMP architectural decision**: enrichment-only in netflow-plugin (Akvorado pattern) vs a separate BGP monitoring plugin with its own DB. This is product-level, not a code change.

582 +4. **Topology drilldown**: the `useFlowsDrilldownData` hook is dead code today. Implement the actor-modal hook when a UX home is decided.

583 +5. **Autocomplete follow-ups (deferred from autocomplete bug fix)**:

584 + - Case-insensitive matching for text facets (today: typing `akamai` will not match `AS20940 Akamai International`). UX call.

585 + - Substring scan over very large archived FST sidecars is bounded by `FACET_AUTOCOMPLETE_LIMIT` early-exit but still streams keys for rare/no-hit terms. Token-prefix or n-gram indexing can be added if measurements warrant.

586 +6. **Health alerts (`health.d/`)**: deferred. The netflow-plugin emits its own self-monitoring metrics (parse errors, decoder latency, ingest queue depth); alerts on those have not yet been authored.

587 +7. **AI skills cross-links to learn docs**: REJECTED for this SOW (not mandatory per user 2026-05-07). Can be picked up in a future skills-maintenance pass; not tracked as a separate SOW because rejected, not deferred.

588 +

589 +## Regression Log

590 +

591 +None yet.

docs/.map/map.schema.json

@@ -106,6 +106,7 @@

106 "notifications",

107 "authentication",

108 "logs",

109 + "flows",

110 "agent_notifications",

111 "cloud_notifications"

112 ],

docs/.map/map.yaml

+89

@@ -476,6 +476,95 @@ sidebar:

476 - meta:

477 label: Shell Scripts

478 edit_url: https://github.com/netdata/netdata/edit/master/src/web/api/exporters/shell/README.md

479 + # Network Flows

480 + - meta:

481 + label: Network Flows

482 + items:

483 + - meta:

484 + label: Overview

485 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/README.md

486 + description: Collect, enrich, and visualize NetFlow, IPFIX, and sFlow data with the Netdata Agent.

487 + - meta:

488 + label: Installation

489 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/installation.md

490 + - meta:

491 + label: Quick Start

492 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/quick-start.md

493 + description: Get network flow monitoring running in five minutes.

494 + - type: integration_placeholder

495 + integration_kind: flows

496 + - meta:

497 + label: Configuration

498 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/configuration.md

499 + description: Full reference for netflow.yaml configuration options.

500 + - meta:

501 + label: Enrichment Concepts

502 + items:

503 + - meta:

504 + label: IP Intelligence

505 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/ip-intelligence.md

506 + description: How GeoIP and ASN data combine to enrich flow records with country, city, and AS-name labels.

507 + - meta:

508 + label: BGP Routing

509 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/bgp-routing.md

510 + description: How live BGP routes (BMP, BioRIS) feed AS path, communities, and next-hop into flow records.

511 + - meta:

512 + label: Network Identity

513 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/network-identity.md

514 + description: How external feeds (cloud IP ranges, IPAM systems) label your network prefixes.

515 + - meta:

516 + label: Static Metadata

517 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/static-metadata.md

518 + - meta:

519 + label: Classifiers

520 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/classifiers.md

521 + - meta:

522 + label: ASN Resolution

523 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/asn-resolution.md

524 + - meta:

525 + label: Decapsulation

526 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/decapsulation.md

527 + - meta:

528 + label: Field Reference

529 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/field-reference.md

530 + description: Complete list of flow fields with per-protocol availability.

531 + - meta:

532 + label: Visualization

533 + items:

534 + - meta:

535 + label: Sankey and Table

536 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/summary-sankey.md

537 + - meta:

538 + label: Time-Series

539 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/time-series.md

540 + - meta:

541 + label: Maps and Globe

542 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/maps-globe.md

543 + - meta:

544 + label: Filters and Facets

545 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/filters-facets.md

546 + - meta:

547 + label: Plugin Health Charts

548 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/dashboard-cards.md

549 + - meta:

550 + label: Retention and Querying

551 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/retention-querying.md

552 + - meta:

553 + label: Sizing and Capacity Planning

554 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/sizing-capacity.md

555 + description: Storage estimation, memory guidance, and performance benchmarks.

556 + - meta:

557 + label: Validation and Data Quality

558 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/validation.md

559 + - meta:

560 + label: Investigation Playbooks

561 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/investigation-playbooks.md

562 + - meta:

563 + label: Anti-patterns

564 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/anti-patterns.md

565 + - meta:

566 + label: Troubleshooting

567 + edit_url: https://github.com/netdata/netdata/edit/master/docs/network-flows/troubleshooting.md

568 # Logs

569 - meta:

570 label: Logs

docs/network-flows/README.md new

+163

@@ -0,0 +1,163 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/README.md"

3 +sidebar_label: "Overview"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['netflow', 'sflow', 'ipfix', 'network flows', 'traffic analysis', 'overview']

7 +endmeta-->

8 +

9 +# Network Flows

10 +

11 +Netdata can collect, store, and visualise network flow data from your routers and switches. You see who is talking to whom on your network, how much data they exchanged, over what protocols, and to which countries — without inspecting packet contents and without an external database.

12 +

13 +This section is for network engineers, security analysts, and IT managers who want to understand what's happening on the wire. The same dashboard answers all three audiences from the same data.

14 +

15 +## What flow data is

16 +

17 +A network flow is a *summary of a conversation*. Routers and switches watch packets as they pass through, group them by source IP, destination IP, source port, destination port, and protocol, and produce one record per flow when the conversation ends or after a timeout. That record contains:

18 +

19 +- The endpoints (IPs, ports, protocol)

20 +- How much data moved (bytes, packets)

21 +- When the flow started and ended

22 +- Optional metadata: AS numbers, interface indexes, TCP flags, ToS, MAC addresses, VLAN IDs

23 +

24 +Think of flow data like an itemised phone bill. You can see who called whom, when, and for how long. You **cannot** read the conversation. That trade-off is the entire value proposition: low storage, no privacy intrusion, complete coverage of all traffic — but no payload visibility.

25 +

26 +## What you can answer

27 +

28 +- Who is using the most bandwidth right now? Last week? Last month?

29 +- Where does our traffic go — by country, ASN, port, or protocol?

30 +- A specific IP appeared in a security alert — what did it talk to, when, and how much?

31 +- Is this a normal pattern for this time of day, or has something changed?

32 +- Should we upgrade this Internet link, and when?

33 +- Which interfaces of which routers are saturated?

34 +

35 +## What you cannot answer

36 +

37 +- **Is the application slow?** Flow data has no payload, no response times, no error messages. Use APM or application logs.

38 +- **What's the latency?** Flow records show duration, not round-trip time. Duration is dominated by timeout configuration, not network performance. Use ICMP probes or hardware telemetry.

39 +- **What did the user actually do?** Flow data sees ports and IPs, not user actions or URLs.

40 +- **Did this packet arrive late?** Flow data is aggregated; sub-second jitter and microbursts are invisible.

41 +

42 +If those are your questions, flow data is the wrong tool. You probably need application performance monitoring (APM), logging, or packet capture.

43 +

44 +## Two things to know on day one

45 +

46 +These two facts are not Netdata-specific. They're how flow data works on every collector. Understanding them up-front saves a lot of head-scratching when you first see the dashboard.

47 +

48 +### Traffic appears doubled by default

49 +

50 +A router exports flow records for both ingress and egress on every monitored interface. A single packet entering interface A and leaving interface B produces two records: one tagged ingress on A, one tagged egress on B.

51 +

52 +If you sum every flow record without filtering, you see roughly **2× the actual traffic**. With a second router on the same path, **4×**.

53 +

54 +To see real numbers: filter by one exporter, one interface, in one direction. The dashboard makes this easy. See the [Anti-patterns page](/docs/network-flows/anti-patterns.md) for the full framing.

55 +

56 +### Conversations are mirrored

57 +

58 +A bidirectional conversation (host A talks to host B, B replies to A) produces at least two flow records — one for each direction. They're real, distinct flows. But on a Sankey diagram, country map, or sorted top-N table without direction filtering, you see both ends of every conversation. That's correct, but it can look like the same traffic appears twice.

59 +

60 +When you see "traffic from your country to a foreign country" *and* "traffic from that foreign country to your country" of similar volume, you're looking at one conversation, not two.

61 +

62 +## What ships with the plugin

63 +

64 +The Netdata netflow plugin decodes:

65 +

66 +- **NetFlow v5** (legacy, IPv4-only)

67 +- **NetFlow v7** (rare, Cisco Catalyst 5000)

68 +- **NetFlow v9** (the modern Cisco / Juniper / FortiGate / Arista format)

69 +- **IPFIX** (RFC 7011, the IETF-standardised successor to NetFlow v9)

70 +- **sFlow v5** (the packet-sampling protocol most switches use)

71 +

72 +A single UDP listener (default `0.0.0.0:2055`) accepts all five. The plugin auto-detects each datagram's protocol from its header.

73 +

74 +Each flow record is enriched at ingestion with:

75 +

76 +- **Country, state, city, coordinates, ASN, AS name** — from a stock GeoIP database (DB-IP-based; refreshable)

77 +- **Exporter name and labels** — from your static-metadata configuration

78 +- **Interface name, description, speed, provider, connectivity, boundary** — from your static-metadata configuration

79 +- **Network labels** for your own CIDRs (name, role, site, region, tenant)

80 +- **Classifier-derived attributes** for rule-based tagging (Akvorado-compatible expression language)

81 +- **Live BGP attributes** (AS path, communities, next-hop) — from BMP, BioRIS, or static prefix configuration

82 +- **Decapsulated inner-packet fields** for SRv6 / VXLAN traffic

83 +

84 +Flow records land in a four-tier journal: raw + 1-minute + 5-minute + 1-hour rollups, with independent retention per tier. The dashboard auto-picks the best tier for each query.

85 +

86 +## What sampling does to your numbers

87 +

88 +Many routers sample. They export one packet in N — typically 1-in-100 to 1-in-2000. Netdata multiplies bytes and packets by the sampling rate at ingestion, so the numbers you see are estimates of actual traffic.

89 +

90 +This works correctly **only if all your exporters use the same sampling rate**. With mixed rates, the multiplication is per-flow and the aggregate becomes a blend of estimates that's hard to interpret. The clean path: keep sampling rates uniform across your network, or run unsampled where the flow rate allows.

91 +

92 +Sampling at 1-in-1000 also misses small flows. A single-packet flow has a 99.9% chance of not being seen at all. If you need to detect small, rare events (security beaconing, scanning), use unsampled or 1-in-100 on critical exporters.

93 +

94 +## What the dashboard looks like

95 +

96 +Six visualisations, all driven by the same query engine:

97 +

98 +- **Sankey + Table** — the default. Top-N flows aggregated by 1-10 fields you pick. Best for "who's responsible".

99 +- **Time-Series** — the same top-N over time. Best for "how does this change".

100 +- **Country map / state map / city map** — geographic views. Best for "where".

101 +- **Globe** — a 3D rendering of the city-level data. Visual demo, less useful for analysis.

102 +

103 +A filter ribbon between the visualisation and the table lets you narrow data by any combination of fields. Selections persist in the URL — copy and share to give a colleague exactly your view.

104 +

105 +Default settings on first open: last 15 minutes, top-25 flows by bytes, grouped as `Source ASN → Protocol → Destination ASN`.

106 +

107 +Default fields are tuned to surface meaningful traffic at a glance. From there, you adjust the time range, change the aggregation, add filters, and dig in.

108 +

109 +## Where to start

110 +

111 +Pick the page that matches your situation:

112 +

113 +- **You're setting up the plugin for the first time** — [Installation](/docs/network-flows/installation.md), then [Quick Start](/docs/network-flows/quick-start.md).

114 +- **You have data, you want to find a bandwidth hog or trace an IP** — [Investigation Playbooks](/docs/network-flows/investigation-playbooks.md).

115 +- **You want to make sure your data is trustworthy** — [Validation and Data Quality](/docs/network-flows/validation.md).

116 +- **You want to avoid the most common mistakes** — [Anti-patterns](/docs/network-flows/anti-patterns.md).

117 +- **You want to understand a specific feature in depth** — see the section index below.

118 +

119 +## Section index

120 +

121 +**Setup and configuration**

122 +

123 +- [Installation](/docs/network-flows/installation.md) — Package names, install commands, file locations

124 +- [Quick Start](/docs/network-flows/quick-start.md) — Configure your first router, see traffic in 15 minutes

125 +- [Configuration](/docs/network-flows/configuration.md) — `netflow.yaml` reference

126 +

127 +**Sources**

128 +

129 +- [NetFlow](/src/crates/netflow-plugin/integrations/netflow.md) — v5, v7, v9

130 +- [IPFIX](/src/crates/netflow-plugin/integrations/ipfix.md) — IETF-standardised, biflow-capable

131 +- [sFlow](/src/crates/netflow-plugin/integrations/sflow.md) — packet-sampling, fundamentally different

132 +

133 +**Enrichment**

134 +

135 +- [GeoIP](/docs/network-flows/enrichment/ip-intelligence.md) — Country, city, AS-name lookups

136 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md) — Naming exporters, interfaces, your networks

137 +- [Classifiers](/docs/network-flows/enrichment/classifiers.md) — Rule-based tagging

138 +- [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md) — Where AS numbers and names come from

139 +- [BMP routing](/docs/network-flows/enrichment/bgp-routing.md) — Live BGP feed for routing attributes

140 +- [BioRIS](/docs/network-flows/enrichment/bgp-routing.md) — RIPE RIS via gRPC

141 +- [Network sources](/docs/network-flows/enrichment/network-identity.md) — HTTP-fetched prefix metadata

142 +- [Decapsulation](/docs/network-flows/enrichment/decapsulation.md) — SRv6 and VXLAN inner-packet extraction

143 +

144 +**Reference**

145 +

146 +- [Field reference](/docs/network-flows/field-reference.md) — All 91 fields and which protocols populate each

147 +- [Retention and querying](/docs/network-flows/retention-querying.md) — The four-tier model and how queries pick a tier

148 +- [Sizing and capacity planning](/docs/network-flows/sizing-capacity.md) — Hardware, throughput, storage estimates

149 +

150 +**Visualisation**

151 +

152 +- [Sankey and Table](/docs/network-flows/visualization/summary-sankey.md) — The default view

153 +- [Time-Series](/docs/network-flows/visualization/time-series.md) — Top-N over time

154 +- [Maps and Globe](/docs/network-flows/visualization/maps-globe.md) — Geographic views

155 +- [Filters and Facets](/docs/network-flows/visualization/filters-facets.md) — Narrowing the data

156 +- [Plugin Health Charts](/docs/network-flows/visualization/dashboard-cards.md) — Operational metrics for the plugin itself

157 +

158 +**Operations**

159 +

160 +- [Validation and Data Quality](/docs/network-flows/validation.md) — Cross-checks and silent failures

161 +- [Investigation Playbooks](/docs/network-flows/investigation-playbooks.md) — Recipes for common questions

162 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — Common mistakes and how to avoid them

163 +- [Troubleshooting](/docs/network-flows/troubleshooting.md) — When something doesn't work

docs/network-flows/anti-patterns.md new

+157

@@ -0,0 +1,157 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/anti-patterns.md"

3 +sidebar_label: "Anti-patterns"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['anti-patterns', 'mistakes', 'pitfalls', 'gotchas', 'misuse']

7 +endmeta-->

8 +

9 +# Anti-patterns and pitfalls

10 +

11 +Flow data is powerful but easy to misuse. The mistakes below are the ones that cause the most lost analyst time and the most wrong conclusions in real deployments. Each entry explains how the mistake happens, what it costs, and how to avoid it.

12 +

13 +## 1. Reading aggregate volume without filtering

14 +

15 +**The mistake.** You open the Network Flows tab, see a total bandwidth number, and assume it represents your real traffic.

16 +

17 +**Why it's wrong.** Routers normally export both ingress and egress flow records on every monitored interface. A single packet entering interface A and leaving interface B produces two records — one tagged ingress on A, one tagged egress on B. With one router and the standard configuration, summing all flow records gives you roughly **2× the actual traffic**. Add a second router on the same path and you see 4×.

18 +

19 +**What it costs.** You think your link carries 2 Gbps when it really carries 1 Gbps. Capacity decisions based on these numbers are wrong by a factor of 2 or more.

20 +

21 +**How to avoid it.** Always filter by one exporter and one direction (Input Interface OR Output Interface, not both) when reading absolute volume numbers. To validate: compare to SNMP interface counters on the same interface — values should be close.

22 +

23 +## 2. Ignoring the sampling rate

24 +

25 +**The mistake.** Your router is configured to sample 1-in-1000 packets. Nobody documented this. The dashboard shows 5 Mbps. You assume that's your traffic.

26 +

27 +**Why it's wrong.** With sampling, a flow record represents one observed packet out of every N. Netdata multiplies bytes and packets by the sampling rate at ingestion, so the dashboard numbers are estimates of actual traffic — *if* the multiplication is consistent. When sampling rates differ across exporters in the same query, the aggregate becomes a blend of estimates that is hard to interpret correctly.

28 +

29 +**What it costs.** Volume analysis is off by orders of magnitude when the rate isn't documented. Small flows are statistically invisible — at 1-in-1000, a single-packet flow has a 99.9% chance of being missed entirely. Security investigations miss low-volume threats like beaconing and probing.

30 +

31 +**How to avoid it.**

32 +

33 +- Use a uniform sampling rate across your network, or run unsampled where flow rates allow.

34 +- For Internet-edge security work, use 1-in-100 or unsampled. Sampling at 1-in-1000 hides small flows.

35 +- Document sampling rates per exporter and audit them quarterly.

36 +- Cross-check flow-derived bandwidth against SNMP. If they diverge by more than 30%, investigate before trusting the data.

37 +

38 +## 3. Trusting GeoIP for internal IPs

39 +

40 +**The mistake.** You enable GeoIP enrichment. Internal IPs (10.x, 172.16-31.x, 192.168.x) appear in random countries on the geographic map.

41 +

42 +**Why it's wrong.** GeoIP databases don't have entries for private IP ranges. Netdata doesn't skip private IPs — it just hands the IP to the database and uses what comes back. With the stock DB-IP database, private ranges are tagged so they render as "AS0 Private IP Address Space" with empty country. With third-party databases, results vary. Some return spurious country data for RFC 1918 addresses.

43 +

44 +**What it costs.** Geographic anomalies look like security incidents. Analysts waste time investigating "traffic from China" that's actually traffic to a server in the 10.x.x.x range.

45 +

46 +**How to avoid it.** Configure your internal IP ranges as static metadata before relying on geographic analysis. Use the [`networks`](/docs/network-flows/enrichment/static-metadata.md) block to declare each internal CIDR with a name, role, and (optionally) overridden country. The labels you set there override whatever GeoIP returns. Validate by spot-checking known IPs against the map.

47 +

48 +## 4. Alerting on absolute volume thresholds

49 +

50 +**The mistake.** You configure an alert: "page me if any IP sends more than 10 GB in an hour."

51 +

52 +**Why it's wrong.** That threshold is a guess. Your backup server legitimately sends 500 GB/hour. An attacker exfiltrating 200 MB/hour is invisible.

53 +

54 +**What it costs.** The alert is either constant noise (false positives) or completely silent (false negatives). Either way, alerts get ignored.

55 +

56 +**How to avoid it.** Establish baselines first. Compare current traffic to the same time period in previous weeks (Tuesday 10 AM vs the average of the last four Tuesdays at 10 AM). Alert on deviation from the baseline, not on absolute values.

57 +

58 +(Netdata's alerting on flow data is in development; for now this pattern lives in your monitoring practice, not in the plugin.)

59 +

60 +## 5. Collecting flows but never looking at them

61 +

62 +**The mistake.** Flow export is enabled on every router. Storage fills up. Nobody opens the dashboard between incidents.

63 +

64 +**Why it's wrong.** Flow data is only useful when someone actively interprets it. Without baselines, watchlists, and routine review, you have data without insight.

65 +

66 +**What it costs.** When an incident happens, you don't know what "normal" looks like, so you can't recognise abnormal. Storage and CPU are spent without operational value.

67 +

68 +**How to avoid it.** Schedule a weekly 15-minute review. Document what "normal" looks like — top 10 talkers, traffic curve shape, protocol distribution, geographic distribution. Add anything new that appears in the top-10 to a watchlist for investigation. Use [Investigation Playbooks](/docs/network-flows/investigation-playbooks.md) for the recurring questions.

69 +

70 +## 6. Confusing flows with sessions

71 +

72 +**The mistake.** You see 50 000 flow records in an hour and report it as "we had 50 000 user sessions".

73 +

74 +**Why it's wrong.** A flow record is a network-level artifact, not an application session. A single page load generates dozens of flows: DNS lookups, the TCP handshake, the TLS handshake, HTTP requests for embedded resources, telemetry pings. A long file transfer may be one flow or many, depending on timeout configuration.

75 +

76 +**What it costs.** Wildly inflated user activity numbers. Misinterpretation of usage patterns.

77 +

78 +**How to avoid it.** Aggregate by source IP and time window for a session-like view. Use ports and protocols to classify, not to count transactions. If you need real session data, use application logs or APM, not flow records.

79 +

80 +## 7. NAT blindness

81 +

82 +**The mistake.** You place the collector outside a NAT gateway because mirroring traffic there is easier.

83 +

84 +**Why it's wrong.** Every internal host appears as the same public IP after NAT. You can't identify the actual source of the traffic.

85 +

86 +**What it costs.** Your top talker is "the firewall". Security can't find the infected host, capacity can't identify the bandwidth hog.

87 +

88 +**How to avoid it.** Collect inside each NAT boundary, or correlate flow data with NAT translation logs (`iptables NFLOG`, vendor NAT logging) to map external 5-tuples back to internal hosts.

89 +

90 +## 8. Geographic firewall of shame

91 +

92 +**The mistake.** You configure an alert: "page security if traffic goes to any country except the home country."

93 +

94 +**Why it's wrong.** CDNs, cloud providers, and SaaS endpoints serve from edge nodes worldwide. Traffic to the same SaaS provider may resolve to Singapore one day and Frankfurt the next. None of this is suspicious.

95 +

96 +**What it costs.** Constant false positives. Trust in the alerting system collapses. Real anomalies get ignored among the noise.

97 +

98 +**How to avoid it.** Whitelist known cloud and CDN ASNs. Use ASN as the primary signal and country as secondary corroboration. If you must alert on country, alert only on countries you have no business relationship with — and review the whitelist quarterly.

99 +

100 +## 9. Treating flow duration as latency

101 +

102 +**The mistake.** You divide flow bytes by flow duration and present that as "speed", or use duration as a proxy for round-trip time.

103 +

104 +**Why it's wrong.** Flow duration is dominated by the active timeout setting and application think time. A flow with a 60-second active timeout is exported every 60 seconds whether the network is fast or slow. There's no relationship between flow duration and latency.

105 +

106 +**What it costs.** False conclusions about network performance. Misdirected troubleshooting.

107 +

108 +**How to avoid it.** Use SNMP for interface utilisation, ICMP probes for round-trip time, APM tools for application performance. Flow data answers "how much" and "between whom", never "how fast".

109 +

110 +## 10. Trying to detect microbursts

111 +

112 +**The mistake.** Users complain about momentary slowness. You look in flow data for the burst.

113 +

114 +**Why it's wrong.** NetFlow active timeout aggregates traffic into windows of 60 seconds or more. sFlow random sampling misses bursts that occur between sampled packets. Neither protocol can resolve sub-second events. The Netdata time-series view also clamps to 60-second buckets.

115 +

116 +**What it costs.** You spend time looking for something flow data physically cannot show.

117 +

118 +**How to avoid it.** For microburst detection use packet capture, switch microburst counters, or hardware-assisted telemetry. Flow data is for sustained patterns, not millisecond events.

119 +

120 +## 11. Reasoning from raw byte counts when sampling is on

121 +

122 +**The mistake.** You see `RAW_BYTES = 5000` for a flow and assume 5000 bytes was the actual traffic.

123 +

124 +**Why it's wrong.** `RAW_BYTES` is the unscaled byte count from the exporter. With sampling at 1-in-1000, the actual traffic was approximately 5 000 000 bytes. The scaled value is in `BYTES`.

125 +

126 +**How to avoid it.** Use `BYTES` (auto-scaled) for normal analysis. Use `RAW_BYTES` only when sampling is uniform across all exporters and you specifically need exact pre-scaling counts.

127 +

128 +## 12. Comparing flow counts across protocols

129 +

130 +**The mistake.** You report "Arista switches see far more flows than Cisco routers" based on flow counts.

131 +

132 +**Why it's wrong.** NetFlow aggregates millions of packets into one flow record. sFlow exports individual packet samples — each becomes its own "flow" record. Their counts are not comparable. Same goes for sampling-rate differences across exporters.

133 +

134 +**How to avoid it.** Aggregate by IP/port/time window before comparing. Compare bytes (after scaling), not flow counts. Document which protocol each exporter speaks.

135 +

136 +## Summary

137 +

138 +| Mistake | One-line fix |

139 +|---|---|

140 +| Doubled aggregate | Filter by exporter + interface + direction |

141 +| Ignored sampling | Document and uniform-rate; cross-check SNMP |

142 +| GeoIP for internal IPs | Configure internal CIDRs in `enrichment.networks` |

143 +| Absolute thresholds | Baseline first, alert on deviation |

144 +| Collect-and-ignore | Weekly 15-minute review with documented baselines |

145 +| Flows ≠ sessions | Aggregate by IP and time window |

146 +| NAT blindness | Collect inside the NAT boundary |

147 +| Geographic firewall of shame | Use ASN, whitelist cloud and CDN providers |

148 +| Duration as latency | Use SNMP/ICMP/APM for latency |

149 +| Microburst hunting | Use packet capture or hardware telemetry |

150 +| Raw bytes when sampling | Use `BYTES`, not `RAW_BYTES`, unless rates are uniform |

151 +| Cross-protocol flow counts | Use bytes (scaled), not flow counts |

152 +

153 +## What's next

154 +

155 +- [Validation and Data Quality](/docs/network-flows/validation.md) — How to confirm your data is trustworthy.

156 +- [Investigation Playbooks](/docs/network-flows/investigation-playbooks.md) — Step-by-step recipes for common questions.

157 +- [Sources](/src/crates/netflow-plugin/integrations/netflow.md) — Per-protocol behaviour that drives many of these gotchas.

docs/network-flows/configuration.md new

+308

@@ -0,0 +1,308 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/configuration.md"

3 +sidebar_label: "Configuration"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['configuration', 'netflow.yaml', 'tuning', 'retention', 'listener']

7 +endmeta-->

8 +

9 +# Configuration

10 +

11 +The netflow plugin reads its configuration from `netflow.yaml`. Defaults are sane out of the box; most operators only adjust three things — the listener address, the journal retention, and (rarely) the per-tier overrides. This page documents every option, with its real default and the file that defines it.

12 +

13 +## Where the file lives

14 +

15 +| Path | Purpose |

16 +|---|---|

17 +| `/etc/netdata/netflow.yaml` | Your configuration. Edits here survive package upgrades. |

18 +| `/usr/lib/netdata/conf.d/netflow.yaml` | The stock file shipped with the package. Reference only. |

19 +

20 +The plugin reads the user file when it exists, and the stock file otherwise. To start customising, copy the stock file:

21 +

22 +```bash

23 +sudo cp /usr/lib/netdata/conf.d/netflow.yaml /etc/netdata/netflow.yaml

24 +```

25 +

26 +## Three things to know before you edit

27 +

28 +1. **Restart required.** There is no live-reload for plugin configuration. After saving the file, run `sudo systemctl restart netdata`. Only the GeoIP databases reload on a timer; everything else needs a restart.

29 +2. **Strict YAML.** Every section refuses unknown keys. A misspelled key fails the plugin at startup with an error in the journal. If you see "the plugin won't start after my edit", check for a typo before anything else.

30 +3. **CLI flags vs YAML.** When the plugin runs as a Netdata Agent plugin (the normal case), only the YAML is read — CLI flags do nothing. The CLI flags shown below apply only if you run the binary directly outside of Netdata.

31 +

32 +## Top-level layout

33 +

34 +```yaml

35 +enabled: true # global on/off

36 +listener: { ... } # UDP socket and journal sync

37 +protocols: { ... } # which protocols to accept; decapsulation; timestamps

38 +journal: { ... } # tier directories, retention, query guardrails

39 +enrichment: { ... } # GeoIP, classifiers, ASN, BMP, BioRIS, network sources

40 +```

41 +

42 +The `listener`, `protocols`, and `journal` sections are flattened — their keys can also appear at the top level (the stock file does this for compatibility). Both forms are accepted.

43 +

44 +## `enabled`

45 +

46 +```yaml

47 +enabled: true

48 +```

49 +

50 +Set to `false` to turn the entire flow plugin off. The plugin still loads but does nothing. Default: `true`.

51 +

52 +## `listener`

53 +

54 +Controls the UDP socket and the journal write cadence.

55 +

56 +```yaml

57 +listener:

58 + listen: "0.0.0.0:2055"

59 + max_packet_size: 9216

60 + sync_every_entries: 1024

61 + sync_interval: "1s"

62 +```

63 +

65 +|---|---|---|---|

70 +

71 +### UDP buffer tuning is not in this file

72 +

73 +If you receive a high flow rate, the kernel UDP receive buffer matters more than `max_packet_size`. Tune at the kernel level:

74 +

75 +```bash

76 +sudo sysctl -w net.core.rmem_max=33554432

77 +sudo sysctl -w net.core.rmem_default=8388608

78 +sudo sysctl -w net.core.netdev_max_backlog=250000

79 +```

80 +

81 +Persist these in `/etc/sysctl.d/99-netflow.conf`. The plugin does not call `setsockopt(SO_RCVBUF)` itself; whatever the kernel default is, that's what the listener gets.

82 +

83 +## `protocols`

84 +

85 +```yaml

86 +protocols:

87 + v5: true

88 + v7: true

89 + v9: true

90 + ipfix: true

91 + sflow: true

92 + decapsulation_mode: none

93 + timestamp_source: input

94 +```

95 +

97 +|---|---|---|---|

105 +

106 +You must keep at least one protocol enabled or the plugin refuses to start.

107 +

108 +### `timestamp_source` values

109 +

110 +- **`input`** (default) — the time the plugin received the datagram. Charts always look "now". This is the safest choice for dashboards.

111 +- **`netflow_packet`** — the time the exporter put in the NetFlow/IPFIX header.

112 +- **`netflow_first_switched`** — the time the flow actually started, from the per-record first-switched field. Records arrive with timestamps in the past (up to your active timeout). This gives the most accurate timeline but charts may show data appearing "behind" real time.

113 +

114 +## `journal`

115 +

116 +This is the section most operators tune. It controls where flow data lives, how much of it lives, and how the query engine guardrails its work.

117 +

118 +```yaml

119 +journal:

120 + journal_dir: flows

121 + size_of_journal_files: 10GB

122 + duration_of_journal_files: 7d

123 + query_1m_max_window: 6h

124 + query_5m_max_window: 24h

125 + query_max_groups: 50000

126 + query_facet_max_values_per_field: 5000

127 + tiers:

128 + raw: { duration_of_journal_files: 24h }

129 + minute_1: { duration_of_journal_files: 14d }

130 + minute_5: { duration_of_journal_files: 30d }

131 + hour_1: { duration_of_journal_files: 365d }

132 +```

133 +

134 +### Top-level retention

135 +

136 +| Key | Default | Notes |

137 +|---|---|---|

138 +| `journal_dir` | `flows` | Relative paths resolve under `NETDATA_CACHE_DIR` (typically `/var/cache/netdata/flows`). Absolute paths are used as-is. |

139 +| `size_of_journal_files` | `10GB` | Disk budget per tier (not total). Minimum `100MB`. Set to `null` to disable size-based retention. |

140 +| `duration_of_journal_files` | `7d` | Time budget per tier. Set to `null` to disable time-based retention. |

141 +

142 +**Important.** The top-level retention applies to **every tier independently** unless you override it per-tier. So with the defaults, all four tiers (raw, 1m, 5m, 1h) share the same 10GB / 7d budget. **This is rarely what you want.** The whole point of having rollup tiers is to keep them around longer than raw. See per-tier overrides below.

143 +

144 +Either limit triggers rotation. With size = 10GB and duration = 7d, the tier expires whichever is hit first.

145 +

146 +### Per-tier overrides

147 +

148 +```yaml

149 +tiers:

150 + raw: # name in YAML

151 + size_of_journal_files: 50GB

152 + duration_of_journal_files: 24h

153 + minute_1:

154 + duration_of_journal_files: 14d

155 + minute_5:

156 + duration_of_journal_files: 30d

157 + hour_1:

158 + duration_of_journal_files: 365d

159 + size_of_journal_files: null # time-only retention for the long tail

160 +```

161 +

162 +| YAML name | Aliases | On-disk directory |

163 +|---|---|---|

164 +| `raw` | — | `flows/raw/` |

165 +| `minute_1` | `1m`, `minute-1`, `minute1` | `flows/1m/` |

166 +| `minute_5` | `5m`, `minute-5`, `minute5` | `flows/5m/` |

167 +| `hour_1` | `1h`, `hour-1`, `hour1` | `flows/1h/` |

168 +

169 +The on-disk directory names are short (`1m`, `5m`, `1h`); the YAML keys are explicit (`minute_1`, `minute_5`, `hour_1`). Mind the difference if you go look at the disk.

170 +

171 +For each per-tier knob (`size_of_journal_files`, `duration_of_journal_files`):

172 +

173 +- **Omit the key** to inherit the top-level default.

174 +- Set to `null` to **disable** that limit on this tier.

175 +- Set to a value to override.

176 +

177 +A typical production profile is the example block above: 24 hours of raw, 2 weeks at 1-minute, 30 days at 5-minute, 1 year at 1-hour. This profile keeps detailed forensics within reach while supporting year-over-year capacity trends.

178 +

179 +### Rotation

180 +

181 +Each tier rotates files at `size_of_journal_files / 20`, clamped between 5 MB and 200 MB. Time-based rotation is fixed at one hour per file. You don't configure these directly.

182 +

183 +### Query guardrails

184 +

185 +| Key | Default | What it limits |

186 +|---|---|---|

187 +| `query_1m_max_window` | `6h` | Above this window, the dashboard skips the 1-minute tier and uses the 5-minute or 1-hour tier. |

188 +| `query_5m_max_window` | `24h` | Above this window, the dashboard skips the 5-minute tier and uses the 1-hour tier. |

189 +| `query_max_groups` | `50000` | Maximum groups returned by a single aggregation query. Past this, results overflow into a single `__overflow__` bucket and the response carries a warning. |

190 +| `query_facet_max_values_per_field` | `5000` | Maximum distinct values returned per facet field. |

191 +

192 +The query-window limits are about responsiveness — large windows on fine-grained tiers are slow. The group/value limits are about memory — wide aggregations on high-cardinality fields can blow up. Raise them carefully.

193 +

194 +## `enrichment`

195 +

196 +Enrichment is a large topic and lives in dedicated pages. The top-level enable/disable knobs:

197 +

198 +```yaml

199 +enrichment:

200 + # default_sampling_rate: 1024 # set to override; default is unset (rate=1)

201 + # override_sampling_rate: { 10.1.0.0/16: 1024 } # per-prefix override map

202 + default_sampling_rate: ~

203 + override_sampling_rate: {}

204 + metadata_static: { exporters: {} }

205 + geoip: { asn_database: [], geo_database: [] }

206 + networks: {}

207 + network_sources: {}

208 + exporter_classifiers: []

209 + interface_classifiers: []

210 + classifier_cache_duration: 5m

211 + asn_providers: [flow, routing, geoip]

212 + net_providers: [flow, routing]

213 + routing_static: { prefixes: {} }

214 + routing_dynamic:

215 + bmp: { enabled: false }

216 + bioris: { enabled: false }

217 +```

218 +

219 +Detailed configuration of each section lives on its own page:

220 +

221 +- [GeoIP](/docs/network-flows/enrichment/ip-intelligence.md)

222 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md)

223 +- [Classifiers](/docs/network-flows/enrichment/classifiers.md)

224 +- [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md)

225 +- [BMP routing](/docs/network-flows/enrichment/bgp-routing.md)

226 +- [BioRIS](/docs/network-flows/enrichment/bgp-routing.md)

227 +- [Network sources](/docs/network-flows/enrichment/network-identity.md)

228 +- [Decapsulation](/docs/network-flows/enrichment/decapsulation.md)

229 +

230 +The enrichment section has no CLI flag — it is YAML-only.

231 +

232 +## Common edits

233 +

234 +### Listen on a different port

235 +

236 +```yaml

237 +listener:

238 + listen: "0.0.0.0:9995"

239 +```

240 +

241 +### Bind to a specific address

242 +

243 +```yaml

244 +listener:

245 + listen: "10.0.0.10:2055"

246 +```

247 +

248 +### Disable a protocol you don't use

249 +

250 +```yaml

251 +protocols:

252 + v5: false

253 +```

254 +

255 +### Move the journal directory

256 +

257 +```yaml

258 +journal:

259 + journal_dir: /var/lib/netflow

260 +```

261 +

262 +Absolute paths are used as-is. Relative paths resolve under `NETDATA_CACHE_DIR`.

263 +

264 +### Strip VXLAN tunnel headers

265 +

266 +```yaml

267 +protocols:

268 + decapsulation_mode: vxlan

269 +```

270 +

271 +The plugin reads the inner 5-tuple from `dataLinkFrameSection` records (IPFIX IE 315) when the exporter ships them.

272 +

273 +### Production retention profile

274 +

275 +```yaml

276 +journal:

277 + size_of_journal_files: 100GB

278 + duration_of_journal_files: 7d

279 + tiers:

280 + raw:

281 + size_of_journal_files: 200GB

282 + duration_of_journal_files: 24h

283 + minute_1:

284 + duration_of_journal_files: 14d

285 + minute_5:

286 + duration_of_journal_files: 30d

287 + hour_1:

288 + duration_of_journal_files: 365d

289 + size_of_journal_files: null

290 +```

291 +

292 +The default 10GB / 7d on every tier is too tight for most production deployments. This profile gives you 24 hours of full-detail forensics, 14 days of 1-minute trends, 30 days of 5-minute snapshots, and a year of hourly aggregates. Storage required scales with your flow rate — see [Sizing and Capacity Planning](/docs/network-flows/sizing-capacity.md).

293 +

294 +## Things that go wrong

295 +

296 +- **The plugin doesn't start.** Check `journalctl -u netdata --since "5 minutes ago" | grep netflow`. The most common cause is a typo in a YAML key (strict mode rejects unknowns).

297 +- **Edits don't take effect.** Restart Netdata. There is no DynCfg integration for the plugin's configuration.

298 +- **CLI flags I added don't do anything.** When running under Netdata, only the YAML is read.

299 +- **Tiers fill up faster than expected.** All tiers share the top-level retention by default. Set explicit per-tier overrides.

300 +- **Queries time out at 30 seconds.** Function calls have a hard 30s timeout in the plugin. If your query is too wide, narrow the time range or add filters that let a higher tier serve it.

301 +- **`__overflow__` appears in results.** A group-by exceeded `query_max_groups` (default 50 000). Either narrow the filter, reduce the number of group-by fields, or raise the limit.

302 +

303 +## What's next

304 +

305 +- [Retention and Querying](/docs/network-flows/retention-querying.md) — How the four tiers work and how the dashboard picks one.

306 +- [Sizing and Capacity Planning](/docs/network-flows/sizing-capacity.md) — How much disk and CPU you need.

307 +- [Validation and Data Quality](/docs/network-flows/validation.md) — How to confirm the data is right.

308 +- [Troubleshooting](/docs/network-flows/troubleshooting.md) — When things break.

docs/network-flows/enrichment/asn-resolution.md new

+121

@@ -0,0 +1,121 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/asn-resolution.md"

3 +sidebar_label: "ASN Resolution"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Enrichment"

6 +keywords: ['asn', 'as resolution', 'bgp', 'enrichment']

7 +endmeta-->

8 +

9 +# ASN resolution

10 +

11 +ASN resolution is how Netdata fills in the `SRC_AS`, `DST_AS`, `SRC_AS_NAME`, and `DST_AS_NAME` fields on every flow record. It uses a configurable provider chain — your flow data first, then dynamic routing if you have BMP/BioRIS/static prefixes set up, then the GeoIP database as a fallback. The chain runs independently for source and destination IPs.

12 +

13 +## Numbers vs names

14 +

15 +The two are resolved by completely different code paths.

16 +

17 +**AS numbers** (`SRC_AS`, `DST_AS`) come from the **provider chain** (`asn_providers`). The chain walks providers in order; the first one that returns a non-zero AS wins.

18 +

19 +**AS names** (`SRC_AS_NAME`, `DST_AS_NAME`) always come from the **ASN database** lookup, regardless of the chain. Whichever AS number ends up resolved, the name is rendered as `AS{n} {organisation}` from the ASN MMDB. If the MMDB doesn't know the name, the rendering is `AS{n}` with no trailing label. If the resolved AS is `0`, the rendering is `AS0 Unknown ASN` (or `AS0 Private IP Address Space` if the MMDB tagged the IP as private).

20 +

21 +There is **no static configuration option** for AS names. You can set `enrichment.networks.<cidr>.asn` to override the AS *number*, but the name is always looked up.

22 +

23 +## The provider chain

24 +

25 +```yaml

26 +enrichment:

27 + asn_providers: [flow, routing, geoip]

28 +```

29 +

30 +This is the default. The plugin walks it left-to-right; the first provider returning a non-zero value wins.

31 +

32 +| Provider | What it reads | Notes |

33 +|---|---|---|

34 +| `flow` | `SRC_AS` / `DST_AS` from the flow record itself | What the exporter sent |

35 +| `flow_except_private` | Same, but treats private/reserved AS numbers as zero | Use when your exporters announce private AS that you don't want to surface |

36 +| `flow_except_default_route` | Same, but treats AS 0 with mask 0 as zero | Use when default-route flows pollute your top-N |

37 +| `routing` | `lookup_routing()` — BMP runtime, BioRIS, static prefixes | Requires routing enrichment to be configured |

38 +| `routing_except_private` | Same as `routing`, with the private filter | |

39 +| `geoip` | **Always returns 0** — terminal short-circuit | See below |

40 +

41 +`geoip` is **terminal**: putting it in the chain stops the chain at that position. Once reached, the chain returns 0, and the GeoIP MMDB's AS data is then re-applied separately (without going through the chain). Because of that, `geoip` is meaningful only as the last entry in the chain — it acts as "let GeoIP fill in if nothing else did".

42 +

43 +If you put `[geoip, flow, routing]`, you effectively set every AS to 0, and only the GeoIP-derived AS makes it through. That is rarely what you want.

44 +

45 +### Common chain configurations

46 +

47 +| Configuration | Behaviour |

48 +|---|---|

49 +| `[flow, routing, geoip]` (default) | Trust the exporter, fall back to routing, then to GeoIP. |

50 +| `[flow, routing]` | No GeoIP at all. Use when you don't trust GeoIP for your traffic mix. |

51 +| `[routing, flow, geoip]` | Trust your BMP/BGP feed first. Use when your routers report stale AS. |

52 +| `[flow_except_private, routing, geoip]` | Drop AS 64512-65534 from flow data; let routing fill in. |

53 +

54 +### What counts as private/reserved

55 +

56 +`is_private_as` returns true for:

57 +

58 +- `0` (unknown / default route)

59 +- `23456` (RFC 4893 transition-period reserved)

60 +- `64496..=65551` (documentation, RFC 6996/RFC 5398/RFC 6793 private/reserved range)

61 +- `>= 4_200_000_000` (32-bit private range and reserved high values)

62 +

63 +These are filtered out by the `*_except_private` variants.

64 +

65 +## The network-prefix chain

66 +

67 +A second chain controls how `SRC_MASK`, `DST_MASK`, and `NEXT_HOP` get resolved:

68 +

69 +```yaml

70 +enrichment:

71 + net_providers: [flow, routing]

72 +```

73 +

74 +This is the default. Same logic: first non-empty value wins. Only `flow` and `routing` are valid here — there is no `geoip` provider for network attributes.

75 +

76 +## AS overrides via static configuration

77 +

78 +If a flow's source or destination IP falls inside a CIDR you've declared under `enrichment.networks`, and that entry includes an `asn` field, the configured value **overrides whatever the chain produced**:

79 +

80 +```yaml

81 +enrichment:

82 + networks:

83 + 198.51.100.0/24:

84 + name: customer-acme

85 + asn: 64500 # forces SRC_AS / DST_AS = 64500 for traffic in this prefix

86 +```

87 +

88 +This override is applied after the chain. It only sets the AS number — the name is still resolved from the ASN database (so it'll render as `AS64500` if your MMDB doesn't have a name, or `AS64500 Acme Corp` if it does).

89 +

90 +## What you get out of the box

91 +

92 +With the default `[flow, routing, geoip]` chain, no routing enrichment configured, and the stock ASN MMDB shipped with native packages:

93 +

94 +- `SRC_AS` / `DST_AS` populated whenever the exporter sends them (most NetFlow v9, IPFIX, sFlow exporters do for public IPs)

95 +- `SRC_AS_NAME` / `DST_AS_NAME` populated whenever the IP is in the ASN MMDB

96 +- For internal RFC 1918 addresses: `*_AS = 0`, `*_AS_NAME = AS0 Private IP Address Space` (because the stock MMDB tags private ranges with that flag)

97 +- For unknown public addresses: `*_AS = 0`, `*_AS_NAME = AS0 Unknown ASN`

98 +

99 +If you don't have an ASN MMDB at all, names render as `AS{n}` for non-zero ASNs and `AS0 Unknown ASN` for zero — the dashboard never shows blank cells.

100 +

101 +## Failure modes

102 +

103 +- **ASN MMDB missing.** With `optional: true` (the default for auto-detected files), the plugin starts and AS names render as `AS{n}` or `AS0 Unknown ASN`. With `optional: false` and a configured path, the plugin fails to start.

104 +- **AS not in any provider.** `*_AS = 0`, `*_AS_NAME = AS0 Unknown ASN`.

105 +- **Wrong order of providers.** Putting `geoip` mid-chain truncates everything after it. Putting `routing` before `flow` makes routing data win over what the exporter sent — fine if your BGP feed is more accurate than your exporter's view.

106 +- **Empty `asn_providers`.** No validation rejects this. The plugin starts but every AS number resolves to 0; only `enrichment.networks.<cidr>.asn` overrides can produce non-zero AS.

107 +

108 +## What can go wrong

109 +

110 +- **AS numbers all zero.** Check the chain. If `[geoip, ...]` is the order, `geoip` short-circuits to 0. Reorder to put `geoip` last.

111 +- **Wrong AS for a known prefix.** Likely the exporter's view differs from the BGP table. Override per-prefix via `enrichment.networks.<cidr>.asn`, or reorder the chain to `[routing, flow, geoip]`.

112 +- **Names show `AS{n}` without an organisation.** The MMDB doesn't have a name for that AS. Either accept it or use a richer MMDB.

113 +- **Names show wrong organisation.** ASN ownership data is best-effort and lags real-world transfers by weeks. Refresh the MMDB. If that doesn't help, file an issue with the database vendor — Netdata is a passive consumer.

114 +

115 +## What's next

116 +

117 +- [GeoIP](/docs/network-flows/enrichment/ip-intelligence.md) — How the ASN MMDB gets installed and refreshed.

118 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md) — Per-prefix AS overrides and network labels.

119 +- [BMP routing](/docs/network-flows/enrichment/bgp-routing.md) — Live BGP feed as an AS source for the `routing` provider.

120 +- [BioRIS](/docs/network-flows/enrichment/bgp-routing.md) — RIPE RIS as an AS source for the `routing` provider.

121 +- [Network sources](/docs/network-flows/enrichment/network-identity.md) — HTTP-fetched prefix metadata.

docs/network-flows/enrichment/bgp-routing.md new

+115

@@ -0,0 +1,115 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/bgp-routing.md"

3 +sidebar_label: "BGP Routing"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Enrichment Concepts"

6 +keywords: ['bgp', 'routing', 'bmp', 'bioris', 'enrichment', 'concept']

7 +endmeta-->

8 +

9 +# BGP Routing

10 +

11 +BGP-routing enrichment fills `SRC_AS`, `DST_AS`, `SRC_MASK`, `DST_MASK`, `NEXT_HOP`, `DST_AS_PATH`, `DST_COMMUNITIES`, and `DST_LARGE_COMMUNITIES` from a live BGP feed. Two transports are supported, both of which feed the same in-memory routing trie:

12 +

13 +- **BMP** (BGP Monitoring Protocol, RFC 7854) — routers push their BGP updates to Netdata over TCP

14 +- **BioRIS** — Netdata pulls BGP data from a [bio-rd](https://github.com/bio-routing/bio-rd) `cmd/ris/` daemon over gRPC

15 +

16 +This page covers the **cross-cutting concept**: how the trie works, how the two sources combine, what survives a restart, and what to expect operationally. For per-protocol setup, follow the integration cards on Learn (BMP and bio-rd / RIPE RIS).

17 +

18 +## What gets enriched

19 +

20 +BMP and BioRIS populate the same fields. When a flow's source or destination IP matches a learned BGP route:

21 +

22 +| Field | Side | Notes |

23 +|---|---|---|

24 +| `SRC_AS` / `DST_AS` | both | When the `routing` provider in `asn_providers` chain reaches BGP data |

25 +| `SRC_MASK` / `DST_MASK` | both | When the `routing` provider in `net_providers` chain reaches BGP data |

26 +| `NEXT_HOP` | dest only | BGP next-hop from the destination route |

27 +| `DST_AS_PATH` | dest only | Full BGP AS path (CSV of ASNs) |

28 +| `DST_COMMUNITIES` | dest only | Standard BGP communities (CSV of u32) |

29 +| `DST_LARGE_COMMUNITIES` | dest only | RFC 8092 large communities |

30 +

31 +Two notes:

32 +

33 +- AS *names* (`*_AS_NAME`) come from the [GeoIP/ASN MMDB](/docs/network-flows/enrichment/ip-intelligence.md), not BGP. BGP gives you accurate AS *numbers* and path/communities; the names come from the ASN database.

34 +- Source-side AS path and communities are **not** surfaced. BGP path attributes are most meaningful for the destination of the traffic.

35 +

36 +## Shared trie

37 +

38 +Both BMP and BioRIS populate a single in-memory routing trie keyed by IP prefix. Each prefix entry holds a list of routes (one per `(peer, route_key)` tuple), so multipath BGP and multiple BGP peers contributing the same prefix coexist cleanly.

39 +

40 +When both BMP and BioRIS are enabled, they contribute to the same trie. Lookups pick the best-matching route across both sources, preferring routes whose exporter or next-hop matches the flow being enriched, falling back to longest-prefix-match.

41 +

42 +This is intentional: a deployment that runs BMP from internal routers and BioRIS for external (RIPE RIS) views gets unified enrichment without duplicate trie entries.

43 +

44 +## Memory growth

45 +

46 +The trie has **no time-based eviction**. Routes are only removed via:

47 +

48 +- Explicit BGP withdrawal (`MP_UNREACH`, `withdraw_routes`)

49 +- Peer Down notification (BMP) — clears all routes for the affected peer

50 +- TCP disconnect (BMP) followed by the `keep` interval expiring (default 5 minutes) — clears all routes for that session

51 +- bio-rd refresh cycle — explicit removal of routes for routers that have disappeared

52 +

53 +A full IPv4+IPv6 BGP table is roughly 1.2M prefixes per peer (2026 figures). Each entry stores the AS-path `Vec<u32>`, communities `Vec<u32>`, large communities `Vec<(u32,u32,u32)>`, plus a `route_key` `String` per path. Expect several hundred MB of resident memory per peer for a full feed.

54 +

55 +Plan capacity accordingly. If you run many peers with full feeds, watch the agent's RSS.

56 +

57 +## Restart behaviour

58 +

59 +The trie is **not persisted**. Restarting the netflow plugin wipes BGP-derived data. Routes are re-learned as routers re-send Initiation + Update messages (for BMP) or as bio-rd's next refresh cycle dumps the RIB (BioRIS).

60 +

61 +Convergence times after restart:

62 +

63 +| Source | Typical convergence |

64 +|---|---|

65 +| FRR over BMP | seconds (FRR re-emits everything immediately) |

66 +| Cisco IOS-XR over BMP | minutes (IOS-XR's initial-refresh has a configurable spread) |

67 +| Juniper JunOS over BMP | seconds to minutes (depends on station options) |

68 +| BioRIS over RIPE RIS | minutes (full DumpRIB takes a while for large feeds) |

69 +

70 +Until convergence, BGP-derived enrichment is incomplete. Plan restarts during low-traffic windows if BGP attribution matters for your workflow.

71 +

72 +## Provider chain integration

73 +

74 +BGP-derived routes contribute to flow enrichment via the `routing` entry in the [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md) provider chain:

75 +

76 +```yaml

77 +enrichment:

78 + asn_providers: [flow, routing, geoip] # default

79 + net_providers: [flow, routing] # default

80 +```

81 +

82 +With the defaults, an exporter-supplied AS number wins over BGP. To prefer BGP over the exporter (useful when your BMP/BioRIS feed is more accurate than the exporter's view), reorder:

83 +

84 +```yaml

85 +enrichment:

86 + asn_providers: [routing, flow, geoip]

87 +```

88 +

89 +`bmp` is accepted as an alias for `routing` in the provider list, for backward compatibility.

90 +

91 +## Integration test gap

92 +

93 +The runtime path of both BMP and BioRIS — TCP listener / gRPC client, framed decode loop, trie apply, per-router cleanup — is **not** integration-tested in this repository. The parsing layers (BMP message parsing, gRPC proto conversion) are well-unit-tested, but end-to-end against real router firmware or real bio-rd daemons is not exercised.

94 +

95 +Implications:

96 +

97 +- The features ship because the parsing is solid and the runtime is built on standard tokio + netgauze + tonic primitives.

98 +- Vendor compatibility (Cisco IOS-XR / IOS-XE, Juniper JunOS, Arista EOS, FRR) is not validated by tests in this repository.

99 +- Treat configuration changes as production-impacting. Validate against your specific gear before relying on BGP-derived data for capacity or security decisions.

100 +

101 +## What can go wrong

102 +

103 +- **No connections forming (BMP).** Routers initiate BMP sessions to the plugin. Check the router side (`show bmp` / `show bmp connections` / `show bmp targets`). The plugin doesn't proactively retry; it waits.

104 +- **gRPC deadline exceeded (BioRIS).** Default timeout 200 ms is aggressive over the public internet. Raise to 2-5 s.

105 +- **Memory growth without bound.** A full BGP feed is permanent (no eviction). Plan capacity.

106 +- **Plugin restart wipes the trie.** Re-converge takes seconds (FRR) to minutes (IOS-XR). Schedule restarts off-peak.

107 +- **AS path inconsistent with the exporter's view.** Different vantage points see different paths. This is normal in BGP. If your exporter and your BMP-feeding router are different boxes with different routing tables, expect divergence.

108 +- **Empty BGP data after enabling.** Check the per-provider integration card for the specific protocol's setup gotchas — e.g., FRR requires `-M bmp` in `/etc/frr/daemons` (otherwise every BMP command silently fails).

109 +

110 +## What's next

111 +

112 +- **BMP** integration card — how to enable the listener, configure routers (Cisco, Juniper, Arista, FRR).

113 +- **bio-rd / RIPE RIS** integration card — how to set up bio-rd, configure the gRPC client.

114 +- [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md) — How BGP plugs into the provider chain.

115 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md) — Per-prefix overrides that win over BGP.

docs/network-flows/enrichment/classifiers.md new

+182

@@ -0,0 +1,182 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/classifiers.md"

3 +sidebar_label: "Classifiers"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Enrichment"

6 +keywords: ['classifiers', 'akvorado', 'enrichment', 'rules', 'expression']

7 +endmeta-->

8 +

9 +# Classifiers

10 +

11 +Classifiers tag exporters and interfaces using small expression-based rules. Where [static metadata](/docs/network-flows/enrichment/static-metadata.md) requires you to enumerate every exporter and every ifIndex, classifiers let you write a few rules that match many cases — by name pattern, by IP, by SNMP description, by speed, and so on.

12 +

13 +The plugin's classifier language is **Akvorado-compatible** for the documented operators and actions. It is implemented as a hand-written expression parser in Rust, not jq/jaq, and supports a subset of Akvorado's full expression language. If you've written Akvorado classifiers before, your rules will likely work; if you've written `expr-lang` rules with arithmetic, ternaries, or lambdas, those features are not available here.

14 +

15 +## Two classifier lists

16 +

17 +| Block | Runs | Sees |

18 +|---|---|---|

19 +| `enrichment.exporter_classifiers` | Once per exporter (cached) | Exporter IP and name, current classification fields |

20 +| `enrichment.interface_classifiers` | Once per (exporter, interface) pair, twice per flow (in + out) | Exporter fields, plus interface index/name/description/speed/VLAN, plus current classification |

21 +

22 +Rules are evaluated in YAML order. The plugin short-circuits the list when all classification slots are filled.

23 +

24 +## What a rule can read

25 +

26 +Identifiers available to **exporter classifiers**:

27 +

28 +- `Exporter.IP` — the exporter's IP, as a string

29 +- `Exporter.Name` — the exporter's friendly name (from static metadata, or falls back to the IP)

30 +- `CurrentClassification.Group`, `.Role`, `.Site`, `.Region`, `.Tenant` — values already set (by static metadata, or by an earlier rule)

31 +

32 +Identifiers available to **interface classifiers**:

33 +

34 +- All of the above

35 +- `Interface.Index` — the SNMP ifIndex (integer)

36 +- `Interface.Name`, `Interface.Description` — from static metadata

37 +- `Interface.Speed` — in bits per second

38 +- `Interface.VLAN` — from the flow record's `SRC_VLAN` / `DST_VLAN` (depending on direction)

39 +- `CurrentClassification.Connectivity`, `.Provider`, `.Boundary`, `.Name`, `.Description` — already set

40 +

41 +The plugin does NOT poll SNMP itself, so `Interface.Name` / `Description` / `Speed` come only from what you've configured under `metadata_static`. If you haven't configured them, those identifiers will be empty.

42 +

43 +## What a rule can do

44 +

45 +### Set classification fields

46 +

47 +| Action | Result |

48 +|---|---|

49 +| `Classify("v")` or `ClassifyGroup("v")` | Set `EXPORTER_GROUP` |

50 +| `ClassifyRole("v")` | Set `EXPORTER_ROLE` |

51 +| `ClassifySite("v")` | Set `EXPORTER_SITE` |

52 +| `ClassifyRegion("v")` | Set `EXPORTER_REGION` |

53 +| `ClassifyTenant("v")` | Set `EXPORTER_TENANT` |

54 +| `ClassifyProvider("v")` | Set `IN_IF_PROVIDER` / `OUT_IF_PROVIDER` |

55 +| `ClassifyConnectivity("v")` | Set `IN_IF_CONNECTIVITY` / `OUT_IF_CONNECTIVITY` |

56 +| `ClassifyExternal()` / `ClassifyInternal()` | Set `IN_IF_BOUNDARY` / `OUT_IF_BOUNDARY` |

57 +| `SetName("v")` | Set `IN_IF_NAME` / `OUT_IF_NAME` (or exporter name when in an exporter rule) |

58 +| `SetDescription("v")` | Set `IN_IF_DESCRIPTION` / `OUT_IF_DESCRIPTION` |

59 +

60 +`Classify*Regex(input, pattern, template)` variants exist for every action above. The pattern is a Rust regex; the template uses `$1`, `$2`, `${name}` capture references.

61 +

62 +### Drop the flow

63 +

64 +`Reject()` discards the flow record. Always guard it behind a condition — at top level it drops everything.

65 +

66 +### Format strings

67 +

68 +`Format("...", arg1, arg2)` mimics Go's `fmt.Sprintf` for `%s`, `%v`, `%d`, `%%`. Use it to build values from multiple inputs:

69 +

70 +```

71 +ClassifyTenant(Format("tenant-%s", Exporter.Name))

72 +```

73 +

74 +## What rules can match against

75 +

76 +Operators (highest to lowest precedence):

77 +

78 +| Form | Meaning |

79 +|---|---|

80 +| `value == X`, `value != X` | equality / inequality |

81 +| `value > X`, `value >= X`, `value < X`, `value <= X` | numeric or lexicographic comparison |

82 +| `value in [a, b, c]` | membership |

83 +| `value contains "x"` | substring (string only) |

84 +| `value startsWith "x"`, `value endsWith "x"` | prefix / suffix (string only) |

85 +| `value matches "pattern"` | regex match (Rust regex) |

86 +| `cond1 && cond2`, `cond1 and cond2` | logical AND |

88 +| `!cond`, `not cond` | negation |

89 +| `(cond)` | grouping |

90 +

91 +Whitespace and newlines are ignored, so multi-line rules work. Strings are JSON-quoted.

92 +

93 +## Important behavioural rules

94 +

95 +### First write wins

96 +

97 +Each classification slot is single-write. Once a rule sets `EXPORTER_GROUP`, no subsequent rule can change it. Order rules from most-specific to least-specific.

98 +

99 +### Static metadata overrides classifiers entirely

100 +

101 +If `metadata_static.exporters` set **any** exporter classification field for this exporter, **none of the exporter classifiers run**. Same for interfaces: if static metadata set any of provider, connectivity, or boundary for an interface, the interface classifiers do not run for that interface.

102 +

103 +This is "Akvorado parity" behaviour — operator-provided classification has priority. Don't try to mix them on the same target.

104 +

105 +### `Classify*` value normalisation

106 +

107 +The string passed to `Classify*` actions is **lowercased and stripped to ASCII alphanumerics + `. + -`**. So `ClassifyRegion("EU West")` becomes `euwest`. If you want to preserve casing or whitespace, use `SetName` or `SetDescription` instead.

108 +

109 +### Runtime errors stop the rule list

110 +

111 +If a rule throws (e.g., comparing a string with `>`), the plugin stops evaluating further rules in that list and keeps whatever was set so far. Use `matches`, `startsWith`, or `contains` instead of `>`/`<` on string fields to avoid this.

112 +

113 +### Cache key includes resolved values

114 +

115 +The interface classifier cache keys by `(exporter, exporter classification, interface)`. When the exporter's classification changes — for example, after you push new static metadata and restart — interface caches naturally invalidate.

116 +

117 +The cache TTL is `classifier_cache_duration`, default 5 minutes (`enrichment.classifier_cache_duration`). It's a last-access TTL — entries live as long as they're queried.

118 +

119 +## Rule examples

120 +

121 +### Exporter classifiers

122 +

123 +```yaml

124 +enrichment:

125 + exporter_classifiers:

126 + # Group exporters by name pattern.

127 + - 'Exporter.Name matches "^edge-.*" && Classify("edge")'

128 + - 'Exporter.Name matches "^core-.*" && Classify("core")'

129 +

130 + # Site by IP prefix.

131 + - 'Exporter.IP startsWith "10.1." && ClassifySite("ny-dc1")'

132 + - 'Exporter.IP startsWith "10.2." && ClassifySite("par-dc1")'

133 +

134 + # Tenant computed from name.

135 + - 'ClassifyTenant(Format("tenant-%s", Exporter.Name))'

136 +

137 + # Pull a token out of the name with a regex.

138 + - 'ClassifyRegionRegex(Exporter.Name, "-([a-z]{2})-[0-9]+$", "$1")'

139 +

140 + # Drop traffic from a test exporter.

141 + - 'Exporter.IP startsWith "192.0.2." && Reject()'

142 +

143 + classifier_cache_duration: 5m

144 +```

145 +

146 +### Interface classifiers

147 +

148 +```yaml

149 +enrichment:

150 + interface_classifiers:

151 + # Provider from a description prefix.

152 + - 'Interface.Description startsWith "BACKBONE-LUMEN" && ClassifyProvider("Lumen")'

153 + - 'Interface.Description startsWith "BACKBONE-COGENT" && ClassifyProvider("Cogent")'

154 +

155 + # Mark transit links by description keyword and tag them external.

156 + - 'Interface.Description contains "TRANSIT" && ClassifyConnectivity("transit") && ClassifyExternal()'

157 +

158 + # Anything matching the IX peering pattern.

159 + - 'Interface.Description matches "(?i)^(IX|peering)-.*" && ClassifyConnectivity("peering") && ClassifyExternal()'

160 +

161 + # 100 Gbps interfaces are core uplinks.

162 + - 'Interface.Speed >= 100000000000 && ClassifyConnectivity("core")'

163 +

164 + # Use exporter classification to scope interface rules.

165 + - 'CurrentClassification.Role == "edge" && ClassifyExternal()'

166 +```

167 +

168 +## What can go wrong

169 +

170 +- **A rule fails to parse and the plugin won't start.** Look at the journal — the error message includes the index and a parser context.

171 +- **Classifiers aren't running on an exporter.** Likely cause: static metadata already set a classification field for that exporter, which suppresses all classifier rules for it.

172 +- **A rule sets a value but it appears differently in the dashboard.** `Classify*` actions normalise (lowercase + strip non-alphanumeric). Use `SetName` for human-readable values.

173 +- **The first rule in the list always wins.** First-write-wins per slot. Order rules from most-specific to least-specific.

174 +- **A rule that worked at startup stops matching later.** Cached results expire after `classifier_cache_duration`. If you change rules, restart the plugin so the cache clears completely.

175 +- **Comparison error stops processing.** Comparing a string with `>` throws — subsequent rules in the list are skipped. Use string-safe operators.

176 +- **`ClassifyExternal` doesn't fire on the egress side.** Interface classifiers run twice — once for the input interface, once for the output. Both invocations see the same classifier list. If your rule sets `ClassifyExternal()` on a specific ifIndex, it applies whether that ifIndex is `IN_IF` or `OUT_IF`.

177 +

178 +## What's next

179 +

180 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md) — Declarative labelling that runs before classifiers.

181 +- [GeoIP](/docs/network-flows/enrichment/ip-intelligence.md) — Country / city / AS-name labelling.

182 +- [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md) — How `SRC_AS` / `DST_AS` get filled in.

docs/network-flows/enrichment/decapsulation.md new

+126

@@ -0,0 +1,126 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/decapsulation.md"

3 +sidebar_label: "Decapsulation"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Enrichment"

6 +keywords: ['decapsulation', 'srv6', 'vxlan', 'tunnel', 'overlay']

7 +endmeta-->

8 +

9 +# Decapsulation

10 +

11 +Decapsulation extracts the inner packet from tunnelled traffic so the dashboard reflects the actual endpoints, not the tunnel endpoints. Two modes are supported: **SRv6** and **VXLAN**.

12 +

13 +This is useful when your routers are observing overlay traffic — VXLAN-encapsulated VM traffic between hypervisors, SRv6-encapsulated data-centre fabric, etc. Without decap, you see the same "10.0.0.x → 10.0.0.y" flow for every VM-to-VM conversation, which tells you nothing.

14 +

15 +## Modes

16 +

17 +```yaml

18 +protocols:

19 + decapsulation_mode: vxlan # one of: none, srv6, vxlan

20 +```

21 +

22 +| Mode | What it strips | What it surfaces |

23 +|---|---|---|

24 +| `none` (default) | nothing | the outer-header view |

25 +| `srv6` | IPv6 outer + extension headers + Routing Header type 4 (SRH) | the inner IPv4 (next-header 4) or IPv6 (next-header 41) packet |

26 +| `vxlan` | outer Ethernet/IP + UDP (port 4789) + 8-byte VXLAN header | the inner Ethernet frame, then the inner L3/L4 |

27 +

28 +GRE, IP-in-IP, GENEVE, and other tunnel types are **not** supported. Only SRv6 and VXLAN.

29 +

30 +## How the modes interact with each protocol

31 +

32 +Decap relies on the exporter shipping inner-packet bytes. That happens in three different ways depending on the source protocol:

33 +

34 +| Source | Inner-packet bytes carried as | Required exporter capability |

35 +|---|---|---|

36 +| **NetFlow v9** | Information Element 104 (`Layer2packetSectionData`, RFC 7270) | Exporter must include IE 104 in the template; it carries the captured frame bytes |

37 +| **IPFIX** | Information Element 315 (`dataLinkFrameSection`, RFC 7133) | Same idea, IPFIX-standard IE |

38 +| **sFlow** | Always — `SampledHeader` records carry the truncated raw packet | sFlow agents send `SampledHeader` by default for header-sampling mode |

39 +

40 +For **NetFlow v9 / IPFIX without IE 104 / 315 in the template, decapsulation does not run** — even with `decapsulation_mode: vxlan` set. Standard flow records pass through unchanged. So enabling decap on the plugin is half the work; you also have to configure your exporter to ship the frame bytes.

41 +

42 +For **sFlow, decap always runs** when the mode is set, because every flow sample carries a `SampledHeader`.

43 +

44 +## What gets surfaced

45 +

46 +When decap succeeds, the inner 5-tuple replaces the outer one in the flow record:

47 +

48 +- `SRC_ADDR` / `DST_ADDR` — inner source/destination IPs

49 +- `SRC_PORT` / `DST_PORT` — inner ports

50 +- `PROTOCOL` — inner L4 protocol

51 +- `ETYPE` — inner EtherType

52 +- `IPTOS`, `IPTTL`, `IPV6_FLOW_LABEL`, `TCP_FLAGS` — inner IP/TCP fields

53 +- `IP_FRAGMENT_ID`, `IP_FRAGMENT_OFFSET` — inner fragmentation

54 +- `ICMPV4_TYPE` / `ICMPV4_CODE` / `ICMPV6_TYPE` / `ICMPV6_CODE` — inner ICMP

55 +- `MPLS_LABELS` — inner MPLS label stack (if present)

56 +- `BYTES` — inner L3 length (so byte counts represent inner traffic, not outer overhead)

57 +

58 +For VXLAN, the inner Ethernet frame is parsed, so `SRC_MAC` / `DST_MAC` / `SRC_VLAN` / `DST_VLAN` come from the inner frame. **The outer MACs and VLANs are lost.**

59 +

60 +For SRv6, the outer is an IPv6 packet (no L2 to lose).

61 +

62 +The **VXLAN VNI is dropped**. Netdata does not surface it. If you need to distinguish overlay segments, you need a different mechanism — VLAN-tagged inner frames work, but pure VNI-based segmentation isn't visible.

63 +

64 +## Decapsulation is destructive on non-tunnel traffic

65 +

66 +When `decapsulation_mode` is set and the exporter ships records via the special L2-section path (NetFlow v9 IE 104 / IPFIX IE 315 / sFlow `SampledHeader`), but the inner packet doesn't match the configured tunnel:

67 +

68 +- For VXLAN mode: a non-VXLAN packet (different UDP port, malformed VXLAN header, or not UDP at all) is **dropped**. The flow does NOT fall back to outer-header view.

69 +- For SRv6 mode: an IPv6 packet without the right extension-header chain leading to next-header 4 or 41 is **dropped**.

70 +- For sFlow with decap on, only `SampledHeader` records are processed. `SampledIPv4`, `SampledIPv6`, `SampledEthernet`, `ExtendedSwitch`, `ExtendedRouter`, `ExtendedGateway` records are all skipped.

71 +

72 +Plain NetFlow / IPFIX flow records that don't go through the special L2-section path are **unaffected** — they pass through normally regardless of the decap setting. So enabling `decapsulation_mode: vxlan` doesn't break your normal flow stream; it only filters the L2-section path.

73 +

74 +This means decapsulation is safe to enable when:

75 +

76 +- All your tunnel-bearing exporters use the same encapsulation, AND

77 +- The L2-section / `SampledHeader` data they ship is exclusively (or near-exclusively) tunnel traffic.

78 +

79 +If you mix VXLAN and SRv6 traffic on the same exporter, you cannot decap both — the plugin has one global setting.

80 +

81 +## Configuring exporters to ship inner-packet bytes

82 +

83 +For decap to work, your exporter must include the inner-packet bytes in its export. This is platform-specific. The CLI snippets below are starting points — verify against the vendor's reference manual before deploying.

84 +

85 +### Cisco IOS-XE / IOS-XR (NetFlow v9 with `datalink mac`)

86 +

87 +```

88 +flow record FNF-WITH-MAC

89 + match ipv4 source address

90 + match ipv4 destination address

91 + match transport source-port

92 + match transport destination-port

93 + match ipv4 protocol

94 + match datalink mac source address input

95 + match datalink mac destination address input

96 + collect counter bytes

97 + collect counter packets

98 + collect timestamp absolute first

99 + collect timestamp absolute last

100 + collect datalink frame-section section header size 128

101 +```

102 +

103 +The `collect datalink frame-section` directive is what causes the exporter to include IE 104. Adjust the section size based on your maximum tunnel header size; 128 bytes covers VXLAN over Ethernet over IPv4. SRv6 inner extraction needs more — 256 or higher.

104 +

105 +### Juniper JunOS (IPFIX with frame export)

106 +

107 +JunOS' IPFIX support varies by platform. On platforms that support frame-section export, configure the template to include `dataLinkFrameSection` (IE 315). Refer to your platform's documentation.

108 +

109 +### sFlow (built-in)

110 +

111 +sFlow agents send `SampledHeader` by default. No special configuration needed beyond enabling sFlow.

112 +

113 +## Failure modes

114 +

115 +- **Exporter doesn't ship IE 104 / IE 315.** The plugin can't decap. Records pass through with outer-header view.

116 +- **Inner packet isn't VXLAN/SRv6.** With decap on, the flow is dropped. There is no "fall back to outer view" — this is intentional, but be aware.

117 +- **Truncated frame section.** The inner Ethernet/IP/L4 parsing fails and the flow is dropped.

118 +- **VXLAN on a non-standard port.** The plugin only matches UDP destination port 4789 (RFC 7348). VXLAN-GPE on 4790 and vendor-custom ports are not detected.

119 +- **VNI not visible.** Bytes 4-6 of the VXLAN header are skipped. If you need VNI-based segmentation, see if your exporter can place the VNI in a separate field; otherwise this isn't surfaceable today.

120 +

121 +## What's next

122 +

123 +- [Configuration](/docs/network-flows/configuration.md) — `protocols.decapsulation_mode` setting reference.

124 +- [Sources / NetFlow](/src/crates/netflow-plugin/integrations/netflow.md) — IE 104 export configuration.

125 +- [Sources / IPFIX](/src/crates/netflow-plugin/integrations/ipfix.md) — IE 315 export configuration.

126 +- [Sources / sFlow](/src/crates/netflow-plugin/integrations/sflow.md) — `SampledHeader` semantics.

docs/network-flows/enrichment/ip-intelligence.md

docs/network-flows/enrichment/network-identity.md new

+134

@@ -0,0 +1,134 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/network-identity.md"

3 +sidebar_label: "Network Identity"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Enrichment Concepts"

6 +keywords: ['network identity', 'network sources', 'ipam', 'cmdb', 'cloud ip ranges', 'enrichment', 'concept']

7 +endmeta-->

8 +

9 +# Network Identity

10 +

11 +Network-identity enrichment labels your own network prefixes with names, roles, sites, regions, tenants, and country / city overrides. Where [IP intelligence](/docs/network-flows/enrichment/ip-intelligence.md) tells you "this IP is in Germany" from a public database, network-identity tells you "this prefix is our staging environment in Frankfurt" from your authoritative source.

12 +

13 +The data comes from external feeds — cloud-provider published prefix lists (AWS, GCP, Azure), IPAM systems (NetBox, Infoblox, BlueCat, phpIPAM), and custom CMDBs. Each source is configured as a separate integration card. This page covers the **cross-cutting concept**: how the lookups combine, what fields can be set, the operational rules.

14 +

15 +## What it populates

16 +

17 +| Field | Notes |

18 +|---|---|

19 +| `SRC_NET_NAME` / `DST_NET_NAME` | Friendly name |

20 +| `SRC_NET_ROLE` / `DST_NET_ROLE` | Role tag (e.g., `dmz`, `office`, `iot`) |

21 +| `SRC_NET_SITE` / `DST_NET_SITE` | Physical site |

22 +| `SRC_NET_REGION` / `DST_NET_REGION` | Region |

23 +| `SRC_NET_TENANT` / `DST_NET_TENANT` | Tenant |

24 +| `SRC_COUNTRY` / `DST_COUNTRY` | Country override (when set explicitly) |

25 +| `SRC_GEO_STATE` / `DST_GEO_STATE` | State / province override |

26 +| `SRC_GEO_CITY` / `DST_GEO_CITY` | City override |

27 +

28 +What network-identity sources cannot set: `SRC_GEO_LATITUDE` / `DST_GEO_LATITUDE`, `SRC_GEO_LONGITUDE` / `DST_GEO_LONGITUDE`. Coordinates are static-only — use the [`networks` block in static metadata](/docs/network-flows/enrichment/static-metadata.md) for those.

29 +

30 +The per-row `asn` field can also override the AS *number* via the resolution chain. The AS *name* still comes from the [ASN MMDB](/docs/network-flows/enrichment/ip-intelligence.md) — there is no `asn_name` override in network-identity sources.

31 +

32 +## Lookup priority

33 +

34 +In the network-attributes resolution merge order:

35 +

36 +1. **GeoIP** seeds the base layer.

37 +2. **Network-identity sources** (cloud IP ranges, IPAM, generic IPAM) merge on top — at each prefix length (least-specific to most-specific).

38 +3. **Static `networks` config** merges last — at each prefix length, **after** network-identity sources.

39 +

40 +So when a prefix is defined in both a remote source and the static config, the static config wins on any non-empty field. This is intentional: explicit operator configuration overrides imported data.

41 +

42 +## How a fetch works

43 +

44 +For each configured source:

45 +

46 +1. The plugin issues an HTTP request (default GET, or POST if configured) at the `interval` cadence.

47 +2. Headers configured under `headers:` are added (typically for authentication).

48 +3. The response body is parsed as JSON.

49 +4. The configured `transform` (a [jaq](https://github.com/01mf02/jaq) jq-equivalent expression) runs over the parsed JSON.

50 +5. The transform must produce a stream of objects, each with a `prefix` field (a CIDR string) and any of the optional attribute fields.

51 +6. The records are merged into the network-attributes trie.

52 +

53 +Each source runs in its own task. Multiple sources fetch in parallel; within a source, only one fetch is in flight at a time.

54 +

55 +On any failure (HTTP error, JSON parse error, jq runtime error, empty result), the source backs off exponentially (starting at `interval / 10`, doubling up to `interval`) and retries. On success it resets to the configured `interval`.

56 +

57 +## The expected jq output shape

58 +

59 +The `transform` is a jq expression compiled by jaq. It receives the entire parsed JSON body and must produce a **stream of objects**.

60 +

61 +Each output object should look like:

62 +

63 +```json

64 +{

65 + "prefix": "10.0.0.0/8",

66 + "name": "internal",

67 + "role": "lan",

68 + "site": "fra1",

69 + "region": "eu-central",

70 + "country": "DE",

71 + "state": "HE",

72 + "city": "Frankfurt",

73 + "tenant": "tenant-a",

74 + "asn": 64500,

75 + "asn_name": "Internal AS"

76 +}

77 +```

78 +

79 +Required: `prefix`. All other fields are optional and default to empty / 0.

80 +

81 +The `asn` field accepts an integer (`64500`), a string (`"64500"`), or the AS notation (`"AS64500"`).

82 +

83 +If the transform produces nothing (empty result), the cycle is treated as a failure and triggers backoff. The same applies to non-object rows — every output element must be an object.

84 +

85 +## TLS verification cannot be disabled

86 +

87 +The configuration accepts the legacy keys `tls.verify` and `tls.skip_verify` for compatibility, but the validation layer **rejects** any attempt to disable verification (`tls.verify: false` or `tls.skip_verify: true`). Self-signed or internal CAs must be supplied via `tls.ca_file`. There is no override.

88 +

89 +This is deliberate. Network-identity data flows directly into enrichment that affects security investigations and capacity decisions — silently accepting MITM-able responses would corrupt every downstream analysis.

90 +

91 +## Single page only

92 +

93 +The fetch is one-shot per cycle. There is no pagination, no cursor handling, no `Link: rel=next` following. If your IPAM exposes paginated endpoints, either:

94 +

95 +- Expose a separate "all prefixes" bulk endpoint (most IPAMs have one).

96 +- Wrap with a server-side script that aggregates all pages and serves the result at one URL.

97 +

98 +## Authentication

99 +

100 +The plugin has no built-in OAuth flow, basic-auth helpers, or token refresh. Set whatever the API needs explicitly:

101 +

102 +```yaml

103 +headers:

104 + Authorization: "Token abc123"

105 +```

106 +

107 +If your endpoint needs short-lived tokens, refresh them outside Netdata and put the current valid token in the headers config (and reload).

108 +

109 +## Available sources

110 +

111 +Each is configured as a separate integration card. See the per-source card for setup details:

112 +

113 +- **AWS IP Ranges** — public AWS prefix list with per-region and per-service tagging

114 +- **GCP IP Ranges** — public GCP prefix list with per-scope and per-service tagging

115 +- **Azure IP Ranges** — published per Azure Service Tags (requires an internal mirror because Azure's URL rotates weekly)

116 +- **NetBox** — open-source IPAM / DCIM, REST API with bearer-token auth

117 +- **Generic JSON-over-HTTP IPAM** — catch-all for Infoblox, BlueCat, phpIPAM, custom CMDBs

118 +

119 +## What can go wrong

120 +

121 +- **Endpoint is paginated.** Only the first page is fetched. Use a bulk endpoint or wrap with a server-side script.

122 +- **Default interval is 60s.** Fast for an IPAM, slow for AWS/GCP ranges. Tune per source — daily is fine for cloud IP ranges, 5-15 minutes for IPAMs that change often.

123 +- **TLS verify cannot be disabled.** Use `tls.ca_file` for internal CAs.

124 +- **Empty result from the transform** is treated as failure. If your endpoint returns no prefixes (legitimate state for a quiet IPAM), the source backs off as if it errored. Workaround: have the upstream return at least one synthetic prefix.

125 +- **Authorization header must be in `headers:`**, not in the URL. URLs with embedded credentials (`https://user:pass@host`) are not specially handled.

126 +- **JSON parse errors are silent in the dashboard.** Watch the Netdata journal (`journalctl -u netdata | grep network_sources`) for warnings.

127 +- **Static config silently wins ties.** When a prefix is defined in both a remote source and `networks:`, the static config's values overwrite the remote ones. This is by design but can surprise operators expecting the remote feed to be authoritative.

128 +

129 +## What's next

130 +

131 +- **AWS IP Ranges, GCP IP Ranges, Azure IP Ranges, NetBox, Generic JSON-over-HTTP IPAM** — per-source integration cards with concrete setup instructions and example jq transforms.

132 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md) — Static `networks` block (overrides network-identity at the same prefix length).

133 +- [IP Intelligence](/docs/network-flows/enrichment/ip-intelligence.md) — The base layer that network-identity merges on top of.

134 +- [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md) — How the per-row `asn` field plugs in.

docs/network-flows/enrichment/static-metadata.md new

+227

@@ -0,0 +1,227 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/enrichment/static-metadata.md"

3 +sidebar_label: "Static Metadata"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Enrichment"

6 +keywords: ['static metadata', 'enrichment', 'networks', 'exporters', 'interface']

7 +endmeta-->

8 +

9 +# Static metadata

10 +

11 +Static metadata is the foundational enrichment for any multi-exporter deployment. It lets you give your routers, your switches, your interfaces, and your own networks the names and labels you want to see on the dashboard — instead of raw IP addresses and SNMP indexes.

12 +

13 +There are two independent configuration blocks. They populate different fields and use different lookup keys, but you typically configure both:

14 +

15 +| Block | Lookup key | What it labels |

16 +|---|---|---|

17 +| `enrichment.metadata_static.exporters` | exporter IP / CIDR + ifIndex | The exporter device and its individual interfaces |

18 +| `enrichment.networks` | source / destination IP | Your own networks (CIDRs you operate) |

19 +

20 +## What it populates

21 +

22 +### From `metadata_static.exporters`

23 +

24 +Per-exporter (matched by source IP of the UDP datagram):

25 +

26 +- `EXPORTER_NAME`, `EXPORTER_GROUP`, `EXPORTER_ROLE`, `EXPORTER_SITE`, `EXPORTER_REGION`, `EXPORTER_TENANT`

27 +

28 +Per-interface (matched by ifIndex from the flow record):

29 +

30 +- `IN_IF_NAME` / `OUT_IF_NAME`

31 +- `IN_IF_DESCRIPTION` / `OUT_IF_DESCRIPTION`

32 +- `IN_IF_SPEED` / `OUT_IF_SPEED` (in **bits per second**)

33 +- `IN_IF_PROVIDER` / `OUT_IF_PROVIDER`

34 +- `IN_IF_CONNECTIVITY` / `OUT_IF_CONNECTIVITY`

35 +- `IN_IF_BOUNDARY` / `OUT_IF_BOUNDARY` (`1` = external, `2` = internal)

36 +

37 +### From `networks`

38 +

39 +Per source/destination IP (matched by CIDR):

40 +

41 +- `SRC_NET_NAME` / `DST_NET_NAME`

42 +- `SRC_NET_ROLE` / `DST_NET_ROLE`

43 +- `SRC_NET_SITE` / `DST_NET_SITE`

44 +- `SRC_NET_REGION` / `DST_NET_REGION`

45 +- `SRC_NET_TENANT` / `DST_NET_TENANT`

46 +- `SRC_COUNTRY` / `DST_COUNTRY`, `SRC_GEO_STATE` / `DST_GEO_STATE`, `SRC_GEO_CITY` / `DST_GEO_CITY` — overrides for the GeoIP-derived fields

47 +- `SRC_GEO_LATITUDE` / `DST_GEO_LATITUDE`, `SRC_GEO_LONGITUDE` / `DST_GEO_LONGITUDE` — overrides for the coordinate fields

48 +- `SRC_AS_NAME` / `DST_AS_NAME` — only when the configured `asn` causes the chain to render `AS{n}` and the MMDB has a matching name

49 +

50 +The `networks` block can also override the AS **number** via the `asn` field, for matching prefixes. AS **names** still come from the ASN database — see [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md).

51 +

52 +## Configuration

53 +

54 +### Naming exporters and interfaces

55 +

56 +```yaml

57 +enrichment:

58 + metadata_static:

59 + exporters:

60 + 192.0.2.10: # single IP (treated as /32)

61 + name: edge-router-1

62 + site: par1

63 + region: eu-west

64 + role: edge

65 + tenant: tenant-a

66 + default: # template applied to interfaces not in if_indexes

67 + description: unclassified port

68 + if_indexes:

69 + 1:

70 + name: Gi0/0/1

71 + description: uplink to ISP-A

72 + speed: 10000000000 # 10 Gbps in bits per second

73 + provider: isp-a

74 + connectivity: transit

75 + boundary: external

76 + 2:

77 + name: Gi0/0/2

78 + description: LAN core

79 + speed: 1000000000

80 + connectivity: lan

81 + boundary: internal

82 +```

83 +

84 +The `if_indexes` map keys by the integer ifIndex the router sends in flow records. If a flow arrives with an ifIndex not present in the map, the `default` interface block is used. The `skip_missing_interfaces: true` option overrides this — when set, missing entries get no interface labels at all.

85 +

86 +### Matching multiple exporters with one block

87 +

88 +CIDR prefixes work too. Longest-prefix match wins.

89 +

90 +```yaml

91 +enrichment:

92 + metadata_static:

93 + exporters:

94 + 198.51.100.0/24: # all routers in this subnet

95 + site: dc-fra1

96 + region: eu-central

97 + role: spine

98 + default:

99 + connectivity: lan

100 + boundary: internal

101 + 198.51.100.10: # specific override for one IP

102 + name: spine-fra1-a

103 + if_indexes:

104 + 1:

105 + name: 100Ge-0/0/1

106 + description: leaf-uplink

107 + speed: 100000000000

108 + connectivity: transit

109 + boundary: external

110 +```

111 +

112 +### Tagging your own networks

113 +

114 +```yaml

115 +enrichment:

116 + networks:

117 + 10.0.0.0/8:

118 + name: corp-internal

119 + role: internal

120 + tenant: tenant-a

121 + 172.16.0.0/12:

122 + name: corp-internal

123 + role: internal

124 + tenant: tenant-a

125 + 192.168.0.0/16:

126 + name: corp-internal

127 + role: internal

128 + tenant: tenant-a

129 + 198.51.100.0/24: # a public block you operate

130 + name: customer-acme

131 + role: customer

132 + site: par1

133 + country: FR

134 + city: Paris

135 + latitude: 48.8566

136 + longitude: 2.3522

137 + asn: 64500 # forces SRC_AS / DST_AS for traffic in this prefix

138 + 203.0.113.0/24: transit-a # shorthand: name only

139 +```

140 +

141 +Two things to know:

142 +

143 +- The `networks` map merges all containing CIDRs in ascending prefix-length order — least-specific first, with more-specific overrides. A `/24` entry inherits any non-empty fields from a containing `/16` entry, and adds or overwrites its own fields.

144 +- The shorthand form (`203.0.113.0/24: transit-a`) sets only the `name`. All other fields are empty.

145 +

146 +## Lookup priority and pipeline order

147 +

148 +Within an exporter:

149 +

150 +1. **`metadata_static.exporters` longest-prefix match** wins for exporter labels and interface labels.

151 +2. **`if_indexes` lookup** runs against the ifIndex from the flow record, falling back to `default` (or returning empty when `skip_missing_interfaces: true`).

152 +

153 +Within a flow's source/destination IP:

154 +

155 +1. **GeoIP** runs first as the base layer.

156 +2. **`network_sources` (remote feeds)** merge on top.

157 +3. **`networks` (static config)** merges last and wins on any non-empty field.

158 +

159 +The two paths run independently. An exporter IP that also matches a `networks` entry will get **both** treatments — exporter labels for the device, network labels for any traffic to or from that IP.

160 +

161 +## Things to know

162 +

163 +### `IN_IF_BOUNDARY` / `OUT_IF_BOUNDARY` semantics

164 +

165 +These label **the interface itself**, not the direction of traffic:

166 +

167 +- `1` = external — the port faces the outside world (Internet, peer, transit)

168 +- `2` = internal — the port faces your own infrastructure

169 +- `0` (or omitted) = undefined — the field is removed from the output

170 +

171 +Filtering for `IN_IF_BOUNDARY=1` cleanly gives you "traffic that arrived from outside". The encoding is intentional even if `1` for "external" looks counter-intuitive.

172 +

173 +The values `external` and `internal` are also accepted as strings (case-insensitive) in the YAML.

174 +

175 +### `speed` is in bits per second

176 +

177 +A 1 Gbps interface is `1000000000`, not `1000`. Operators thinking in megabits or gigabits will get the speed wrong by a factor of 1000 to 1 000 000. The plugin treats `speed: 0` as "not set" and removes the field from the output.

178 +

179 +### CIDR prefixes accept single IPs

180 +

181 +`192.0.2.10` and `192.0.2.10/32` are equivalent. Use whichever is clearer.

182 +

183 +### `networks.<cidr>.asn` overrides only the number

184 +

185 +Setting `asn: 64500` overrides whatever the [ASN resolution chain](/docs/network-flows/enrichment/asn-resolution.md) computed. The AS *name* still comes from the ASN database — there is no `asn_name` config field.

186 +

187 +### Coordinates are silently dropped if invalid

188 +

189 +`latitude: 91.5` (out of range) sets the field to an empty string with no error. Same for non-finite values. Validate manually if your data is important.

190 +

191 +### Renamed interfaces don't auto-track

192 +

193 +`if_indexes` keys by the numeric ifIndex. If a router renumbers its interfaces (line-card reseat, stack rebuild), the old ifIndex no longer matches and the per-interface block silently no longer applies. Audit after hardware changes.

194 +

195 +### Static metadata blocks classifiers

196 +

197 +If `metadata_static.exporters` set **any** classification field (group / role / site / region / tenant) for an exporter, the [classifiers](/docs/network-flows/enrichment/classifiers.md) do not run for that exporter at all. The same applies to interfaces: if static metadata set any of provider / connectivity / boundary, the interface classifiers don't run for that interface. Plan accordingly.

198 +

199 +## Sampling rate overrides

200 +

201 +Sampling rates can also be configured per exporter prefix here, in case your exporter doesn't carry the rate or you want to override it:

202 +

203 +```yaml

204 +enrichment:

205 + default_sampling_rate: 1 # global fallback

206 + override_sampling_rate:

207 + 10.1.0.0/16: 1024 # override for this network of exporters

208 +```

209 +

210 +`default_sampling_rate` applies when the flow record doesn't carry a rate and no override matches. `override_sampling_rate` always wins when its prefix matches the exporter IP. Both accept either an integer (uniform rate) or a CIDR-keyed map.

211 +

212 +## What can go wrong

213 +

214 +- **Wrong CIDR matches.** Overlapping ranges merge ascending — a more-specific entry that leaves a field blank will inherit the supernet's value. To clear a field on a more-specific entry, you must set it explicitly to a sentinel value, not leave it blank.

215 +- **Forgotten internal range.** Until you declare your RFC 1918 / RFC 6598 / link-local ranges as `networks` entries, GeoIP can return spurious data for them.

216 +- **Renamed interface no longer matches.** ifIndex keys are numeric; renames or hardware changes break the mapping silently.

217 +- **Stale exporter prefix.** A new device with a different management IP doesn't match an old block. Audit when you replace gear.

218 +- **`speed: 1000` means 1 kbps.** Use bits per second.

219 +- **`boundary: 0` is indistinguishable from "not set"**, both result in field removal. If you want explicit "undefined" use the string `"undefined"`.

220 +- **Lat / lng silent drop.** Invalid values become empty strings. The map quietly stops drawing the marker.

221 +

222 +## What's next

223 +

224 +- [GeoIP](/docs/network-flows/enrichment/ip-intelligence.md) — How country / city / coordinates and AS names get resolved.

225 +- [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md) — The provider chain that picks AS numbers.

226 +- [Classifiers](/docs/network-flows/enrichment/classifiers.md) — Rule-based labelling that runs only when static metadata didn't already classify the exporter or interface.

227 +- [Network sources](/docs/network-flows/enrichment/network-identity.md) — Fetching `networks`-style data from remote endpoints.

docs/network-flows/field-reference.md new

+339

@@ -0,0 +1,339 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/field-reference.md"

3 +sidebar_label: "Field Reference"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['fields', 'flow record', 'schema', 'reference']

7 +endmeta-->

8 +

9 +# Field Reference

10 +

11 +Each flow record carries up to 91 fields. Some come straight from the exporter, others are added by enrichment after decode. This page is the canonical list — what each field means, where it comes from, and which protocols populate it.

12 +

13 +In the dashboard, fields appear by their canonical name (uppercase, e.g., `SRC_AS_NAME`). The dashboard is case-insensitive when typing into the filter ribbon.

14 +

15 +## How to read the protocol columns

16 +

17 +| Symbol | Meaning |

18 +|---|---|

19 +| ✓ | Always populated by this protocol when the data is available |

20 +| ◐ | Populated only when the exporter includes the relevant Information Element in its template (v9 / IPFIX) or the relevant record type (sFlow) |

21 +| — | Never populated by this protocol; expect this field to be empty |

22 +

23 +Enrichment-only fields are marked **enrichment** — the decoder never fills them; they come from configured GeoIP databases, static metadata, classifiers, or routing sources.

24 +

25 +## Counters and sampling

26 +

27 +The four most-used fields. Read these first.

28 +

29 +| Field | Type | Description |

30 +|---|---|---|

31 +| `BYTES` | uint64 | Bytes in the flow, **already multiplied by `SAMPLING_RATE`** at ingest. The dashboard's volume numbers come from this. |

32 +| `PACKETS` | uint64 | Packets in the flow, already multiplied by `SAMPLING_RATE`. |

33 +| `RAW_BYTES` | uint64 | Bytes the exporter actually reported, before scaling. Use when sampling is uniform across all your exporters and you want exact counts. |

34 +| `RAW_PACKETS` | uint64 | Packets the exporter actually reported, before scaling. |

35 +| `FLOWS` | uint64 | Number of flows aggregated into this record. Always 1 for raw records. |

36 +| `SAMPLING_RATE` | uint64 | Packets-per-sample reported by the exporter. `1` means unsampled. Used as the multiplier for BYTES and PACKETS. |

37 +

38 +Every protocol populates these. sFlow always sends a sampling rate (per-sample). NetFlow v5 reads a header rate. NetFlow v7 has no rate field and is treated as unsampled. NetFlow v9 and IPFIX may include the rate per-record or via Sampling Options.

39 +

40 +## Identity — who and what

41 +

42 +| Field | Type | v5 | v7 | v9 | IPFIX | sFlow | Description |

43 +|---|---|---|---|---|---|---|---|

44 +| `FLOW_VERSION` | string | ✓ | ✓ | ✓ | ✓ | ✓ | One of `v5`, `v7`, `v9`, `ipfix`, `sflow`. |

45 +| `EXPORTER_IP` | IP | ✓ | ✓ | ✓ | ✓ | ✓ | The device that sent this flow. For sFlow, the agent address takes precedence over the UDP source IP. |

46 +| `EXPORTER_PORT` | uint16 | ✓ | ✓ | ✓ | ✓ | ✓ | Source UDP port of the exporter. |

47 +| `SRC_ADDR` | IP | ✓ | ✓ | ◐ | ◐ | ◐ | Source IP. v9/IPFIX from IE 8/27, sFlow from sampled header or `SampledIPv4`/`SampledIPv6`. |

48 +| `DST_ADDR` | IP | ✓ | ✓ | ◐ | ◐ | ◐ | Destination IP. |

49 +| `SRC_PORT` | uint16 | ✓ | ✓ | ◐ | ◐ | ◐ | Source L4 port. |

50 +| `DST_PORT` | uint16 | ✓ | ✓ | ◐ | ◐ | ◐ | Destination L4 port. |

51 +| `PROTOCOL` | uint8 | ✓ | ✓ | ✓ | ✓ | ◐ | IP protocol number. TCP=6, UDP=17, ICMP=1, ICMPv6=58, GRE=47, ESP=50. |

52 +| `ETYPE` | uint16 | ✓ (IPv4) | ✓ (IPv4) | ◐ | ◐ | ◐ | EtherType. 2048 = IPv4, 34525 = IPv6. |

53 +| `DIRECTION` | enum | — | — | ◐ | ◐ | — | `ingress`, `egress`, or `undefined`. |

54 +

55 +NetFlow v5 and v7 are IPv4-only. For v9, IPFIX, and sFlow, IPv6 fields populate when the exporter sends them.

56 +

57 +## Routing — addresses and AS

58 +

60 +|---|---|---|---|

73 +

74 +Static-network configuration can override `SRC_MASK` / `DST_MASK` and `SRC_AS` / `DST_AS` with more specific values from your CIDR-to-attribute map.

75 +

76 +## Interfaces

77 +

79 +|---|---|---|---|

94 +

95 +`*_BOUNDARY` is counter-intuitive: 1 means "external" (the Internet side). It's defined that way so that filtering for `IN_IF_BOUNDARY=1` cleanly gives you "traffic that came in from the Internet".

96 +

97 +## Layer 2

98 +

99 +| Field | Type | v5 | v7 | v9 | IPFIX | sFlow | Description |

100 +|---|---|---|---|---|---|---|---|

101 +| `SRC_MAC` | MAC | — | — | ◐ | ◐ | ◐ | Source MAC. v9 IE 56, IPFIX IE 56/81. sFlow from `SampledHeader` or `SampledEthernet`. |

102 +| `DST_MAC` | MAC | — | — | ◐ | ◐ | ◐ | Destination MAC. v9 IE 80, IPFIX IE 80/57. |

103 +| `SRC_VLAN` | uint16 | — | — | ◐ | ◐ | ◐ | Source VLAN. v9 IE 58, IPFIX IE 58/243. **For sFlow, only from `ExtendedSwitch` records — NOT from 802.1Q tags inside a sampled packet header.** |

104 +| `DST_VLAN` | uint16 | — | — | ◐ | ◐ | ◐ | Destination VLAN. |

105 +| `MPLS_LABELS` | string | — | — | ◐ | ◐ | ◐ | MPLS label stack as comma-separated decimal label values (label only, not EXP/S/TTL). |

106 +

107 +## NAT

108 +

110 +|---|---|---|---|---|---|---|

111 +| `SRC_ADDR_NAT` | IP | — | ◐ | ◐ | — | Post-NAT source address. v9 IE 225, IPFIX IE 225/281. |

112 +| `DST_ADDR_NAT` | IP | — | ◐ | ◐ | — | Post-NAT destination address. |

113 +| `SRC_PORT_NAT` | uint16 | — | ◐ | ◐ | — | Post-NAT source port. |

114 +| `DST_PORT_NAT` | uint16 | — | ◐ | ◐ | — | Post-NAT destination port. |

115 +

116 +## Protocol metadata

117 +

118 +| Field | Type | Description |

119 +|---|---|---|

120 +| `IPTTL` | uint8 | IP TTL. v9 uses Min/MaxTtl; IPFIX uses IE 192/52. |

121 +| `IPTOS` | uint8 | IP Type of Service / DSCP byte. |

122 +| `IPV6_FLOW_LABEL` | uint32 | IPv6 flow label (20-bit). v9/IPFIX only. |

123 +| `TCP_FLAGS` | uint8 | OR of all TCP control bits seen in the flow (SYN/ACK/FIN/RST/PSH/URG). |

124 +| `IP_FRAGMENT_ID` | uint32 | IPv4 ident or IPv6 fragment ID. |

125 +| `IP_FRAGMENT_OFFSET` | uint16 | Non-zero means fragmented. |

126 +| `ICMPV4_TYPE` | uint8 | ICMPv4 type. |

127 +| `ICMPV4_CODE` | uint8 | ICMPv4 code. |

128 +| `ICMPV6_TYPE` | uint8 | ICMPv6 type. |

129 +| `ICMPV6_CODE` | uint8 | ICMPv6 code. |

130 +| `FORWARDING_STATUS` | uint8 | RFC 7270 outcome code: `64..127` = forwarded, `128..191` = dropped, `192..255` = consumed. |

131 +

132 +## Timestamps

133 +

134 +| Field | Type | Description |

135 +|---|---|---|

136 +| `FLOW_START_USEC` | uint64 | Microseconds since epoch. From v5/v7 first-switched + sysUptime; from v9 first-switched normalised against system init time; from IPFIX `flowStartMicroseconds` family. Not populated for sFlow. |

137 +| `FLOW_END_USEC` | uint64 | Microseconds since epoch. Same sources. Not populated for sFlow. |

138 +| `OBSERVATION_TIME_MILLIS` | uint64 | IPFIX observation time (`observationTimeMilliseconds`). |

139 +

140 +## Geolocation (enrichment-only)

141 +

142 +| Field | Type | Description |

143 +|---|---|---|

144 +| `SRC_COUNTRY` | string | ISO 3166 country code. |

145 +| `DST_COUNTRY` | string | ISO 3166 country code. |

146 +| `SRC_GEO_STATE` | string | State / province. |

147 +| `DST_GEO_STATE` | string | State / province. |

148 +| `SRC_GEO_CITY` | string | City. |

149 +| `DST_GEO_CITY` | string | City. |

150 +| `SRC_GEO_LATITUDE` | string | Decimal latitude (string-encoded). Hidden in tables by default. |

151 +| `DST_GEO_LATITUDE` | string | Decimal latitude. |

152 +| `SRC_GEO_LONGITUDE` | string | Decimal longitude. |

153 +| `DST_GEO_LONGITUDE` | string | Decimal longitude. |

154 +

155 +City, latitude, and longitude are **not preserved in the rollup tiers** (1m, 5m, 1h). Aggregating on them forces the query to tier 0 (raw). Country and state survive into rollups.

156 +

157 +## Network labels (enrichment-only)

158 +

159 +These are the labels you assign to your own networks via static-metadata or network-sources configuration. The decoder never fills them.

160 +

161 +| Field | Type | Description |

162 +|---|---|---|

163 +| `SRC_NET_NAME` | string | Friendly name for the source network. |

164 +| `DST_NET_NAME` | string | Friendly name for the destination network. |

165 +| `SRC_NET_ROLE` | string | Role tag (e.g., `dmz`, `office`, `printing`, `iot`). |

166 +| `DST_NET_ROLE` | string | Role tag. |

167 +| `SRC_NET_SITE` | string | Physical site (e.g., `dc-fra1`). |

168 +| `DST_NET_SITE` | string | Physical site. |

169 +| `SRC_NET_REGION` | string | Region (e.g., `eu`, `us-east`). |

170 +| `DST_NET_REGION` | string | Region. |

171 +| `SRC_NET_TENANT` | string | Tenant (multi-tenant deployments). |

172 +| `DST_NET_TENANT` | string | Tenant. |

173 +

174 +## Exporter labels (enrichment-only)

175 +

176 +Labels you attach to your exporters via static-metadata or classifiers.

177 +

178 +| Field | Type | Description |

179 +|---|---|---|

180 +| `EXPORTER_NAME` | string | Friendly name. Falls back to an IP-derived string if no enrichment match. |

181 +| `EXPORTER_GROUP` | string | Group tag. |

182 +| `EXPORTER_ROLE` | string | Role tag (e.g., `edge`, `core`, `wan`). |

183 +| `EXPORTER_SITE` | string | Site tag. |

184 +| `EXPORTER_REGION` | string | Region tag. |

185 +| `EXPORTER_TENANT` | string | Tenant tag. |

186 +

187 +## Per-protocol availability summary

188 +

189 +For exporter-derived fields (not enrichment), the protocols differ. The shortest version:

190 +

191 +- **NetFlow v5**: IPv4 5-tuple, AS, interfaces, next-hop, IPTOS, TCP flags, bytes, packets, sampling rate (header), first/last switched timestamps. No IPv6, MAC, VLAN, NAT, ICMP, MPLS.

192 +- **NetFlow v7**: same as v5 minus the sampling rate.

193 +- **NetFlow v9**: depends on the template. Theoretically all the IEs Netdata maps (see [the IPFIX/v9 IE map](#what-ies-are-mapped) below). IPv6 supported.

194 +- **IPFIX**: superset of v9. Adds biflow (initiator/responder counters and `reverseInformationElement` IEs). Wider IE coverage. ICMP type and code as separate IEs.

195 +- **sFlow v5**: depends on which sFlow record types the agent emits. From `SampledHeader` you get most fields after parsing the truncated packet (Ethernet/IPv4/IPv6/TCP/UDP/ICMP/MPLS). VLANs come only from `ExtendedSwitch`. AS path and BGP communities come from `ExtendedGateway`. Counter samples are dropped.

196 +

197 +## What IEs are mapped

198 +

199 +For NetFlow v9 and IPFIX, only specific Information Elements end up in flow-record fields. The rest of the template is parsed (so the decoder can walk past them) but the values are dropped.

200 +

201 +The mapped IEs cover the standard set: identity (8/12/27/28, 7/11), counters (1/2/23/24/231/232/298/299), interfaces (10/14/252/253), protocol (4/5/6), ToS/DSCP (5/55), TTL (52/192), VLANs (58/59/243/254), MACs (56/80/57/81), NAT (225/226/281/282/227/228), AS (16/17), prefixes (44/45), masks (9/13/29/30), MPLS (70-79), ICMP (32/176-179, 139), fragmentation (54/88), IPv6 flow label (31), forwarding status (89), direction (61/239), sampling (34/50/305/306), timestamps (21/22/152/153/322 and the seconds/microseconds variants), and the data-link section for decapsulation (315).

202 +

203 +Vendor enterprise IEs are recognised only for one Juniper case (PEN 2636 `commonPropertiesId`) used to surface forwarding status. Cisco AVC, Cisco NEL/NSEL NAT events, and similar vendor-private fields are parsed (so the decoder doesn't fail) but their values are not exposed in flow records.

204 +

205 +If you need a specific IE mapped, open an issue with sample fixtures.

206 +

207 +## Filtering and aggregation hints

208 +

209 +Some fields are queryable but not aggregatable:

210 +

211 +- `BYTES`, `PACKETS`, `FLOWS`, `RAW_BYTES`, `RAW_PACKETS`, `SAMPLING_RATE` — these are sums in tables and sankeys; you cannot filter or group-by them.

212 +- `FLOW_START_USEC`, `FLOW_END_USEC`, `OBSERVATION_TIME_MILLIS` — timestamps, used by the time-range picker; not used as facets.

213 +- The four geo-coordinate fields (`SRC_GEO_LATITUDE/LONGITUDE`, `DST_GEO_LATITUDE/LONGITUDE`) are stored but hidden in the table by default and not exposed as facets.

214 +

215 +The dashboard also exposes two **virtual facets** that don't exist in the canonical schema:

216 +

217 +- `ICMPV4` — a synthesised string from `ICMPV4_TYPE` and `ICMPV4_CODE`, useful for filtering ICMPv4 messages by their named type/code combination (e.g., "echo-request").

218 +- `ICMPV6` — same for ICMPv6.

219 +

220 +Filtering on either of these virtual fields runs against the underlying `*_TYPE` and `*_CODE` fields.

221 +

222 +## A note on field counts

223 +

224 +You may see "89 fields" or "91 fields" in different parts of the codebase. The current canonical list has **91 entries**. The schema has grown over time and not every reference has caught up. The list above is exhaustive for the current release.

225 +

226 +## Master index — every field at a glance

227 +

228 +Use this table as the single reference when you know the field name and want every dimension in one place. Sorted alphabetically.

229 +

230 +Column legend:

231 +

232 +- **v5 / v7 / v9 / IPFIX / sFlow** — `✓` always populated, `◐` only when the exporter sends the relevant IE/record, `—` never.

233 +- **Source** — `decoder` (filled by parsing the protocol), `enrichment` (filled by post-decode lookups; the wire never carries it), or `both` (decoder may fill, enrichment may overlay/override).

234 +- **Tiers** — which tiers preserve the field. `all` means raw + 1m + 5m + 1h. `raw` means raw only (dropped at rollup).

235 +- **Selectivity** — which query roles the field plays. `facet` (autocomplete + filter ribbon), `group-by` (Sankey/timeseries/maps aggregation), `filter` (selections), `metric` (BYTES/PACKETS/FLOWS — sums in tables, not faceted), `time` (used by the time-range picker), `hidden` (queryable but not in the default columns).

236 +- **Notes** — IE numbers / sFlow record types when relevant, plus the enrichment chain for enrichment-derived fields.

237 +

239 +|---|---|---|---|---|---|---|---|---|---|---|

240 +| `BYTES` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric, filter | Counter; scaled by `SAMPLING_RATE` at ingest. sFlow derives from decoded L3 length |

241 +| `DIRECTION` | string | — | — | ◐ | ◐ | — | decoder | all | facet, group-by, filter | v9 IE 61, IPFIX IE 61/239. sFlow has no native direction |

242 +| `DST_ADDR` | IP | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 12/28; sFlow `SampledHeader`/`SampledIPv4`/`SampledIPv6`. Raw-only |

243 +| `DST_ADDR_NAT` | IP | — | — | ◐ | ◐ | — | decoder | raw | facet, group-by, filter | v9 IE 226/282; IPFIX `postNATdestinationIPv4/IPv6Address` |

244 +| `DST_AS` | uint32 | ✓ | ✓ | ◐ | ◐ | ◐ | both | all | facet, group-by, filter | decoder IE 17 / sFlow `ExtendedGateway` last AS in path. Enrichment chain: `asn_providers` (default `[flow, routing, geoip]`); per-CIDR `enrichment.networks.<cidr>.asn` overrides |

245 +| `DST_AS_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `format_as_name(DST_AS, attrs.asn_name)` → `AS{n} {name}`; falls back to `AS0 Unknown ASN` or `AS0 Private IP Address Space` |

246 +| `DST_AS_PATH` | string | — | — | — | — | ◐ | both | raw | filter | sFlow `ExtendedGateway` BGP path. Routing enrichment overlay (BMP / BioRIS) for non-sFlow exporters |

247 +| `DST_COMMUNITIES` | string | — | — | — | — | ◐ | both | raw | filter | sFlow `ExtendedGateway` communities. Routing enrichment overlay (BMP / BioRIS) |

248 +| `DST_COUNTRY` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | GeoIP MMDB on `DST_ADDR` → optional override from `enrichment.networks.<cidr>.country` |

249 +| `DST_GEO_CITY` | string | — | — | — | — | — | enrichment | raw | facet, group-by, filter | GeoIP city MMDB. Raw-only (dropped at rollup) |

250 +| `DST_GEO_LATITUDE` | string | — | — | — | — | — | enrichment | raw | filter, hidden | GeoIP coordinates. Raw-only; hidden in default table view |

251 +| `DST_GEO_LONGITUDE` | string | — | — | — | — | — | enrichment | raw | filter, hidden | GeoIP coordinates. Raw-only; hidden in default table view |

252 +| `DST_GEO_STATE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | GeoIP subdivision. Preserved in rollups |

253 +| `DST_LARGE_COMMUNITIES` | string | — | — | — | — | — | enrichment | raw | filter | RFC 8092 large communities from routing enrichment (BMP / BioRIS) |

254 +| `DST_MAC` | MAC | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9 IE 80/57; IPFIX same. sFlow from `SampledHeader` datalink or `SampledEthernet` |

255 +| `DST_MASK` | uint8 | ✓ | ✓ | ◐ | ◐ | ◐ | both | raw | facet, group-by, filter | v9 IE 13/29; sFlow `ExtendedRouter`. Enrichment overlay via `net_providers` (default `[flow, routing]`) plus per-CIDR overrides |

256 +| `DST_NET_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.name` (static) merged with network sources by ascending prefix length |

257 +| `DST_NET_REGION` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.region` from static + network sources |

258 +| `DST_NET_ROLE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.role` from static + network sources |

259 +| `DST_NET_SITE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.site` from static + network sources |

260 +| `DST_NET_TENANT` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.tenant` from static + network sources |

261 +| `DST_PORT` | uint16 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 11. sFlow from `SampledIPv4`/`SampledIPv6` or `SampledHeader` transport parse. Raw-only |

262 +| `DST_PORT_NAT` | uint16 | — | — | ◐ | ◐ | — | decoder | raw | facet, group-by, filter | v9 IE 228; IPFIX `postNAPTdestinationTransportPort` |

263 +| `DST_PREFIX` | IP | ✓ | ✓ | ◐ | — | — | decoder | raw | filter | v5/v7 derived from `DST_ADDR` & `DST_MASK`. v9 IE 45 (`Ipv4DstPrefix`). IPFIX has no canonical mapping; sFlow none |

264 +| `DST_VLAN` | uint16 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 59; IPFIX IE 254 (`PostVlanId`/`PostDot1qVlanId`). sFlow only via `ExtendedSwitch` (NOT from 802.1Q tag in `SampledHeader`) |

265 +| `ETYPE` | uint16 | ✓ (IPv4) | ✓ (IPv4) | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v5/v7 hardcoded to 2048. v9/IPFIX IE 60 `IpProtocolVersion` (4→2048, 6→34525). sFlow from sampled L2 etype |

266 +| `EXPORTER_GROUP` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.group`. Classifiers fill it when static metadata didn't |

267 +| `EXPORTER_IP` | IP | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | facet, group-by, filter | UDP source IP for NetFlow. sFlow uses datagram `agent_address` (override) |

268 +| `EXPORTER_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.name` → falls back to IP-derived name |

269 +| `EXPORTER_PORT` | uint16 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | facet, group-by, filter | UDP source port from socket |

270 +| `EXPORTER_REGION` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.region`. Classifiers may fill |

271 +| `EXPORTER_ROLE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.role`. Classifiers may fill |

272 +| `EXPORTER_SITE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.site`. Classifiers may fill |

273 +| `EXPORTER_TENANT` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.tenant`. Classifiers may fill |

274 +| `FLOWS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric, filter | Always 1 for raw records; sums during rollup aggregation |

275 +| `FLOW_END_USEC` | uint64 | ✓ | ✓ | ◐ | ◐ | — | decoder | raw | time | v5/v7 from header `sysUpTime` + `LastSwitched`. v9 from `LastSwitched`/`flowEndMilliseconds` normalised against `system_init`. IPFIX from `flowEndMilliseconds` family. Not populated for sFlow |

276 +| `FLOW_START_USEC` | uint64 | ✓ | ✓ | ◐ | ◐ | — | decoder | raw | time | Same sources as `FLOW_END_USEC`. Not populated for sFlow |

277 +| `FLOW_VERSION` | string | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | facet, group-by, filter | One of `v5`, `v7`, `v9`, `ipfix`, `sflow` |

278 +| `FORWARDING_STATUS` | uint8 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9/IPFIX IE 89; IPFIX also from Juniper PEN 2636 `commonPropertiesId`. sFlow synthesises `128` (dropped) when `output_format` is `discarded` |

279 +| `ICMPV4_CODE` | uint8 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | IPFIX IE 177 `IcmpCodeIpv4` + IE 32 low byte. v9 IE 178 `IcmpCodeValue` + IE 32. sFlow from decoded ICMP header |

280 +| `ICMPV4_TYPE` | uint8 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | IPFIX IE 176 `IcmpTypeIpv4` + IE 32 high byte. v9 IE 32 `IcmpType` + IE 177 `IcmpTypeValue`. sFlow from decoded ICMP header |

281 +| `ICMPV6_CODE` | uint8 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | IPFIX IE 179 `IcmpCodeIpv6` + IE 139 low byte. v9 IE 179 `ImpIpv6CodeValue`. sFlow from decoded ICMPv6 header |

282 +| `ICMPV6_TYPE` | uint8 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | IPFIX IE 178 `IcmpTypeIpv6` + IE 139 high byte. v9 IE 178 `IcmpIpv6TypeValue`. sFlow from decoded ICMPv6 header |

283 +| `IN_IF` | uint32 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 10 `InputSnmp`; IPFIX IE 10/252. sFlow flow-sample `input` (single index only; LOCAL→0) |

284 +| `IN_IF_BOUNDARY` | uint8 | — | — | — | — | — | enrichment | all | facet, group-by, filter | Per-interface static metadata or interface classifier output. `1`=external, `2`=internal |

285 +| `IN_IF_CONNECTIVITY` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | Per-interface static metadata or interface classifier (e.g., `transit`, `peering`, `customer`) |

286 +| `IN_IF_DESCRIPTION` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.if_indexes.<idx>.description` or set via classifier `SetDescription()` |

287 +| `IN_IF_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.if_indexes.<idx>.name` or set via classifier `SetName()` |

288 +| `IN_IF_PROVIDER` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | Static metadata or interface classifier provider tag |

289 +| `IN_IF_SPEED` | uint64 | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.if_indexes.<idx>.speed` (bps) |

290 +| `IPTOS` | uint8 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 5 `SrcTos` / IE 55 `DstTos`. IPFIX IE 5/55. sFlow from `SampledIPv4` tos / `SampledIPv6` priority / parsed L3 |

291 +| `IPTTL` | uint8 | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9 IE 52/192 (`Min/MaxTtl`). IPFIX same. sFlow from parsed L3 header |

292 +| `IPV6_FLOW_LABEL` | uint32 | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 31 `FlowLabelIpv6`. sFlow from parsed IPv6 header |

293 +| `IP_FRAGMENT_ID` | uint32 | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9 IE 54 `Ipv4Ident`. IPFIX IE 54 `FragmentIdentification`. sFlow from parsed IPv4 header |

294 +| `IP_FRAGMENT_OFFSET` | uint16 | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 88 `FragmentOffset`. sFlow from parsed IPv4 header |

295 +| `MPLS_LABELS` | string | — | — | ◐ | ◐ | ◐ | decoder | raw | filter | v9 IE 70-79 `MplsLabel1..10`. IPFIX IE 70 `MplsTopLabelStackSection` + 71-79 `MplsLabelStackSection2..10`. sFlow from MPLS in `SampledHeader`. Comma-separated decimal labels |

296 +| `NEXT_HOP` | IP | ✓ | ✓ | ◐ | ◐ | ◐ | both | all | facet, group-by, filter | v9 IE 15/18/62/63; IPFIX same. sFlow `ExtendedRouter`/`ExtendedGateway`. Enrichment overlay via `net_providers` chain (default `[flow, routing]`) |

297 +| `OBSERVATION_TIME_MILLIS` | uint64 | — | — | ◐ | — | — | decoder | raw | time | v9 IE 323 `ObservationTimeMilliseconds`. IPFIX has no canonical mapping in this build |

298 +| `OUT_IF` | uint32 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 14 `OutputSnmp`; IPFIX IE 14/253. sFlow flow-sample `output` (single index only; LOCAL→0) |

299 +| `OUT_IF_BOUNDARY` | uint8 | — | — | — | — | — | enrichment | all | facet, group-by, filter | Same semantics as `IN_IF_BOUNDARY` |

300 +| `OUT_IF_CONNECTIVITY` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | Static metadata or interface classifier connectivity tag |

301 +| `OUT_IF_DESCRIPTION` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.if_indexes.<idx>.description` |

302 +| `OUT_IF_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.if_indexes.<idx>.name` |

303 +| `OUT_IF_PROVIDER` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | Static metadata or interface classifier provider tag |

304 +| `OUT_IF_SPEED` | uint64 | — | — | — | — | — | enrichment | all | facet, group-by, filter | `metadata_static.exporters.<ip>.if_indexes.<idx>.speed` (bps) |

305 +| `PACKETS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | all | metric, filter | Counter; scaled by `SAMPLING_RATE` at ingest. sFlow always 1 per sample |

306 +| `PROTOCOL` | uint8 | ✓ | ✓ | ✓ | ✓ | ◐ | decoder | all | facet, group-by, filter | v5/v7 protocol_number; v9 IE 4; IPFIX IE 4 `ProtocolIdentifier`. sFlow from `SampledIPv4`/`SampledIPv6` or parsed L3 |

307 +| `RAW_BYTES` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | raw | metric | Pre-sampling byte count from the exporter |

308 +| `RAW_PACKETS` | uint64 | ✓ | ✓ | ✓ | ✓ | ✓ | decoder | raw | metric | Pre-sampling packet count from the exporter |

309 +| `SAMPLING_RATE` | uint64 | ✓ (header) | — | ◐ | ◐ | ✓ | decoder | raw | metric | v5 from header `sampling_interval`. v7 has no rate (treated as unsampled). v9/IPFIX from IE 34/305/306 or Sampling Options template. sFlow per-sample rate |

310 +| `SRC_ADDR` | IP | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 8/27. sFlow `SampledHeader`/`SampledIPv4`/`SampledIPv6`. Raw-only |

311 +| `SRC_ADDR_NAT` | IP | — | — | ◐ | ◐ | — | decoder | raw | facet, group-by, filter | v9 IE 225/281; IPFIX `postNATsourceIPv4/IPv6Address` |

312 +| `SRC_AS` | uint32 | ✓ | ✓ | ◐ | ◐ | ◐ | both | all | facet, group-by, filter | decoder IE 16 / sFlow `ExtendedGateway` `src_as`. Enrichment chain: `asn_providers` (default `[flow, routing, geoip]`); per-CIDR `enrichment.networks.<cidr>.asn` overrides |

313 +| `SRC_AS_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `format_as_name(SRC_AS, attrs.asn_name)` → `AS{n} {name}`; falls back to `AS0 Unknown ASN` or `AS0 Private IP Address Space` |

314 +| `SRC_COUNTRY` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | GeoIP MMDB on `SRC_ADDR` → optional override from `enrichment.networks.<cidr>.country` |

315 +| `SRC_GEO_CITY` | string | — | — | — | — | — | enrichment | raw | facet, group-by, filter | GeoIP city MMDB. Raw-only |

316 +| `SRC_GEO_LATITUDE` | string | — | — | — | — | — | enrichment | raw | filter, hidden | GeoIP coordinates. Raw-only; hidden in default table view |

317 +| `SRC_GEO_LONGITUDE` | string | — | — | — | — | — | enrichment | raw | filter, hidden | GeoIP coordinates. Raw-only; hidden in default table view |

318 +| `SRC_GEO_STATE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | GeoIP subdivision. Preserved in rollups |

319 +| `SRC_MAC` | MAC | — | — | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9 IE 56/81; IPFIX same. sFlow from `SampledHeader` datalink or `SampledEthernet` |

320 +| `SRC_MASK` | uint8 | ✓ | ✓ | ◐ | ◐ | ◐ | both | raw | facet, group-by, filter | v9 IE 9/29; sFlow `ExtendedRouter`. Enrichment overlay via `net_providers` (default `[flow, routing]`) plus per-CIDR overrides |

321 +| `SRC_NET_NAME` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.name` (static) merged with network sources by ascending prefix length |

322 +| `SRC_NET_REGION` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.region` from static + network sources |

323 +| `SRC_NET_ROLE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.role` from static + network sources |

324 +| `SRC_NET_SITE` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.site` from static + network sources |

325 +| `SRC_NET_TENANT` | string | — | — | — | — | — | enrichment | all | facet, group-by, filter | `enrichment.networks.<cidr>.tenant` from static + network sources |

326 +| `SRC_PORT` | uint16 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | raw | facet, group-by, filter | v9/IPFIX IE 7. sFlow from `SampledIPv4`/`SampledIPv6` or transport parse. Raw-only |

327 +| `SRC_PORT_NAT` | uint16 | — | — | ◐ | ◐ | — | decoder | raw | facet, group-by, filter | v9 IE 227; IPFIX `postNAPTsourceTransportPort` |

328 +| `SRC_PREFIX` | IP | ✓ | ✓ | ◐ | — | — | decoder | raw | filter | v5/v7 derived from `SRC_ADDR` & `SRC_MASK`. v9 IE 44 (`Ipv4SrcPrefix`). IPFIX has no canonical mapping; sFlow none |

329 +| `SRC_VLAN` | uint16 | — | — | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | v9 IE 58; IPFIX IE 58/243 (`VlanId`/`Dot1qVlanId`). sFlow only via `ExtendedSwitch` (NOT from 802.1Q tag in `SampledHeader`) |

330 +| `TCP_FLAGS` | uint8 | ✓ | ✓ | ◐ | ◐ | ◐ | decoder | all | facet, group-by, filter | OR of all TCP control bits seen in the flow. v9/IPFIX IE 6. sFlow from parsed TCP header in `SampledHeader` |

331 +

332 +The two virtual facets (`ICMPV4`, `ICMPV6`) aren't in this table because they don't exist in the canonical schema — they are synthesised string facets that filter on `ICMPV4_TYPE`/`ICMPV4_CODE` (or v6) under the hood. See the previous section.

333 +

334 +## What's next

335 +

336 +- [Configuration](/docs/network-flows/configuration.md) — `netflow.yaml` reference.

337 +- [Retention and Querying](/docs/network-flows/retention-querying.md) — How the four tiers store data and which fields they preserve.

338 +- [Visualisation](/docs/network-flows/visualization/summary-sankey.md) — Reading the dashboard.

339 +- [Validation and Data Quality](/docs/network-flows/validation.md) — How to know your data is right.

docs/network-flows/installation.md new

+165

@@ -0,0 +1,165 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/installation.md"

3 +sidebar_label: "Installation"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['installation', 'package', 'netdata-plugin-netflow', 'setup']

7 +endmeta-->

8 +

9 +# Installation

10 +

11 +The netflow plugin is **packaged separately from the main Netdata Agent**. You install it on the same host where Netdata runs, after Netdata itself is in place.

12 +

13 +The package name is **`netdata-plugin-netflow`** on both Debian and RPM distributions. It is not installed by the standard `netdata` package or by the netdata-updater on its own — you have to install it explicitly on native-package systems.

14 +

15 +The static install (the kickstart `--static-only` path) bundles the plugin automatically. If you used the kickstart installer with the static option, no extra step is needed.

16 +

17 +## Prerequisites

18 +

19 +- A working Netdata Agent on the host that will receive flow data.

20 +- That host must be reachable on UDP from your routers and switches (default port `2055`).

21 +- Linux. The plugin is Linux-only.

22 +

23 +## Install on Debian / Ubuntu / Mint

24 +

25 +```bash

26 +sudo apt update

27 +sudo apt install netdata-plugin-netflow

28 +sudo systemctl restart netdata

29 +```

30 +

31 +## Install on RHEL / Fedora / CentOS / Rocky / Alma

32 +

33 +```bash

34 +sudo dnf install netdata-plugin-netflow

35 +sudo systemctl restart netdata

36 +```

37 +

38 +(`yum install` works on older systems where `dnf` isn't present.)

39 +

40 +## Install on openSUSE

41 +

42 +```bash

43 +sudo zypper install netdata-plugin-netflow

44 +sudo systemctl restart netdata

45 +```

46 +

47 +## Static install (kickstart)

48 +

49 +If you installed Netdata using:

50 +

51 +```bash

52 +wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh && \

53 + sh /tmp/netdata-kickstart.sh --static-only

54 +```

55 +

56 +…the netflow plugin is already installed under `/opt/netdata/usr/libexec/netdata/plugins.d/netflow-plugin`. No extra step.

57 +

58 +To verify:

59 +

60 +```bash

61 +ls /opt/netdata/usr/libexec/netdata/plugins.d/netflow-plugin

62 +```

63 +

64 +## Source build

65 +

66 +Building from source requires a Rust toolchain (rustc + cargo, version 1.83 or later). When CMake detects Rust, the plugin is built and installed alongside the rest of Netdata.

67 +

68 +```bash

69 +git clone https://github.com/netdata/netdata.git

70 +cd netdata

71 +sudo ./netdata-installer.sh

72 +```

73 +

74 +**Caveat:** source builds do **not** include the stock GeoIP / IP-intelligence database files. The plugin starts fine without them, but country, city, and AS-name fields will be empty until you run the downloader once:

75 +

76 +```bash

77 +sudo /usr/sbin/topology-ip-intel-downloader

78 +```

79 +

80 +This populates `/var/cache/netdata/topology-ip-intel/` with the DB-IP-based MMDB files. The plugin auto-detects the cache copy on its next 30-second poll. See [GeoIP enrichment](/docs/network-flows/enrichment/ip-intelligence.md) for details and refresh scheduling.

81 +

82 +## What gets installed

83 +

84 +| Path | Purpose |

85 +|---|---|

86 +| `/usr/libexec/netdata/plugins.d/netflow-plugin` | The plugin binary (mode 0750, root:netdata) |

87 +| `/usr/sbin/topology-ip-intel-downloader` | Helper for refreshing the GeoIP / IP-intel MMDBs |

88 +| `/usr/lib/netdata/conf.d/netflow.yaml` | Stock configuration (read-only reference; copy to `/etc/netdata/netflow.yaml` to customise) |

89 +| `/usr/lib/netdata/conf.d/topology-ip-intel.yaml` | IP-intel downloader configuration |

90 +| `/usr/share/netdata/topology-ip-intel/topology-ip-asn.mmdb` | Stock ASN database (DB-IP) |

91 +| `/usr/share/netdata/topology-ip-intel/topology-ip-geo.mmdb` | Stock geographic database (DB-IP) |

92 +

93 +(Paths assume native packages. Static installs put everything under `/opt/netdata/`.)

94 +

95 +## Verify the plugin is running

96 +

97 +After installation and restart:

98 +

99 +```bash

100 +sudo journalctl -u netdata --since "5 minutes ago" | grep -E 'netflow|listener'

101 +```

102 +

103 +You should see entries indicating that the plugin loaded its config and that the UDP listener bound to its port.

104 +

105 +Quick sanity check:

106 +

107 +```bash

108 +sudo ss -unlp | grep 2055

109 +```

110 +

111 +A line for `netflow-plugin` confirms the listener is up.

112 +

113 +## Open Netdata to confirm

114 +

115 +Open the Netdata UI in your browser. The **Network Flows** tab should appear in the top navigation. The plugin's operational charts also appear under the standard charts page in the `netflow` family.

116 +

117 +If the tab doesn't appear, or appears empty:

118 +

119 +- Check that the plugin process is running: `pgrep -fa netflow-plugin`.

120 +- Check Netdata Cloud SSO: the Network Flows function requires authenticated access to the agent's space.

121 +- See [Troubleshooting](/docs/network-flows/troubleshooting.md).

122 +

123 +## Configuring flow sources

124 +

125 +Installing the plugin enables it. To actually see flow data, you need to configure a router, switch, or software exporter to send NetFlow / IPFIX / sFlow datagrams to this host's UDP port 2055.

126 +

127 +That's the next step:

128 +

129 +- [Quick Start](/docs/network-flows/quick-start.md) — A 15-minute path to your first flow data.

130 +- [Sources / NetFlow](/src/crates/netflow-plugin/integrations/netflow.md) — Vendor configurations for NetFlow.

131 +- [Sources / IPFIX](/src/crates/netflow-plugin/integrations/ipfix.md) — Vendor configurations for IPFIX.

132 +- [Sources / sFlow](/src/crates/netflow-plugin/integrations/sflow.md) — Vendor configurations for sFlow.

133 +

134 +## Uninstall

135 +

136 +```bash

137 +# Debian / Ubuntu

138 +sudo apt remove netdata-plugin-netflow

139 +

140 +# RHEL / Fedora / CentOS / Rocky / Alma

141 +sudo dnf remove netdata-plugin-netflow

142 +

143 +# openSUSE

144 +sudo zypper remove netdata-plugin-netflow

145 +```

146 +

147 +Remove the configuration if you also want to clean up:

148 +

149 +```bash

150 +sudo rm /etc/netdata/netflow.yaml /etc/netdata/topology-ip-intel.yaml

151 +```

152 +

153 +The flow journals at `/var/cache/netdata/flows/` and `/var/cache/netdata/topology-ip-intel/` are not removed by the package manager. Delete them manually if you want to reclaim the disk:

154 +

155 +```bash

156 +sudo rm -rf /var/cache/netdata/flows /var/cache/netdata/topology-ip-intel

157 +```

158 +

159 +(Warning: this deletes all your historical flow data.)

160 +

161 +## What's next

162 +

163 +- [Quick Start](/docs/network-flows/quick-start.md) — Configure your first source and see traffic in the dashboard.

164 +- [Configuration](/docs/network-flows/configuration.md) — Tune the listener, retention, and enrichment.

165 +- [Troubleshooting](/docs/network-flows/troubleshooting.md) — When something doesn't work.

docs/network-flows/investigation-playbooks.md new

+188

@@ -0,0 +1,188 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/investigation-playbooks.md"

3 +sidebar_label: "Investigation Playbooks"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['playbooks', 'investigation', 'workflows', 'troubleshooting traffic']

7 +endmeta-->

8 +

9 +# Investigation playbooks

10 +

11 +Step-by-step recipes for common questions, all using the Netdata Network Flows tab. Each playbook fits in a 5-15 minute investigation window.

12 +

13 +## Playbook 1 — "The link is saturated, who's responsible?"

14 +

15 +**The situation.** SNMP shows your Internet link at 95% utilisation. Users complain about slowness.

16 +

17 +**The goal.** Identify the talker(s) consuming the bandwidth.

18 +

19 +**Steps.**

20 +

21 +1. **Open the Network Flows tab** with the default view (Sankey + Table). Set the time range to **the last 15 minutes** — recent enough to be live, wide enough to smooth bursts.

22 +

23 +2. **Filter to the saturated interface.** In the filter ribbon, set:

24 + - `Exporter Name` = the router with the saturated link

25 + - `Output Interface Name` = the interface name (or `Input Interface Name` if you want incoming traffic)

26 +

27 + This eliminates the doubling effect and shows only one direction.

28 +

29 +3. **Change the aggregation to "who's responsible".** Click the group-by selector and change the fields to:

30 + - `Source ASN` → `Destination ASN` (for an Internet-edge link)

31 +

32 + Or for an internal link:

33 + - `Source IP` → `Destination IP`

34 +

35 +4. **Read the Sankey.** The widest band is your top talker pair. Click on the wide band to drill in.

36 +

37 +5. **If a single ASN/IP dominates** — that's your answer. Click the value to add it as a filter and look at the table for the specific 5-tuple details.

38 +

39 +6. **If traffic is evenly distributed** across many sources — the problem is aggregate demand, not a single offender. The link genuinely needs more capacity, or you need traffic shaping. Move to Playbook 3.

40 +

41 +**What to record.**

42 +

43 +- Timestamp range of the investigation

44 +- Top 3 talker pairs and their byte volumes

45 +- Whether this is a one-time spike or sustained

46 +- The URL of the dashboard view (preserves all filters and aggregation)

47 +

48 +**Common findings.**

49 +

50 +- Backup software running during business hours.

51 +- A SaaS sync or cloud upload.

52 +- Misconfigured automation (e.g., logs being shipped to the wrong place).

53 +- New user / new application doing something unexpected.

54 +

55 +## Playbook 2 — "Investigating a specific IP"

56 +

57 +**The situation.** A security alert references an IP address. You need to know what it talked to, when, and how much.

58 +

59 +**The goal.** Construct a timeline and traffic profile for that IP.

60 +

61 +**Steps.**

62 +

63 +1. **Open the Network Flows tab** with **the last 24 hours** as the time range.

64 +

65 +2. **Filter by the IP.** In the filter ribbon:

66 + - For inbound investigation: `Destination IP` = the IP

67 + - For outbound investigation: `Source IP` = the IP

68 + - For both directions: filter both, separately, in two browser tabs

69 +

70 + IP filtering forces tier 0 (raw retention) — the time depth is bounded by your raw-tier retention. If you need to look further back than that, the data isn't there.

71 +

72 +3. **Switch to Time-Series view.** This shows when the IP was active. Look for:

73 + - When did activity start? End?

74 + - Is it constant, periodic, or bursty?

75 + - Does it correlate with a known event (deployment, business hours, maintenance window)?

76 +

77 +4. **Switch back to Sankey + Table.** Change the group-by to surface the relevant context:

78 + - `Destination IP` → `Destination Port` → `Destination Country` (if the suspect is a source)

79 + - `Source IP` → `Source ASN` (if the suspect is a destination)

80 +

81 +5. **Read the table.** The top rows show the IP's most-talked-to peers, ranked by bytes. Look for:

82 + - Unknown external IPs in unexpected geographies.

83 + - Connections on unusual ports (anything not in your normal protocol mix).

84 + - Sustained outbound transfers (potential exfiltration) vs short bursts (likely normal).

85 +

86 +6. **For each suspicious peer, drill in.** Add the peer IP to the filter ribbon, switch back to Time-Series. Confirm the timeline aligns with the original alert.

87 +

88 +**What to record.**

89 +

90 +- Time range of all activity by the IP

91 +- Top destinations and their byte/packet counts

92 +- Whether the activity is consistent with a legitimate use (backup, SaaS sync) or anomalous

93 +- The URL of each dashboard view used in the investigation

94 +

95 +**Caveats.**

96 +

97 +- IP filter forces tier 0; older data may not be available.

98 +- If the IP is internal and you haven't declared it under `enrichment.networks`, GeoIP may misrepresent its country.

99 +- If the IP is a NAT public address, multiple internal hosts may be hidden behind it. Cross-check with NAT translation logs.

100 +

101 +## Playbook 3 — "Justifying a link upgrade"

102 +

103 +**The situation.** A WAN circuit is at 80% utilisation during peak hours. Finance wants justification before approving an upgrade.

104 +

105 +**The goal.** Produce a defensible trend showing growth and projecting the date of saturation.

106 +

107 +**Steps.**

108 +

109 +1. **Open the Network Flows tab** with **the last 30 days** as the time range. (Adjust based on tier-1/5/60 retention. If your retention is shorter, use whatever you have.)

110 +

111 +2. **Filter to the WAN interface.** Set `Exporter Name` and `Output Interface Name` (or input — pick one direction). This removes the doubling effect.

112 +

113 +3. **Switch to Time-Series view.** The chart now shows ~30 days of bandwidth on the link. The bucket size auto-adjusts to roughly 1 hour at this range.

114 +

115 +4. **Identify the trend.** Look at the daily peaks (one curve cycle = one day). The peak should be growing month-over-month. Eyeball the slope.

116 +

117 +5. **Identify the growth driver.** Switch back to Sankey + Table, group by `Destination ASN` or `Application` (port). Compare top consumers from the start of the period to the end. New entries that weren't there 30 days ago are growth drivers.

118 +

119 +6. **Compute the upgrade need.** Take the current peak (e.g., 80% of 100 Mbps = 80 Mbps), project forward at the observed monthly growth rate (e.g., 10%/month = ~30%/quarter), and find when it crosses 100% (or 70% if you want headroom).

120 +

121 + Example: if peak grows from 70 Mbps to 80 Mbps over 30 days, that's roughly 14% monthly growth. At that rate it crosses 100 Mbps in ~2 months and 200 Mbps would buy you ~1 year.

122 +

123 +**What to record.**

124 +

125 +- Trend chart (screenshot or shareable URL)

126 +- Growth driver: the specific applications / services consuming the new bandwidth

127 +- Projected saturation date and recommended upgrade timeline

128 +- Sampling rate of the exporter (so the numbers can be interpreted)

129 +

130 +**Caveats.**

131 +

132 +- Always note the sampling rate. A change in sampling rate during the analysis window invalidates the trend.

133 +- A large spike one day shouldn't drive the projection. Use weekly peaks (averaged across same-day-of-week) for stability.

134 +- If your retention is shorter than 30 days, use what you have but caveat the projection.

135 +

136 +## Playbook 4 — "Scoping a security alert"

137 +

138 +**The situation.** Your IDS / EDR / SIEM fired an alert: an internal host communicated with a known-malicious external IP. You have the internal IP, the external IP, and a rough time window.

139 +

140 +**The goal.** Determine the scope and timeline. Did other internal hosts talk to the same external IP? When did it start? How much data was exchanged?

141 +

142 +**Steps.**

143 +

144 +1. **Open the Network Flows tab** with the time range covering 24 hours before the alert through now.

145 +

146 +2. **Filter by the external IP.** In the filter ribbon: `Destination IP` = the external IP.

147 +

148 + This forces tier 0. Time depth is your raw-tier retention.

149 +

150 +3. **Switch to Time-Series view.** When did communication start? Is it ongoing? Did it correlate with the alert time?

151 +

152 +4. **Switch to Sankey + Table.** Group by `Source IP`. The result is "every internal IP that talked to this external IP, ranked by bytes".

153 +

154 + - If only one internal host appears, scope is contained.

155 + - If multiple appear, you have a broader scope. Investigate each.

156 +

157 +5. **For the alerted internal host**, swap the filter: `Source IP` = the internal host (remove the external filter). Group by `Destination IP` → `Destination Country` → `Destination ASN`. Look for other suspicious peers.

158 +

159 +6. **Reverse-direction check.** Switch the filter to the external IP as `Source IP` (now you're looking at incoming traffic from it). Internal hosts that received connections from the external IP show up — useful for inbound C2 / probe analysis.

160 +

161 +7. **Geographic check.** Switch to Country Map. The location of the external IP gives a quick "where" — useful to compare against what your threat intelligence said.

162 +

163 +**What to record.**

164 +

165 +- All internal IPs that communicated with the external IP, time ranges, byte counts

166 +- Other suspicious destinations the alerted host talked to in the same window

167 +- Whether traffic is ongoing or stopped

168 +- The dashboard URL of each view (for the incident report)

169 +

170 +**Caveats.**

171 +

172 +- Sampled flows can miss small connections. Beaconing at low rates may not be visible at 1-in-1000 sampling. If you sample, your security investigation has a floor.

173 +- An external IP behind a CDN may be one of many destinations served by that infrastructure. ASN-level analysis (`Destination ASN`) is often more informative than IP.

174 +- The malicious IP being public doesn't mean the internal host was compromised — false positives in threat intel are common. Cross-check with the host's logs.

175 +

176 +## A note on the dashboard

177 +

178 +All the playbooks above use the same controls: time range, filters, group-by fields, view switcher. Once you're comfortable with these four, every investigation becomes a permutation. Mastering the tool means knowing which permutation fits the question.

179 +

180 +The URL preserves all your selections — copy it and paste into your incident-management ticket so anyone reviewing has the exact same view you saw.

181 +

182 +## What's next

183 +

184 +- [Sankey and Table](/docs/network-flows/visualization/summary-sankey.md) — Full reference for the default view.

185 +- [Filters and Facets](/docs/network-flows/visualization/filters-facets.md) — How to narrow effectively.

186 +- [Time-Series](/docs/network-flows/visualization/time-series.md) — Trends over the time range.

187 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — What "wrong" looks like and why.

188 +- [Validation and Data Quality](/docs/network-flows/validation.md) — Confirming your numbers before acting on them.

docs/network-flows/quick-start.md new

+177

@@ -0,0 +1,177 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/quick-start.md"

3 +sidebar_label: "Quick Start"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['quick start', 'netflow', 'sflow', 'ipfix', 'getting started', 'setup']

7 +endmeta-->

8 +

9 +# Quick Start

10 +

11 +Get flow monitoring running in 15 minutes. The path: install the plugin, configure your first router, open the dashboard, and read it correctly.

12 +

13 +## Before you start

14 +

15 +- The Netdata Agent is running on the host that will collect flow data.

16 +- The [netflow plugin is installed](/docs/network-flows/installation.md) on that host.

17 +- You can configure flow export on at least one router or switch.

18 +- The router can reach the agent's IP on UDP port 2055.

19 +

20 +If the plugin isn't installed yet, follow the [Installation page](/docs/network-flows/installation.md) first.

21 +

22 +## Step 1 — Configure your router

23 +

24 +Pick the closest match to your platform. The configurations below set sensible defaults: 60-second active timeout (industry best practice), 60-second template refresh (so a collector restart recovers in under a minute), and monitoring on both directions of an interface.

25 +

26 +### Cisco IOS / IOS-XE (Flexible NetFlow, v9)

27 +

28 +```

29 +flow exporter NETDATA

30 + destination 10.0.0.10 ! Netdata agent IP

31 + source GigabitEthernet0/0/0 ! source interface

32 + transport udp 2055

33 + export-protocol netflow-v9

34 + template data timeout 60

35 +!

36 +flow record NETDATA-RECORD

37 + match ipv4 source address

38 + match ipv4 destination address

39 + match transport source-port

40 + match transport destination-port

41 + match ipv4 protocol

42 + match interface input

43 + collect interface output

44 + collect counter bytes

45 + collect counter packets

46 + collect timestamp sys-uptime first

47 + collect timestamp sys-uptime last

48 +!

49 +flow monitor NETDATA-MONITOR

50 + record NETDATA-RECORD

51 + exporter NETDATA

52 + cache timeout active 60

53 + cache timeout inactive 15

54 +!

55 +interface GigabitEthernet0/0/1

56 + ip flow monitor NETDATA-MONITOR input

57 + ip flow monitor NETDATA-MONITOR output

58 +```

59 +

60 +### Juniper JunOS (J-Flow v9)

61 +

62 +```

63 +set forwarding-options sampling instance NETDATA family inet output flow-server 10.0.0.10 port 2055

64 +set forwarding-options sampling instance NETDATA family inet output flow-server 10.0.0.10 version9 template ipv4-template

65 +set services flow-monitoring version9 template ipv4-template flow-active-timeout 60

66 +set services flow-monitoring version9 template ipv4-template flow-inactive-timeout 15

67 +set services flow-monitoring version9 template ipv4-template template-refresh-rate seconds 60

68 +set interfaces ge-0/0/1 unit 0 family inet sampling input

69 +set interfaces ge-0/0/1 unit 0 family inet sampling output

70 +```

71 +

72 +### Arista EOS (sFlow)

73 +

74 +```

75 +sflow run

76 +sflow source-interface Loopback0

77 +sflow destination 10.0.0.10 2055

78 +sflow polling-interval 30

79 +sflow sample dangerous 2000

80 +!

81 +interface Ethernet1

82 + sflow enable

83 +```

84 +

85 +EOS treats sample rates below 16 384 as "aggressive" — the `dangerous` keyword is required to opt in. For higher-rate interfaces, drop the `dangerous` keyword and use 16 384 or above.

86 +

87 +### Linux host (`softflowd`, NetFlow v9)

88 +

89 +For Linux servers, hypervisors, or any host that doesn't natively speak NetFlow:

90 +

91 +```bash

92 +sudo softflowd -i eth0 -n 10.0.0.10:2055 -v 9 -t maxlife=60 -t expint=15

93 +```

94 +

95 +For more vendors and details, see [Sources / NetFlow](/src/crates/netflow-plugin/integrations/netflow.md), [IPFIX](/src/crates/netflow-plugin/integrations/ipfix.md), and [sFlow](/src/crates/netflow-plugin/integrations/sflow.md).

96 +

97 +## Step 2 — Open the dashboard

98 +

99 +In your browser, open the Netdata UI and click the **Network Flows** tab.

100 +

101 +By default you'll see:

102 +

103 +- A Sankey diagram on top, with a sortable table beneath

104 +- The default time range — last 15 minutes (Netdata's global picker)

105 +- Top-25 flows by bytes

106 +- Aggregated as **Source ASN → Protocol → Destination ASN**

107 +

108 +Within 60-90 seconds of the router being configured, flow records should start appearing.

109 +

110 +## Step 3 — Read the dashboard correctly

111 +

112 +Before drawing any conclusion, read this. It's the single biggest source of confusion when people first look at flow data.

113 +

114 +### Traffic looks doubled

115 +

116 +Routers normally export both ingress and egress flow records on every monitored interface. A packet that enters interface A and leaves interface B produces **two** records — one ingress on A, one egress on B.

117 +

118 +If you look at total bandwidth without filtering, you see roughly **2× the real traffic**. Add a second router on the same path and you see 4×.

119 +

120 +**To see real bandwidth on a specific link**, filter to one exporter and one direction:

121 +

122 +1. In the filter ribbon: `Exporter Name = <your router>`.

123 +2. Add: `Input Interface Name = <the interface>` (for incoming) **or** `Output Interface Name = <the interface>` (for outgoing). Pick one. Not both.

124 +

125 +That's the actual traffic on that link in that direction.

126 +

127 +### Conversations look mirrored

128 +

129 +Each bidirectional conversation produces two flow records — one for the request direction, one for the response. The Sankey, country map, and time-series all show both. When you see traffic between Country X and Country Y *and* traffic between Country Y and Country X of similar volume, that's the same conversation, not two.

130 +

131 +This is correct behaviour. To see only one direction of a conversation, filter by `Source ASN` (your network) for outbound or `Destination ASN` for inbound.

132 +

133 +## Step 4 — Verify it's working

134 +

135 +If the Sankey is empty after 60-90 seconds, work through this:

136 +

137 +1. **Datagrams arriving at the host.**

138 +

139 + ```bash

140 + sudo tcpdump -i any -nn -c 20 'udp port 2055'

141 + ```

142 +

143 + If you see packets, the network path is fine. If not, check the router's exporter status, the firewall, and the source IP the router uses.

144 +

145 +2. **Listener bound on the host.**

146 +

147 + ```bash

148 + sudo ss -unlp | grep 2055

149 + ```

150 +

151 + Should show `netflow-plugin` listening. If not, see [Troubleshooting](/docs/network-flows/troubleshooting.md).

152 +

153 +3. **Plugin actually decoding.**

154 +

155 + Open the standard Netdata charts page and find `netflow.input_packets`. If `udp_received` is rising but `parsed_packets` isn't, datagrams are arriving but failing to decode. Check `parse_errors` and `template_errors` to narrow down. See [Plugin Health Charts](/docs/network-flows/visualization/dashboard-cards.md).

156 +

157 +4. **Plugin log lines.**

158 +

159 + ```bash

160 + sudo journalctl -u netdata --since "5 minutes ago" | grep -i netflow

161 + ```

162 +

163 +## What's next

164 +

165 +You now have flow data flowing in. The natural next steps:

166 +

167 +- [Configuration](/docs/network-flows/configuration.md) — Tune retention so older data is preserved (the default 7-day shared retention is rarely enough).

168 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md) — Give your routers and your internal networks friendly names and labels. Without this, dashboards show raw IPs.

169 +- [Investigation Playbooks](/docs/network-flows/investigation-playbooks.md) — Concrete recipes for the questions flow data is good at answering.

170 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — Mistakes to avoid as you develop confidence with the data.

171 +- [Validation and Data Quality](/docs/network-flows/validation.md) — How to confirm your numbers are correct.

172 +

173 +For more sources or vendors:

174 +

175 +- [NetFlow](/src/crates/netflow-plugin/integrations/netflow.md) — More vendor configurations, sampling caveats.

176 +- [IPFIX](/src/crates/netflow-plugin/integrations/ipfix.md) — When and why to prefer IPFIX over NetFlow v9.

177 +- [sFlow](/src/crates/netflow-plugin/integrations/sflow.md) — Different protocol, different semantics.

docs/network-flows/retention-querying.md new

+153

@@ -0,0 +1,153 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/retention-querying.md"

3 +sidebar_label: "Retention and Querying"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['retention', 'tiers', 'querying', 'tier selection', 'rollup']

7 +endmeta-->

8 +

9 +# Retention and Querying

10 +

11 +Netdata stores flow data in four tiers. The tier model is transparent — you do not pick a tier when you query, the dashboard picks for you. Understanding how it picks helps you interpret what you're seeing and avoid surprises when older data isn't there.

12 +

13 +## The four tiers

14 +

16 +|---|---|---|---|

17 +| Raw | per-flow | `flows/raw/` | `raw` |

21 +

22 +The raw tier stores every flow record as it arrived. The other three are rollup tiers — they aggregate raw flows into time-bucketed groups by identity (exporter, ASN, country, ports — see below).

23 +

24 +## What survives the rollup

25 +

26 +Rollup tiers (1m, 5m, 1h) deliberately drop a few fields to keep cardinality manageable. **The dropped fields are: `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, `DST_PORT`, `SRC_GEO_CITY`, `DST_GEO_CITY`, `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE`, `SRC_GEO_LONGITUDE`, `DST_GEO_LONGITUDE`.**

27 +

28 +Everything else survives — country, state, ASN, AS path, BGP communities, exporter and interface labels, protocol, TCP flags, ToS/DSCP, ICMP type/code, MPLS labels, VLANs, MACs, next-hop, post-NAT addresses, and the bytes/packets sums. So rollups are perfectly fine for most country / ASN / interface / protocol questions, but useless if you need to ask "which IP".

29 +

30 +This is why filtering or grouping by IP/port/city/lat/lon forces the query to the raw tier — there is no other tier that has those fields.

31 +

32 +## How the dashboard picks a tier

33 +

34 +For every query the dashboard sends to the plugin, the planner makes a single decision: which tier (or tiers) can satisfy this?

35 +

36 +**Rules:**

37 +

38 +1. **Any IP/port/city/lat/lon filter or group-by → raw tier.** No exception. The rollup tiers don't have those fields.

39 +2. **A non-empty full-text search → raw tier.** Full-text search runs as a regex against the raw journal payload, which only the raw tier carries.

40 +3. **Otherwise, pick the coarsest tier that satisfies the time range and bucket-count requirement.**

41 + - Time-Series view needs at least 100 buckets in the window. So:

42 + - <100 minutes → 1-minute tier

43 + - 100 minutes to 8h20m → 5-minute tier

44 + - 8h20m and longer → 1-hour tier

45 + - Table / Sankey / Maps don't have a bucket-count constraint, but the configured query-window guardrails (`query_1m_max_window` default 6h, `query_5m_max_window` default 24h) skip a tier when the window is too wide.

46 +

47 +When the planner picks a tier and the time range crosses tier-aligned boundaries, the query is **stitched** — head fragment in a finer tier, aligned middle in the chosen tier, tail fragment in a finer tier. You don't see this; the results merge cleanly. It exists so wide windows that don't quite align to one-hour boundaries still work.

48 +

49 +The plugin reports the chosen tier in the response stats (`query_tier` = `0`, `1`, `5`, or `60`). The dashboard uses this for diagnostic banners.

50 +

51 +## What "no data" actually means

52 +

53 +If you ask for a 30-day window with an IP filter and tier-0 retention is 24 hours, you get an empty response. No error, no banner reading "data has expired" — just an empty result set. The dashboard renders this as "No data".

54 +

55 +The reason is a layered fallback in the planner: if a span asks for tier 0 and the files for that span have been rotated out, the planner tries the smaller tiers (1m, 5m, 1h), but those don't have IP fields, so they cannot satisfy a query that filters on IP. Result: the span returns no flows.

56 +

57 +Other spans within the same query that don't need raw data may still return flows. So it's also possible to see partial coverage — half the time range filled, half empty.

58 +

59 +For Time-Series, "no data" appears as zero values in the affected buckets, not as a special "missing" indicator. The chart still draws; the empty regions are flat lines at zero.

60 +

61 +## What forces tier 0 in practice

62 +

63 +Quick reference for "why is my query slow / showing less time?":

64 +

65 +- Adding `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, or `DST_PORT` as a filter

66 +- Adding any of those fields to the group-by

67 +- Switching to the city map (it uses `SRC_GEO_CITY`/`DST_GEO_CITY` plus latitudes/longitudes)

68 +- Typing anything into the global search ribbon

69 +

70 +If you see the time depth in your dashboard suddenly shrink after you applied a filter, you've hit the raw-tier limit.

71 +

72 +## Default retention and the most common misconfiguration

73 +

74 +The default `size_of_journal_files: 10GB` and `duration_of_journal_files: 7d` apply to **every tier independently**. With defaults, all four tiers (raw, 1m, 5m, 1h) are capped at 10GB / 7d.

75 +

76 +This is rarely what you want. The whole point of having rollup tiers is to keep them around longer than raw. A more useful production profile:

77 +

78 +```yaml

79 +journal:

80 + size_of_journal_files: 100GB # top-level inherited by tiers without an override

81 + duration_of_journal_files: 7d

82 + tiers:

83 + raw:

84 + size_of_journal_files: 200GB

85 + duration_of_journal_files: 24h

86 + minute_1:

87 + duration_of_journal_files: 14d

88 + minute_5:

89 + duration_of_journal_files: 30d

90 + hour_1:

91 + duration_of_journal_files: 365d

92 + size_of_journal_files: null # time-only, no size cap on the long tail

93 +```

94 +

95 +This gives you 24 hours of full-detail forensics, 14 days of 1-minute trends, 30 days of 5-minute snapshots, and a year of hourly aggregates.

96 +

97 +See [Sizing and Capacity Planning](/docs/network-flows/sizing-capacity.md) for how to estimate the actual disk footprint per tier from your flow rate.

98 +

99 +## How queries work, briefly

100 +

101 +The dashboard sends one of two query modes to the plugin:

102 +

103 +- **`flows`** — the normal aggregation request. Returns top-N groups, sums of bytes and packets, optional facet counts.

104 +- **`autocomplete`** — for the filter ribbon. Returns up to 100 facet values matching the user's term. Matching policy is per-field: text fields use substring matching, IP and numeric fields use prefix. Term is capped at 256 bytes. Runs against in-memory facet snapshots and on-disk FST sidecars; never scans tier files. Resulting filters apply as exact equality, not substring.

105 +

106 +A `flows` query carries:

107 +

108 +- A time range (`after` / `before`, or `last`).

109 +- A list of `group_by` fields (up to 10).

110 +- A list of `selections` — per-field IN-lists for filtering.

111 +- Optional `facets` to enrich the response with per-facet value counts.

112 +- A `top_n` (one of 25, 50, 100, 200, 500).

113 +- A `sort_by` (`bytes` or `packets`).

114 +- An optional regex `query` (full-text search; forces tier 0).

115 +- A `view` (`table-sankey`, `timeseries`, `country-map`, `state-map`, `city-map`).

116 +

117 +Defaults if you don't specify: time range = last 15 minutes, `group_by = ["SRC_AS_NAME", "PROTOCOL", "DST_AS_NAME"]`, `top_n = 25`, `sort_by = bytes`, `view = table-sankey`.

118 +

119 +The plugin enforces a hard timeout of **30 seconds** per query. If your query is too wide, narrow the time range, add a filter that lets a higher tier serve it, or reduce the group-by depth.

120 +

121 +## Group-by limits and overflow

122 +

123 +Two configuration limits guard against pathological queries:

124 +

125 +- `query_max_groups` (default `50000`) — total distinct groups in an aggregation. Past this, results overflow into a single `__overflow__` bucket and the response carries a warning.

126 +- `query_facet_max_values_per_field` (default `5000`) — distinct values returned per facet field.

127 +

128 +If you see `__overflow__` rows, your query is too wide for the current limit. Either narrow the filter, drop a high-cardinality `group_by` field, or raise the limit (carefully — the limit exists for memory reasons).

129 +

130 +## Full-text search

131 +

132 +The global search ribbon supports full-text search. It runs as a **regex** match against the raw journal payload. A search of `8.8.8.8` is the regex `8.8.8.8`, where each `.` matches any byte — so it can match unrelated text. To match the literal string, escape with backslashes: `8\.8\.8\.8`.

133 +

134 +Any non-empty full-text search forces the query to tier 0. Time depth is therefore limited by raw-tier retention.

135 +

136 +## URL sharing

137 +

138 +The dashboard URL preserves all of: time range, view, top-N, sort, group-by, selections, full-text search. Copy the URL and share it — the recipient sees exactly what you see, provided they have access to the same Netdata Cloud space.

139 +

140 +## Things that surprise people

141 +

142 +- **An IP filter shrinks the time depth.** This is correct behaviour, but the dashboard doesn't always make it obvious. If your time range is wider than tier-0 retention, drop the IP filter to see the broader rollup data.

143 +- **The city map can't go back as far as the country map.** The city map needs the city/lat/lon fields (raw-only); the country map only needs `SRC_COUNTRY`/`DST_COUNTRY` (preserved in rollups).

144 +- **`__overflow__` is a real value.** It will show up in result tables, sankey diagrams, and group-by listings. It means "everything that didn't fit in the top groups for this query" — narrow the filter or raise the limit.

145 +- **30-second timeout is hard.** A query that runs to the timeout returns whatever it has so far with a warning. Don't expect more than 30s of work per query.

146 +- **Tier files use short names** (`1m`, `5m`, `1h` on disk) but YAML uses the explicit names (`minute_1`, `minute_5`, `hour_1`). Mind the difference.

147 +

148 +## What's next

149 +

150 +- [Configuration](/docs/network-flows/configuration.md) — `netflow.yaml` reference, including per-tier retention overrides.

151 +- [Sizing and Capacity Planning](/docs/network-flows/sizing-capacity.md) — Disk and CPU estimates from your flow rate.

152 +- [Field Reference](/docs/network-flows/field-reference.md) — Which fields exist and which survive into rollups.

153 +- [Visualisation](/docs/network-flows/visualization/summary-sankey.md) — How the dashboard uses the tier model to render views.

docs/network-flows/sizing-capacity.md new

+137

@@ -0,0 +1,137 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/sizing-capacity.md"

3 +sidebar_label: "Sizing and Capacity Planning"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['sizing', 'capacity planning', 'storage', 'cpu', 'memory', 'benchmarks']

7 +endmeta-->

8 +

9 +# Sizing and Capacity Planning

10 +

11 +Use the following benchmarks and formulas to estimate the CPU, memory, and storage requirements for your Network Flows deployment.

12 +

13 +## What was measured

14 +

15 +These numbers come from a release-mode benchmark on an Intel i9-12900K workstation with a Seagate FireCuda 530 NVMe SSD (ext4). The benchmark runs the full ingest pipeline — raw journal plus the 1-minute, 5-minute, and 1-hour tiers — writing to real disk-backed journals. **Enrichment is not loaded** (no GeoIP/MMDB, no static metadata, no classifiers); enrichment adds CPU on top of the figures below.

16 +

17 +Cardinality is synthetic: low-cardinality cycles 256 unique flow records, high-cardinality cycles 4 096 unique records. Real exporter traffic falls between the two.

18 +

19 +CPU is reported as percent of one core (100% = one core fully consumed). The post-decode ingest path is currently single-threaded, so the practical ceiling per agent is bounded by one core's worth of CPU.

20 +

21 +## Practical headline

22 +

23 +On this hardware class, plan for:

24 +

25 +| Scenario | Practical ceiling |

26 +|---|---|

27 +| High-cardinality, all three protocols, four storage tiers, no enrichment, including UDP receive and decode | **~20-25 000 flows/s** |

28 +| Low-cardinality | comfortably above 60 000 flows/s |

29 +

30 +The 20-25k figure is conservative and includes decode cost (~10 µs/flow on top of the post-decode numbers below).

31 +

32 +## Detailed measurements (post-decode, paced)

33 +

34 +These tables show the cost of pre-decoded flows traversing the full ingest pipeline. To get the full UDP-to-disk cost, add roughly 10 µs/flow for protocol decoding.

35 +

36 +### Low cardinality

37 +

39 +|---:|---:|---:|---:|---|

40 +| 1 000 | 1.3% | 1.0% | 1.5% | ~25 MiB |

41 +| 10 000 | 12.6% | 11.5% | 16.9% | ~75 MiB |

42 +| 30 000 | 35.7% | 32.9% | 46.2% | ~85 MiB |

43 +| 60 000 | 70.3% | 64.1% | 87.1% | ~85 MiB |

44 +

45 +All three protocols deliver 100% of offered rate at every tested point. Saturation is above 60 000 flows/s and was not reached in this matrix.

46 +

47 +### High cardinality

48 +

50 +|---:|---:|---:|---:|---|

51 +| 10 000 | 10 000 | 9 970 | 9 990 | 28-37% |

52 +| 20 000 | 20 000 | 19 970 | 19 980 | 56-74% |

53 +| 30 000 | 29 331 | 29 985 | 29 257 | 84-98% |

54 +| 40 000 | 29 087 | 35 771 | 30 543 | 99% (plateau) |

55 +| 60 000 | 26 475 | 28 835 | 30 227 | 99% (plateau) |

56 +

57 +Saturation is around 30 000 flows/s on this host. Beyond the knee, the achieved rate plateaus at roughly the saturation value while the offered rate grows.

58 +

59 +:::warning

60 +These are host-specific reference points. Actual throughput depends on your CPU clock, disk speed, real flow cardinality, the number of populated fields, and any enrichment you enable (GeoIP, classifiers, static networks, ASN providers, BMP routing).

61 +:::

62 +

63 +## Storage

64 +

65 +Storage is governed by two things, not by the flow rate alone:

66 +

67 +- **Retention policy per tier** — caps how long each tier is kept and how much disk it can use.

68 +- **Cardinality and dedup** — flow records are indexed and key-value pairs are deduplicated. Low-cardinality traffic stores fewer bytes per flow than high-cardinality traffic, because repeated values share dictionary entries.

69 +

70 +Because the journals are not append-only logs, `flow_rate × bytes_per_flow × time` is not a valid estimator.

71 +

72 +### Empirical measurement on this hardware class

73 +

74 +A 15-minute run of paced ingest at 10 000 flows/s with the full pipeline active (raw + 1m + 5m + 1h tiers, real disk-backed journals) produced:

75 +

76 +| | Low cardinality (256 unique records) | High cardinality (4 096 unique records) |

77 +|---|---:|---:|

78 +| Flows ingested | 9.00 million | 8.97 million |

79 +| On-disk total | 6.46 GiB | 7.29 GiB |

80 +| Bytes per stored flow | **771** | **872** |

81 +| Write amplification (real I/O / logical encoded) | 1.79× | 2.00× |

82 +| Raw tier (final) | 6.45 GiB | 7.13 GiB |

83 +| 1-minute tier | 8 MiB | 112 MiB |

84 +| 5-minute tier | 8 MiB | 40 MiB |

85 +| 1-hour tier | 0 (rollup not reached in 15 min) | 16 MiB |

86 +

87 +Two key observations:

88 +

89 +- **Dedup is effective.** High cardinality stores only 13% more per flow despite 16× more unique field combinations. Real exporter traffic, which has heavy repetition (same src/dst/protocol patterns), will compress closer to the low-cardinality figure.

90 +- **Raw is 99% of the on-disk cost** at 15 minutes. The rollup tiers are small in absolute size because each rollup row aggregates many raw flows.

91 +

92 +### Bounding storage for capacity planning

93 +

94 +Set retention limits explicitly and let them bound the disk footprint:

95 +

96 +- raw: typically 24 hours

97 +- 1-minute tier: 14 days

98 +- 5-minute tier: 30 days

99 +- 1-hour tier: 365 days

100 +

101 +Configure per-tier `size_of_journal_files` (hard cap) and `duration_of_journal_files` (time cap). The plugin enforces whichever limit is hit first.

102 +

103 +For your own measurement, run the plugin against representative traffic for at least 15 minutes and inspect `du -sh` on each tier directory. The `bench_storage_footprint_child` test in this repository ships the same measurement harness used to produce the table above.

104 +

105 +## Memory

106 +

107 +Memory consumption is dominated by:

108 +

109 +- **Active journal rows** — flow records currently being accumulated before they are flushed to disk

110 +- **Field indexes** — structures that map field values (IPs, ASNs, ports) for fast filtering

111 +- **Facet indexes** — structures that power the filter sidebar

112 +- **GeoIP MMDB** — the IP-intelligence database (DB-IP-based by default) loaded into memory for enrichment

113 +

114 +The plugin exposes memory charts you can monitor:

115 +

116 +- `netflow.memory_resident_bytes` — total memory in use

117 +- `netflow.memory_allocator_bytes` — memory from the system allocator

118 +- `netflow.memory_accounted_bytes` — memory broken down by component (indexes, GeoIP, facets)

119 +- `netflow.memory_tier_index_bytes` — memory used by tiered storage indexes

120 +- `netflow.decoder_scopes` — protocol decoder memory usage

121 +

122 +## Disk I/O

123 +

124 +The plugin writes flow records to journal files continuously. Writes are dominated by the raw tier; the rollup tiers add a small amount on top. SSDs are recommended for collectors that handle thousands of flows per second — the index updates and frequent fsync calls benefit substantially from low-latency storage.

125 +

126 +Read operations only happen during queries. There is no background read activity in steady state.

127 +

128 +:::tip

129 +For production deployments, monitor the `netflow.memory_resident_bytes` chart and set a threshold alert. If resident memory grows steadily without stabilising, check your cardinality and consider reducing retention or increasing the sync interval.

130 +:::

131 +

132 +## What's next

133 +

134 +- [Configuration](/docs/network-flows/configuration.md) — Per-tier retention configuration and tuning knobs.

135 +- [Retention and Querying](/docs/network-flows/retention-querying.md) — How tiers are picked at query time.

136 +- [Validation and Data Quality](/docs/network-flows/validation.md) — How to confirm the numbers in your environment.

137 +- [Plugin Health Charts](/docs/network-flows/visualization/dashboard-cards.md) — Monitoring the plugin itself.

docs/network-flows/troubleshooting.md new

+243

@@ -0,0 +1,243 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/troubleshooting.md"

3 +sidebar_label: "Troubleshooting"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['troubleshooting', 'debugging', 'plugin health', 'failures']

7 +endmeta-->

8 +

9 +# Troubleshooting

10 +

11 +Concrete recipes for the most common failures, organised by symptom. Most issues are diagnosable from the [plugin health charts](/docs/network-flows/visualization/dashboard-cards.md), the Netdata journal logs, and a couple of OS-level commands.

12 +

13 +## The plugin doesn't start

14 +

15 +The plugin won't come up at all, or starts and immediately exits.

16 +

17 +**Symptoms:**

18 +- Netdata reports `netflow-plugin` as not running, or restart-looping.

19 +- Nothing in the Network Flows tab.

20 +- An error in `journalctl -u netdata`.

21 +

22 +**Likely causes:**

23 +

24 +| Cause | What to check |

25 +|---|---|

27 +| Required GeoIP DB missing (`optional: false`) | Same log search. Look for `failed to load database`. Either fix the path or set `optional: true`. |

28 +| Listen address conflict | Look for `failed to bind`. Another process is on the configured port (default 2055). |

29 +| Validation error | Look for `must be greater than 0` and similar. The plugin validates the full config at startup. |

30 +| `enabled: false` was set | Look for `netflow plugin disabled by config`. The plugin honours this and shuts down cleanly — looks like "not running" if you don't read the log. |

31 +

32 +**Recovery:**

33 +

34 +```bash

35 +# Read the failure

36 +sudo journalctl -u netdata --since "5 minutes ago" | grep -E 'netflow|failed to|error'

37 +

38 +# Validate the YAML (use an online linter or `yamllint`)

39 +yamllint /etc/netdata/netflow.yaml

40 +

41 +# After fixing, restart

42 +sudo systemctl restart netdata

43 +```

44 +

45 +## The plugin starts, but no flows appear

46 +

47 +The plugin is running, but the Network Flows tab is empty.

48 +

49 +**First check:** is anything reaching the plugin?

50 +

51 +```bash

52 +sudo tcpdump -i any -nn -c 50 'udp port 2055'

53 +```

54 +

55 +- **No packets in 30 seconds** — exporter not sending, or firewall blocking. Check the exporter's status (`show flow exporter` on Cisco, equivalents elsewhere) and the network path. The plugin can't help here; the data isn't reaching it.

56 +- **Packets arriving** — keep going.

57 +

58 +**Second check:** is the listener bound?

59 +

60 +```bash

61 +sudo ss -unlp | grep -E ':2055|netflow'

62 +```

63 +

64 +If nothing matches, the plugin isn't listening. See "doesn't start" above.

65 +

66 +**Third check:** what do the plugin's own counters say?

67 +

68 +Open `netflow.input_packets` on the standard Netdata charts page. The dimensions tell the story:

69 +

70 +- `udp_received > 0`, `parsed_packets == 0` — datagrams arriving, none decoding successfully. Wrong protocol on the listener, or all datagrams malformed.

71 +- `udp_received > 0`, `parsed_packets > 0`, but no per-protocol counter (`netflow_v9`, `ipfix`, etc.) is moving — the protocol you're sending may be disabled in the plugin config. Check `protocols.v9`, `protocols.ipfix`, etc. in `netflow.yaml`.

72 +- `parse_errors` rising in lockstep with `udp_received` — datagrams aren't valid for the protocols the plugin supports. Capture a sample (`tcpdump -w sample.pcap`) and inspect with Wireshark.

73 +

74 +## Partial data — some flows are dropped

75 +

76 +Counters show received traffic but you suspect data loss.

77 +

78 +**Template errors (NetFlow v9, IPFIX):**

79 +

80 +```bash

81 +# Watch the template_errors dimension

82 +# In the dashboard: netflow.input_packets > template_errors

83 +```

84 +

85 +If it's climbing, the exporter is sending data records before their templates. Either:

86 +

87 +- The exporter restarted and the plugin's template cache is stale. Wait for the exporter to send the next template (typically every 30-60 seconds, depending on its config), or restart the exporter to force an immediate template refresh.

88 +- Templates are sent rarely (Cisco's default template refresh is 30 minutes). After a plugin restart, you'll see template errors for that long. **Fix on the router side**: lower the template refresh interval to 60 seconds.

89 +- The exporter is using template IDs that collide with another exporter's templates. Most common cause: two exporters NATted behind the same public IP. Place the plugin inside the NAT boundary or give each exporter a distinct address.

90 +

91 +**UDP kernel drops:**

92 +

93 +The plugin doesn't count these. Check at the OS level:

94 +

95 +```bash

96 +sudo ss -uam sport = :2055 # check 'd' columns for drops

97 +cat /proc/net/udp | head -20 # RcvbufErrors column

98 +```

99 +

100 +If drops are occurring, the kernel UDP receive buffer is too small for the burst rate. Tune:

101 +

102 +```bash

103 +sudo sysctl -w net.core.rmem_max=33554432

104 +sudo sysctl -w net.core.rmem_default=8388608

105 +sudo sysctl -w net.core.netdev_max_backlog=250000

106 +```

107 +

108 +Persist in `/etc/sysctl.d/99-netflow.conf`.

109 +

110 +**Per-protocol switch off:**

111 +

112 +```yaml

113 +protocols:

114 + v5: false # are you accidentally rejecting v5 datagrams?

115 +```

116 +

117 +## Data is wrong — numbers don't match expectations

118 +

119 +**Volume looks doubled:**

120 +

121 +This is the most common report. With one router, traffic appears 2× because every packet generates an ingress record AND an egress record. With two routers on the same path, 4×. Filter to one exporter + one direction (input interface OR output interface) to see real volume. See [Anti-patterns](/docs/network-flows/anti-patterns.md).

122 +

123 +**Bandwidth doesn't match SNMP:**

124 +

125 +Several legitimate causes:

126 +

127 +- **Doubling**, as above. Filter properly before comparing.

128 +- **Sampling rate not honoured.** The plugin auto-multiplies bytes by the sampling rate, but if the exporter doesn't carry the rate (NetFlow v7 has no field for it; v5 sometimes sends 0 instead of the actual rate; v9 may not send the Sampling Options Template), the result is undercounted.

129 +- **Mixed sampling rates across exporters.** If your dashboard aggregates exporters with different rates, the result blends estimates and isn't comparable to any single SNMP measurement.

130 +- **SNMP includes layer-2 traffic** (ARP, STP, LLDP, routing protocols) that flow data filters out. Expect SNMP to be 5-15% higher than flow on a healthy collector. More than that, investigate.

131 +

132 +See [Validation and Data Quality](/docs/network-flows/validation.md).

133 +

134 +**Internal IPs in random countries:**

135 +

136 +GeoIP databases don't have entries for RFC 1918 / private space. The plugin doesn't skip private IPs — it just hands the IP to the database and uses what comes back. For the stock DB-IP build, private ranges are tagged so they render as "AS0 Private IP Address Space" with empty country. For other MMDBs, private ranges may resolve to weird countries.

137 +

138 +**Fix:** declare your internal CIDRs under `enrichment.networks` with country / role / name labels. See [Static metadata](/docs/network-flows/enrichment/static-metadata.md).

139 +

140 +**AS resolution chain misbehaving:**

141 +

142 +If `SRC_AS` / `DST_AS` are zero everywhere despite the exporter sending them, check the `asn_providers` chain:

143 +

144 +- `[geoip, ...]` — `geoip` is a terminal short-circuit. The chain stops at `geoip` (it returns 0). Reorder: `[flow, routing, geoip]`.

145 +- `[]` (empty) — no validation rejects this. Every AS is forced to 0.

146 +

147 +See [ASN resolution](/docs/network-flows/enrichment/asn-resolution.md).

148 +

149 +**Decapsulation eating non-tunnel traffic:**

150 +

151 +If you've enabled `decapsulation_mode: vxlan` and traffic that isn't VXLAN suddenly disappears from the L2-section path, that's by design — the decap is destructive on non-matching traffic. Standard NetFlow / IPFIX records (no IE 104 / IE 315) are unaffected.

152 +

153 +## Performance issues

154 +

155 +**High CPU:**

156 +

157 +```bash

158 +top -p $(pgrep -f netflow-plugin)

159 +```

160 +

161 +If `netflow-plugin` is using a lot of CPU:

162 +

163 +- Check `netflow.input_packets` — high `udp_received` rate? You're at the limit of what one core can do for the post-decode hot path. Each instance is single-process; you can't scale horizontally on one host.

164 +- If `udp_received` is moderate but CPU is high, classifier rules with complex regex might be the cause. Check `enrichment.classifier_cache_duration` — if too short, classifiers re-evaluate too often.

165 +- Investigate with `perf top` or similar to find the hot function.

166 +

167 +See [Sizing and Capacity Planning](/docs/network-flows/sizing-capacity.md) for measured throughput limits on this hardware class.

168 +

169 +**Memory growth:**

170 +

171 +```bash

172 +# Watch the resident memory chart over time

173 +# netflow.memory_resident_bytes - rss dimension

174 +```

175 +

176 +- If `rss` climbs and `netflow.memory_accounted_bytes` shows `unaccounted` growing, that's an unattributed allocation — could be allocator fragmentation, possibly a leak.

177 +- If `tier_indexes` or `open_tiers` is the climbing dimension, ingest is outpacing tier flushes. Check `netflow.materialized_tier_ops` for `flushes` rate and `*_errors`.

178 +- If `netflow.decoder_scopes` is growing without bound, your exporter is rotating template IDs. Investigate per-router behaviour.

179 +

180 +**Disk fill:**

181 +

182 +```bash

183 +sudo du -sh /var/cache/netdata/flows/*

184 +```

185 +

186 +Default retention is `10GB / 7d` per tier — the same budget applies to all four tiers, so total can reach roughly 40 GB plus some. If your config left this default and your collector is busy, expect to hit it. See [Configuration](/docs/network-flows/configuration.md) for per-tier overrides — most production deployments need them.

187 +

188 +## Things that look like bugs but aren't

189 +

190 +- **Traffic appears 2×.** Standard ingress + egress monitoring. Filter to one direction.

191 +- **Bidirectional conversations show twice.** A→B and B→A are real, distinct flows. Filter to one direction or one ASN to see one side.

192 +- **Internal IPs in odd countries.** GeoIP doesn't know about your private space. Declare it explicitly.

193 +- **City map empty over long windows.** City + lat/lon are tier-0-only. Default tier-0 retention is short. Use the country map for long ranges.

194 +- **`__overflow__` row in results.** Your aggregation produced more groups than `query_max_groups`. Narrow the filter or reduce group-by depth.

195 +- **30-second query timeout.** Hard limit. Narrow time range, add filters, or reduce group-by depth.

196 +- **Sampled byte counts not exact.** sFlow is statistical by design; even NetFlow with sampling is an estimate. Cross-check against SNMP for sanity, accept some divergence.

197 +- **`enabled: false` makes the plugin look crashed.** It's intentional — the plugin tells the parent to stop respawning it. Look for the "disabled by config" line in the journal.

198 +

199 +## Diagnostic command quick reference

200 +

201 +```bash

202 +# What's happening

204 +

205 +# What's arriving on the wire

206 +sudo tcpdump -i any -nn -c 50 'udp port 2055'

207 +

208 +# Is the listener bound

209 +sudo ss -unlp | grep 2055

210 +

211 +# UDP kernel drops

212 +sudo ss -uam sport = :2055

213 +cat /proc/net/udp

214 +

215 +# Disk usage by tier

216 +sudo du -sh /var/cache/netdata/flows/*

217 +

218 +# Process resources

219 +top -p $(pgrep -f netflow-plugin)

220 +

221 +# Capture a sample for offline analysis

222 +sudo tcpdump -w /tmp/netflow-sample.pcap -c 200 'udp port 2055'

223 +```

224 +

225 +## When to file an issue

226 +

227 +Collect this before opening a bug report:

228 +

229 +- Plugin version (`netdata --version` from the running daemon).

230 +- A sample of `netflow.input_packets` chart for the failure window — all dimensions visible.

231 +- A sample of `netflow.memory_resident_bytes` if performance-related.

232 +- A captured pcap (`tcpdump -w` from the agent's interface) reproducing the issue.

233 +- Sanitised `netflow.yaml` (redact internal IPs, customer names, secrets).

234 +- Relevant log lines from `journalctl -u netdata`.

235 +

236 +Open issues against [github.com/netdata/netdata](https://github.com/netdata/netdata) with `area/collectors/netflow` in the title.

237 +

238 +## What's next

239 +

240 +- [Plugin Health Charts](/docs/network-flows/visualization/dashboard-cards.md) — The charts referenced above.

241 +- [Validation and Data Quality](/docs/network-flows/validation.md) — How to spot silent data corruption.

242 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — Why some "weird" results are actually normal.

243 +- [Configuration](/docs/network-flows/configuration.md) — Tuning that affects most of the symptoms above.

docs/network-flows/validation.md new

+133

@@ -0,0 +1,133 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/validation.md"

3 +sidebar_label: "Validation and Data Quality"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows"

6 +keywords: ['validation', 'snmp cross-check', 'data quality', 'silent failures', 'sanity check']

7 +endmeta-->

8 +

9 +# Validation and Data Quality

10 +

11 +Flow data is statistical. It can be wrong in subtle ways that the dashboard cannot detect — silent UDP drops, undocumented sampling rate changes, exporters that stopped sending. This page is the routine you should run when you set up the plugin, when something looks suspicious, and periodically thereafter.

12 +

13 +The goal: distinguish "the data is correct" from "the data looks plausible but isn't".

14 +

15 +## The biggest risks are silent failures

16 +

17 +The most dangerous failures don't generate alerts. They look like data is flowing — just less of it, or skewed, or scaled wrong. Six common silent failures:

18 +

19 +1. **UDP datagram drops** — kernel drops happen when the receive buffer fills. Plugin sees fewer datagrams than the network sent. Counters are smaller; nothing logs the drop.

20 +2. **Sampling rate misinterpretation** — exporter samples 1-in-1000, no one documented it. Bytes look 1000× smaller than reality.

21 +3. **Sampling rate change** — someone reconfigures a router. Trends show a phantom 10× spike. No alert fires.

22 +4. **Wrong interfaces being exported** — flow export was enabled on three of five interfaces. Some traffic is invisible.

23 +5. **Template loss after collector restart** — v9 / IPFIX records arrive but cannot be decoded until the next template arrives. Counts dip silently.

24 +6. **Stale GeoIP / ASN database** — country and AS-name fields drift away from reality over weeks.

25 +

26 +For each, the system appears to be working. The only way to detect them is cross-validation against an independent source.

27 +

28 +## The minimum viable validation routine

29 +

30 +Run this once after deployment, then quarterly, plus whenever something looks off.

31 +

32 +### 1. SNMP cross-check (every 5 minutes if you have an SNMP collector handy)

33 +

34 +Compare flow-derived bandwidth on a specific interface to the SNMP `ifInOctets` / `ifOutOctets` counter for that same interface. They should be close.

35 +

36 +The flow-derived bandwidth: filter the dashboard to one exporter, one input interface (or one output interface — pick a direction), and read the bytes/s rate.

37 +

38 +The SNMP-derived bandwidth: from your SNMP monitoring (Netdata's snmp.d, your separate SNMP system, or your network team).

39 +

40 +**Acceptable difference: roughly 5-15%.** SNMP includes layer-2 traffic (ARP, STP, LLDP, routing protocols, interface-level multicast) that flow data filters out. Expect SNMP slightly higher.

41 +

42 +**Not acceptable: more than 30% gap.** That indicates one of:

43 +

44 +- UDP drops (kernel-level). Run `sudo ss -uam sport = :2055` and check the `dRcv` column.

45 +- Sampling rate not honoured. The exporter is sampling but not communicating the rate to the plugin (NetFlow v7, NetFlow v5 with rate=0, v9 / IPFIX without the Sampling Options Template).

46 +- Wrong interfaces being exported. Cross-check `show flow exporter` (or vendor equivalent) against your expectations.

47 +- Template loss. Watch `netflow.input_packets > template_errors` on the plugin health charts.

48 +

49 +**Plugin reporting wildly more than SNMP** indicates the doubling effect (see below).

50 +

51 +### 2. Doubling sanity check

52 +

53 +If your dashboard's total bandwidth exceeds the **physical link capacity**, you're double-counting. Standard NetFlow / IPFIX configuration produces two flow records per packet (one ingress, one egress). With multiple monitored routers on the same path, even more.

54 +

55 +Verify by: filter to one exporter and one interface in one direction (input OR output, not both). Compare to SNMP for that same interface. They should agree within 5-15%. The difference between "all flows summed" and "filtered to one direction" is exactly the doubling factor.

56 +

57 +### 3. Sampling rate sanity check

58 +

59 +For each exporter, document:

60 +

61 +- Does it sample? At what rate?

62 +- Does it carry the rate in flow records (NetFlow v9 / IPFIX) or in the header (v5)?

63 +- For NetFlow v9 / IPFIX, does the exporter send a Sampling Options Template? At what frequency?

64 +

65 +If the exporter samples and the plugin doesn't see the rate, bytes are undercounted.

66 +

67 +To verify the plugin sees the rate: query a known flow on the dashboard and look at `RAW_BYTES` and `BYTES`. If they differ, the plugin is multiplying — sampling rate is being honoured. If they're identical, the plugin sees rate 1 (no scaling).

68 +

69 +### 4. Per-exporter health check

70 +

71 +The plugin doesn't publish per-exporter ingest counters today. To verify each exporter is sending:

72 +

73 +- Filter the dashboard to one exporter at a time. Check the byte rate. A healthy edge router during business hours should show non-zero traffic.

74 +- An exporter that abruptly drops to zero is offline (silently). The plugin won't tell you — your monitoring practice has to.

75 +

76 +### 5. Template cache health (NetFlow v9 / IPFIX)

77 +

78 +On the plugin health chart `netflow.input_packets`, watch `template_errors`. In steady state, it should be near zero. A sustained non-zero rate means data records are arriving before their templates — usually because the exporter sends templates rarely (every 30 minutes is common Cisco default) and the plugin's template cache was wiped (restart with no persistence, or first-time setup).

79 +

80 +The plugin persists template state across restarts to `decoder_state_dir`, so a routine restart shouldn't cause this. If it does, check the cache directory permissions.

81 +

82 +### 6. GeoIP / ASN database freshness

83 +

84 +The plugin doesn't publish a "MMDB last loaded" signal. To verify your databases aren't stale:

85 +

86 +```bash

87 +ls -la /var/cache/netdata/topology-ip-intel/ /usr/share/netdata/topology-ip-intel/

88 +```

89 +

90 +Files older than ~60 days are likely stale. Refresh:

91 +

92 +```bash

93 +sudo /usr/sbin/topology-ip-intel-downloader

94 +```

95 +

96 +The plugin polls the files every 30 seconds — a successful refresh picks up automatically without restart.

97 +

98 +### 7. Internal IP enrichment validation

99 +

100 +Before relying on geographic analysis, spot-check that internal IPs are properly handled. Filter to an internal source IP you know and look at the `SRC_COUNTRY` and `SRC_AS_NAME` fields:

101 +

102 +- Empty / "AS0 Private IP Address Space" — correct.

103 +- Some random country — your GeoIP database is returning data for private space. Declare the range under `enrichment.networks` (see [Static metadata](/docs/network-flows/enrichment/static-metadata.md)).

104 +

105 +## Quick reference: what to monitor and what alerts to consider

106 +

107 +| Signal | Where | What to alert on |

108 +|---|---|---|

109 +| `udp_received` rate dropped | `netflow.input_packets` chart | Sustained 0 during business hours |

110 +| `template_errors` rising | `netflow.input_packets` chart | Sustained > 1% of `udp_received` |

111 +| `parse_errors` rising | `netflow.input_packets` chart | Sustained > 5% of `udp_received` |

112 +| Memory growing (`unaccounted`) | `netflow.memory_accounted_bytes` | RSS grows linearly without ingest growth |

113 +| `decoder_scopes` unbounded growth | `netflow.decoder_scopes` chart | Monotonic growth over hours |

114 +| Disk full warnings | `netflow.raw_journal_ops` `write_errors` | Any non-zero |

115 +| SNMP-flow gap | external | More than 30% on a steady-state link |

116 +| Sampling rate change | router config diff (yours) | Any change to active timeout or sampling |

117 +

118 +## When to file a "data is wrong" investigation

119 +

120 +Start an investigation when **two independent signals disagree**:

121 +

122 +- SNMP says 500 Mbps; flow data says 50 Mbps. Investigate sampling, drops, exporter coverage.

123 +- Flow data shows traffic to a country; threat intelligence says that country's ASN doesn't host known infrastructure. Investigate GeoIP or anycast.

124 +- Last week's top talker disappeared this week. Investigate exporter health, routing changes, business-side changes.

125 +

126 +For each, read the [Anti-patterns](/docs/network-flows/anti-patterns.md) page first — most "data is wrong" reports are actually expected behaviour misread.

127 +

128 +## What's next

129 +

130 +- [Plugin Health Charts](/docs/network-flows/visualization/dashboard-cards.md) — The charts referenced above.

131 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — Misreadings to rule out before declaring a bug.

132 +- [Investigation Playbooks](/docs/network-flows/investigation-playbooks.md) — Concrete recipes for common questions.

133 +- [Troubleshooting](/docs/network-flows/troubleshooting.md) — Recovery for the symptoms above.

docs/network-flows/visualization/dashboard-cards.md new

+103

@@ -0,0 +1,103 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/dashboard-cards.md"

3 +sidebar_label: "Plugin Health Charts"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Visualization"

6 +keywords: ['plugin health', 'metrics', 'operational charts', 'monitoring']

7 +endmeta-->

8 +

9 +# Plugin Health Charts

10 +

11 +The netflow plugin publishes its own operational charts under the `netdata.netflow.*` chart context. These appear on the standard Netdata charts page (alongside system metrics like CPU and memory), **not** inside the Network Flows tab. They are how you monitor the plugin itself: is it receiving data, are templates flowing, is memory growing, is disk being written.

12 +

13 +This is also where you look first when something seems wrong — long before opening the Network Flows tab.

14 +

15 +All charts update every 1 second.

16 +

17 +## The charts

18 +

19 +| Chart | Type | What it shows |

20 +|---|---|---|

21 +| `netflow.input_packets` | line, packets/s | Datagrams: received, parsed, errored, per-protocol counts |

22 +| `netflow.input_bytes` | line, bytes/s | UDP byte rate |

23 +| `netflow.raw_journal_ops` | line, ops/s | Raw journal: writes, sync calls, errors |

24 +| `netflow.raw_journal_bytes` | line, bytes/s | Raw journal logical bytes written |

25 +| `netflow.materialized_tier_ops` | line, ops/s | Rollup tiers: rows produced per tier, flushes, errors |

26 +| `netflow.materialized_tier_bytes` | stacked, bytes/s | Rollup tier byte rate, broken down by tier |

27 +| `netflow.open_tiers` | stacked, rows | Rows currently open per tier |

28 +| `netflow.journal_io_ops` | line, ops/s | Decoder-state persist operations and errors |

29 +| `netflow.journal_io_bytes` | line, bytes/s | Decoder-state persist byte rate |

30 +| `netflow.decoder_scopes` | line, scopes | Distinct (exporter, observation domain) scopes the decoder tracks |

31 +| `netflow.memory_resident_bytes` | line, bytes | Process RSS, peak RSS, breakdown |

32 +| `netflow.memory_resident_mapping_bytes` | stacked, bytes | RSS broken down by what's in it |

33 +| `netflow.memory_allocator_bytes` | line, bytes | Allocator-internal stats |

34 +| `netflow.memory_accounted_bytes` | stacked, bytes | RSS attributed to known components, plus `unaccounted` |

35 +| `netflow.memory_tier_index_bytes` | stacked, bytes | Tier-index memory drilldown |

36 +

37 +## Reading the most useful charts

38 +

39 +### `netflow.input_packets`

40 +

41 +The single most important chart. Five families of dimensions:

42 +

43 +- **`udp_received`** — datagrams pulled off the socket. If this is zero, nothing is reaching the plugin (firewall, no exporter, wrong port).

44 +- **`parse_attempts`, `parsed_packets`** — should track each other on a healthy collector. If `parse_attempts` is high but `parsed_packets` is low, datagrams are arriving but failing to decode.

45 +- **`parse_errors`** — counts datagrams that failed parsing for any reason (truncated, malformed, unsupported version).

46 +- **`template_errors`** — counts data records arriving before their template (v9 / IPFIX). Should be near zero in steady state. A sustained non-zero rate means the exporter is sending templates too rarely or your collector has lost template state.

47 +- **`netflow_v5`, `netflow_v7`, `netflow_v9`, `ipfix`, `sflow`** — per-protocol successful counts. Useful to identify "which protocol is actually arriving".

48 +

49 +### `netflow.decoder_scopes`

50 +

51 +Cardinality of decoder state. Reports how many distinct `(exporter, observation domain)` template caches the plugin currently holds. Watch for unbounded growth — an exporter that frequently rotates template IDs (rare but real) will inflate this without bound.

52 +

53 +### `netflow.materialized_tier_*`

54 +

55 +Show the rollup pipeline working. `*_rows` should track ingest. `flushes` should tick steadily; if it stops, tiering is stalled.

56 +

57 +### `netflow.memory_resident_bytes` and `netflow.memory_accounted_bytes`

58 +

59 +If RSS climbs over time:

60 +

61 +- Check `netflow.memory_accounted_bytes` to see where it's going.

62 +- The `unaccounted` dimension is `RSS - sum(known components)`. **A growing `unaccounted` is your leak signal.**

63 +- `tier_indexes` and `open_tiers` are normal sources of growth — they should track ingest rate.

64 +- `geoip_asn` and `geoip_geo` are mmap'd MMDB files. Their size grows as the kernel pages the file in under read pressure.

65 +

66 +### `netflow.memory_resident_mapping_bytes`

67 +

68 +This one breaks RSS down by what's mapped. Useful when you want to attribute "this process is using 800 MB" — heap, journals (per tier), MMDB files, anonymous mappings, etc.

69 +

70 +## What's NOT in these charts

71 +

72 +A few signals that aren't published today:

73 +

74 +- **Per-exporter ingest counter.** No per-source rate dimension. Decoder-scope cardinality tells you how many sources, not how busy each one is.

75 +- **UDP socket drops.** Kernel-level drops (full receive buffer, NIC drops) are not surfaced. Use the OS-level metrics: `cat /proc/net/udp` (column `RcvbufErrors`) or `ss -uam`.

76 +- **Template cache hit ratio.** `template_errors` counts misses; there's no corresponding "hits" counter to form a ratio.

77 +- **GeoIP staleness signal.** No "MMDB last loaded" timestamp or version. The mapping memory dimensions tell you if a database is loaded, not how old it is.

78 +- **Per-tier query latency.** These charts cover ingest and storage; query-side performance isn't observable.

79 +- **BioRIS counters.** They're collected internally but not published as chart dimensions today.

80 +

81 +If you need any of these, mention it in an issue — they're not hard to add but haven't been needed enough yet.

82 +

83 +## How to use these charts for diagnosis

84 +

85 +| Symptom | Look at | What it means |

86 +|---|---|---|

87 +| Network Flows tab is empty | `netflow.input_packets` `udp_received` | Zero = no datagrams arriving (firewall? wrong port?). Non-zero with `parsed_packets` zero = wrong protocol or all datagrams malformed. |

88 +| Sudden drop in flows | per-protocol dimensions | Identifies which protocol stopped (helps narrow whether it's a router, a router class, or all routers). |

89 +| Templates failing | `template_errors` rising | Exporter not sending templates often enough; collector lost cache; cache mismatch after firmware update. |

90 +| Cache growing without bound | `decoder_scopes` rising over hours | Exporter churn or unstable template IDs. Investigate per-router behaviour. |

91 +| Memory pressure | `netflow.memory_resident_bytes`, `netflow.memory_accounted_bytes` | If `rss` climbs and `unaccounted` is the dimension growing → unattributed allocation, possibly a leak. If `tier_indexes` or `open_tiers` climbs → ingest backpressure, flushing stalled. |

92 +| Disk write stalls | `netflow.raw_journal_ops` `write_errors`, `sync_errors` | Disk full, permission denied, fs error. |

93 +| Decoder state not persisting | `netflow.journal_io_ops` | `decoder_state_persist_calls` should tick periodically. `*_errors` should be 0. |

94 +

95 +## Where these are NOT shown

96 +

97 +These charts are **not** in the Network Flows tab. Look for them on the standard Netdata charts page, in the family `netflow`. The Network Flows tab itself shows traffic data, not plugin health.

98 +

99 +## What's next

100 +

101 +- [Troubleshooting](/docs/network-flows/troubleshooting.md) — Concrete diagnostic workflows.

102 +- [Validation and Data Quality](/docs/network-flows/validation.md) — Cross-checking plugin counters against SNMP.

103 +- [Configuration](/docs/network-flows/configuration.md) — Tuning that affects what these charts show.

docs/network-flows/visualization/filters-facets.md new

+84

@@ -0,0 +1,84 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/filters-facets.md"

3 +sidebar_label: "Filters and Facets"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Visualization"

6 +keywords: ['filters', 'facets', 'autocomplete', 'search', 'fts', 'negative match', 'visualization']

7 +endmeta-->

8 +

9 +# Filters and Facets

10 +

11 +The filter ribbon (between the visualisation and the table) is how you narrow flow data to the subset you want. Filters apply to every view — Sankey, table, time-series, country/state/city maps, globe — at once.

12 +

13 +## What you can filter on

14 +

15 +Around 80 fields are available as facets. They're a subset of the full 91-field schema:

16 +

17 +- **Excluded** — metric fields (`BYTES`, `PACKETS`, `FLOWS`, `RAW_BYTES`, `RAW_PACKETS`, `SAMPLING_RATE`), timestamp fields (`FLOW_START_USEC`, `FLOW_END_USEC`, `OBSERVATION_TIME_MILLIS`), and the four geo coordinate fields (latitude/longitude). These don't make sense as categorical filters.

18 +- **Two virtual facets**: `ICMPV4` and `ICMPV6` — synthesised on the fly from `PROTOCOL` plus the type and code fields. Filtering on `ICMPV4 = "Echo Request"` gives you that ICMP message type without writing two separate filters.

19 +

20 +Everything else — IPs, ports, protocol, AS numbers and names, country, state, city, exporter labels, interfaces, MACs, VLANs, NAT addresses, TCP flags, ToS, etc. — is filterable.

21 +

22 +## Filter logic

23 +

24 +Within a single field: **OR**. Selecting `PROTOCOL = TCP` and `PROTOCOL = UDP` shows TCP-or-UDP.

25 +

26 +Across different fields: **AND**. Adding `SRC_COUNTRY = US` to the above shows TCP-or-UDP from the US.

27 +

28 +## No negative match (yet)

29 +

30 +You cannot directly say "everything except X". The workaround is to select all values and remove the unwanted one — works for low-cardinality fields like `PROTOCOL` (a handful of values). For high-cardinality fields like `SRC_AS_NAME`, the autocomplete only surfaces the top 100 values, so there's no practical way to "select all and remove".

31 +

32 +This is a real limitation. Negative match is a known feature gap.

33 +

34 +## Autocomplete

35 +

36 +Type into a facet field and the dashboard suggests existing values from your live data. The list:

37 +

38 +- Shows up to **100 matching values**, sorted alphabetically.

39 +- Matching policy is per-field. Free-form text fields (`SRC_AS_NAME`, `EXPORTER_NAME`, `IN_IF_DESCRIPTION`, MAC addresses, AS paths, BGP communities, country/city/state names) match by **substring**, so typing `Akamai` finds `AS20940 Akamai International`. IPs and short numeric fields (ports, protocols, ASN numbers, interface speeds) match by **prefix**, so typing `10.0.` narrows to that range.

40 +- Runs against an **in-memory snapshot of the live journal** plus on-disk FST sidecars for promoted high-cardinality fields. Autocomplete never reads the raw flow tiers, and is fast even on busy collectors.

41 +- The autocomplete `term` is hard-capped at 256 bytes; longer requests are rejected.

42 +

43 +For high-cardinality fields, autocomplete is the only practical way to discover values. You can't scroll a list of millions of IP addresses, but you can find one by typing what you remember.

44 +

45 +**Autocomplete and regular filtering are different paths.** When you select a value from the dropdown, the resulting filter is **exact equality**, not substring. The dropdown only helps you discover values; the filter that gets applied is `key = value` (or `key in [values]`) and uses indexes — never a substring scan over flow data.

46 +

47 +## Full-text search

48 +

49 +The search box at the top of the filter ribbon performs a regex match against the raw journal payload bytes. Notes:

50 +

51 +- The search is **regex**, not literal. `8.8.8.8` is a regex where `.` matches any byte — so it can match `8a8b8c8`, `888x888`, etc. To match the literal string, escape with backslashes: `8\.8\.8\.8`.

52 +- The match is **byte-level** against the journal payload, so it can find substrings inside enriched fields (AS names, exporter names, country codes).

53 +- Any non-empty search **forces tier 0**. The full-text search only works against the raw journal — it doesn't apply to the rollup tiers. Time depth is therefore bounded by raw-tier retention.

54 +- The plugin's "fast aggregation" path is also disabled when full-text search is active, because aggregation needs to scan every record. Expect somewhat slower responses than tier-based aggregation queries.

55 +

56 +For "find anything containing this string in any field", the search is the right tool. For "filter by an exact value of a specific field", use the facet on that field — it's faster and doesn't trigger tier-0 mode.

57 +

58 +## URL preservation

59 +

60 +Every filter and selection is preserved in the dashboard URL. Copy the URL and share it; the recipient sees the exact same view, provided they have access to the same Netdata Cloud space. The dashboard also remembers your last selections per session, so you'll land on the same configuration when you return.

61 +

62 +A practical note: filters use a structured representation (per-field IN-list) that's easy to encode as a JSON payload but awkward to URL-encode. The dashboard handles this transparently for sharing — but anyone scripting their own queries against the function should use JSON-payload requests to the function, not GET-style args.

63 +

64 +## Facet limits

65 +

66 +`query_facet_max_values_per_field` (default `5000`) caps how many distinct values a single facet can return per query. Past that, the facet stops accumulating; the response carries an indicator. Useful when you have an extremely high-cardinality field — autocomplete still surfaces the top 100, but the full list is bounded.

67 +

68 +You can raise this limit in `netflow.yaml`. Higher values use more memory at query time.

69 +

70 +## Things that go wrong

71 +

72 +- **Search for `192.168.1.1` matches unrelated rows.** Regex semantics: each `.` is "any byte". Escape: `192\.168\.1\.1`.

73 +- **Time depth shrinks unexpectedly after typing in search.** Full-text search forces tier 0. Clear the search to use rollup tiers and longer time ranges.

74 +- **Negative match isn't there.** Workaround: select-all-minus-one for low-cardinality fields. For high-cardinality fields, no good workaround exists today.

75 +- **Filter on an ICMP virtual facet seems slower than expected.** `ICMPV4` / `ICMPV6` virtual facets aren't optimised by the journal index — they're evaluated per-record. The query still returns; the cost shows up as longer wall time on busy collectors.

76 +- **`query_max_groups` exceeded.** Result rows after the limit fold into `__overflow__`. Narrow the filter or reduce group-by depth.

77 +- **GET-style args don't carry selections.** When integrating the function call yourself, send a JSON payload — the dashboard does this automatically.

78 +

79 +## What's next

80 +

81 +- [Sankey and Table](/docs/network-flows/visualization/summary-sankey.md) — The view that filters drive most often.

82 +- [Retention and Querying](/docs/network-flows/retention-querying.md) — Why filters can shift the tier the query uses.

83 +- [Field Reference](/docs/network-flows/field-reference.md) — Which fields are available as facets.

84 +- [Investigation Playbooks](/docs/network-flows/investigation-playbooks.md) — Practical filter-driven workflows.

docs/network-flows/visualization/maps-globe.md new

+114

@@ -0,0 +1,114 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/maps-globe.md"

3 +sidebar_label: "Maps and Globe"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Visualization"

6 +keywords: ['country map', 'state map', 'city map', 'globe', 'visualization']

7 +endmeta-->

8 +

9 +# Maps and Globe

10 +

11 +Four geographic views, all driven by the same aggregation engine as the Sankey and Time-Series:

12 +

13 +- **Country map** — countries connected by edges weighted by traffic

14 +- **State map** — same, at state/province granularity

15 +- **City map** — same, at city level (down to street-level granularity, depending on your GeoIP database)

16 +- **Globe** — a 3D view of city-level connections rendered as arcs over the globe

17 +

18 +Use these to spot geographic patterns at a glance — unexpected destinations, asymmetric traffic, CDN routing.

19 +

20 +![Country map, top 500](https://github.com/user-attachments/assets/f9f09cf2-40c5-4bda-bf56-19b04b6cddf1)

21 +

22 +Country map with top-N pushed to 500, so practically every country with traffic shows up. Edge thickness is bandwidth aggregated per country pair.

23 +

24 +## How they work

25 +

26 +For each map view, the dashboard:

27 +

28 +1. Forces a specific aggregation. You don't pick `group_by` for these views — the view picks for you.

29 +2. Runs the aggregation across your time range and filters.

30 +3. Renders the top-N (25/50/100/200/500) results as edges on the map.

31 +4. Same aggregation drives the side-panel list of countries / cities. The list and the map are two views of the same data.

32 +

33 +The forced aggregations are:

34 +

35 +| View | Forced group-by |

36 +|---|---|

37 +| Country map | `SRC_COUNTRY`, `DST_COUNTRY` |

38 +| State map | `SRC_COUNTRY`, `SRC_GEO_STATE`, `DST_COUNTRY`, `DST_GEO_STATE` |

39 +| City map | `SRC_COUNTRY`, `SRC_GEO_STATE`, `SRC_GEO_CITY`, latitude, longitude (source + destination) |

40 +| Globe | Same as city map |

41 +

42 +Edge width is proportional to your sort metric (bytes or packets). The geographic coordinates needed to draw cities and arcs come from the response itself — they're already enriched into each flow record by the time the dashboard renders. You don't need a separate city-coordinates database in the dashboard.

43 +

44 +## Country and state vs city / globe

45 +

46 +The country map and state map can use the rollup tiers. They're cheap over long time windows.

47 +

48 +The city map and the globe **need raw-tier data**. City, latitude, and longitude are dropped from the rollup tiers (1m / 5m / 1h) to keep cardinality manageable. So:

49 +

50 +- Country / state map over the last 30 days — fine, uses the 1-hour tier.

51 +- City map over the last 30 days — likely empty. Tier 0 retention defaults to 7 days (shared budget across all tiers); often less in practice.

52 +

53 +If your city map looks empty over a long window, try the country map first to confirm data is arriving, then narrow the time range until the city map fills in.

54 +

55 +## Tooltips

56 +

57 +Hover over a country, state, city, or arc to see a tooltip. The tooltip shows the same fields as the underlying row — endpoints, byte and packet counts. Click does **not** drill down to a different view; the maps are read-only with respect to navigation. To change perspective (e.g., "show me traffic for this country only"), use the filter ribbon to add a `SRC_COUNTRY` or `DST_COUNTRY` selection.

58 +

59 +![State map zoomed over the US, hovering an Attica↔California link](https://github.com/user-attachments/assets/6f124a7c-e12f-453f-8599-59e48bc839e8)

60 +

61 +State map with top-N at 500, zoomed over the US. The tooltip on the link between Attica (Greece) and California shows bidirectional traffic — bytes and packets in each direction.

62 +

63 +![City map zoomed over Europe](https://github.com/user-attachments/assets/e752e1e3-4f6a-4366-b2e2-6af04d4bc2fe)

64 +

65 +City map with top-N at 500, zoomed over Europe. Dozens of European cities appear connected by edges weighted by bandwidth.

66 +

67 +![Globe view over the Atlantic, US ↔ EU links](https://github.com/user-attachments/assets/c83a963d-797f-44f8-9ae1-e9aba7e16eec)

68 +

69 +Globe view, top-N at 500, rotated over the Atlantic. The 3D projection shows US cities and EU cities at the curvy edges, with arcs (bandwidth-thickness) bridging them.

70 +

71 +## Things to know

72 +

73 +### GeoIP is required

74 +

75 +Without a GeoIP database, country / state / city / coordinate fields are empty and the maps are blank. The default install includes a stock DB-IP database — see [GeoIP enrichment](/docs/network-flows/enrichment/ip-intelligence.md). Source builds need the operator to run the downloader once.

76 +

77 +### Internal IPs in random countries

78 +

79 +If you see "traffic from China" or "traffic to Russia" coming from your own network, that's almost always GeoIP misidentifying internal IPs. The fix is to declare your internal CIDRs explicitly under `enrichment.networks` with a country override. See [Static metadata](/docs/network-flows/enrichment/static-metadata.md). Don't trust GeoIP for RFC 1918 / RFC 6598 / link-local addresses.

80 +

81 +### CDN traffic shifts

82 +

83 +Your traffic to a SaaS provider may resolve to one country today and another tomorrow because the CDN's routing changed. This is normal CDN behaviour, not a security incident. ASN-based aggregation is more stable for cloud / CDN traffic than country-based — see the [Anti-patterns page](/docs/network-flows/anti-patterns.md) "Geographic firewall of shame".

84 +

85 +### Mirroring

86 +

87 +Bidirectional conversations show up as two arcs (A→B and B→A). With the default 25 top-N, that means about 12 actual conversations get rendered, not 25. To see one direction only, filter on a specific source or destination.

88 +

89 +### Globe vs City Map

90 +

91 +The globe and city map use the same data. The globe is purely a different rendering of the same response — useful for visual presentation, less useful for analysis (the 3D projection makes precise reading harder than a 2D map).

92 +

93 +## What controls are available

94 +

95 +- **Time range** — Netdata's global time picker

96 +- **Filters** — facet selections + autocomplete + full-text search

97 +- **Top-N** — 25 / 50 / 100 / 200 / 500

98 +- **Sort by** — bytes or packets (determines edge weight and the side-list ranking)

99 +- **Group-by** — locked to the view-specific aggregation; not user-configurable for maps

100 +

101 +## Things that go wrong

102 +

103 +- **City map empty.** Time range exceeds tier-0 retention. Narrow the range, or use country/state map for a wider view.

104 +- **Random countries appearing for internal traffic.** Declare your internal CIDRs in `enrichment.networks`.

105 +- **Ireland or Singapore showing up unexpectedly.** Probably AWS/GCP/Azure shifting CDN routing. ASN-based aggregation is more stable.

106 +- **A whole country disappears.** Your filter excluded it. Check the filter ribbon.

107 +- **No data on globe but city map works.** Both should fail or succeed identically — they consume the same response. If they diverge, that's a dashboard bug worth reporting.

108 +

109 +## What's next

110 +

111 +- [GeoIP enrichment](/docs/network-flows/enrichment/ip-intelligence.md) — Required for any geographic visualization.

112 +- [Static metadata](/docs/network-flows/enrichment/static-metadata.md) — Declare your internal networks to override GeoIP for RFC 1918.

113 +- [Filters and Facets](/docs/network-flows/visualization/filters-facets.md) — Narrowing geographic views.

114 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — Why "alert on traffic to country X" is fragile.

docs/network-flows/visualization/summary-sankey.md new

+117

@@ -0,0 +1,117 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/summary-sankey.md"

3 +sidebar_label: "Sankey and Table"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Visualization"

6 +keywords: ['sankey', 'table', 'top-n', 'aggregation', 'visualization']

7 +endmeta-->

8 +

9 +# Sankey and Table

10 +

11 +The Network Flows tab opens with two views stacked: a Sankey diagram on top, a sortable table beneath. Both render the same data, the same top-N aggregation, the same field selection. Selecting fields, filtering, and sorting affects both at once.

12 +

13 +This is your default view. Most investigative workflows start here.

14 +

15 +![Sankey top 25 with table, filtered by source IP](https://github.com/user-attachments/assets/314e7195-3fff-4bc7-a01a-0d59ded513d7)

16 +

17 +Sankey + table side by side, with a `SRC_ADDR` filter applied. The table at the bottom shows the same 25 rows as the diagram above; clicking a column header re-sorts both. The filter ribbon at the top is how you narrow to one source.

18 +

19 +## What you see by default

20 +

21 +When you first open the tab:

22 +

23 +- **Time range**: last 15 minutes (Netdata's global time picker default — it applies to metrics, logs, flows, and topology together)

24 +- **View**: Sankey + Table

25 +- **Top-N**: 25 (selectable: 25 / 50 / 100 / 200 / 500)

26 +- **Sort by**: bytes (alternative: packets)

27 +- **Aggregation fields**: `Source ASN → Protocol → Destination ASN`

28 +- **No filters applied** — the dashboard remembers your last selections, so on subsequent visits you'll land on whatever you had open

29 +

30 +The Sankey shows the top 25 conversations between Source ASN, Protocol, and Destination ASN, weighted by bytes. The table below shows the same 25 rows, with bytes and packets columns appended.

31 +

32 +## How to read the Sankey

33 +

34 +A Sankey diagram has columns of nodes and weighted bands flowing between them.

35 +

36 +- Each **column** corresponds to one of your selected aggregation fields, in the order you specified.

37 +- Each **node** is one distinct value in that column (e.g., one ASN, or one country).

38 +- Each **band** is one row in the underlying top-N — its width is proportional to the bytes (or packets) for that combination.

39 +

40 +With the default 3-column setup (Source ASN → Protocol → Destination ASN), you see the top 25 (Source ASN, Protocol, Destination ASN) tuples by traffic volume. A wide band from `AS65000` to `tcp` to `AS15169` says "AS65000 sent a lot of TCP to AS15169 in this time window".

41 +

42 +You can pick **1 to 10 fields** as columns. Order matters — the Sankey draws bands left-to-right in the order you list. There are roughly 84-85 fields available for aggregation; metric fields (`BYTES`, `PACKETS`, sampling rate, timestamps) and the geo coordinates (latitude/longitude) are not selectable here.

43 +

44 +![Sankey top 25, table collapsed](https://github.com/user-attachments/assets/c846e80f-b1d1-4330-bb46-162d021604fa)

45 +

46 +Same view with the table folded away — useful when you want the Sankey's full vertical real estate. Click the table header to expand it again.

47 +

48 +## Top-N is "top-N grouped tuples"

49 +

50 +When you set top-N to 25, the response contains the **25 top group-by tuples**, ranked by your sort metric. The 26th-largest and beyond are folded into a synthetic `__other__` row that represents "everything else, summed".

51 +

52 +If your aggregation produces enormously many distinct tuples (more than `query_max_groups`, default 50 000), an additional `__overflow__` row appears, summing everything that didn't fit in the in-memory accumulator. Both `__other__` and `__overflow__` are real rows in the response and may show up in the Sankey and the table — they aren't bugs, they're "everything off the bottom of the list".

53 +

54 +To narrow further: filter, or change the aggregation columns. Bumping top-N higher (200, 500) helps for shallow searches; for serious investigation, filter.

55 +

56 +## How to read the Table

57 +

58 +The same data, sortable and column-customisable.

59 +

60 +- One row per top-N tuple

61 +- One column per aggregation field

62 +- Plus `bytes` and `packets` columns, both sortable

63 +

64 +`SRC_AS_NAME` and `DST_AS_NAME` columns get extra width because AS names are long. Latitude / longitude columns are present in the underlying data but **hidden by default** — they're carried through so the city map and globe views can use them, but they aren't useful in tabular form.

65 +

66 +Click any column header to re-sort. Click a value to add a filter on that field. The same filter applies to the Sankey.

67 +

68 +## The filter ribbon

69 +

70 +A filter strip sits between the Sankey and the table. Three things you can do here:

71 +

72 +- **Select facet values** — click a field, pick one or more values. The query updates.

73 +- **Autocomplete** — type into a facet field; the dashboard suggests existing values from the live data. Useful for high-cardinality fields like AS names.

74 +- **Free-text search** — anything you type in the search box runs as a regex against the raw journal data.

75 +

76 +Filter logic is "AND across fields, OR within a field". Selecting `PROTOCOL = TCP` and `PROTOCOL = UDP` shows TCP-or-UDP. Selecting `PROTOCOL = TCP` plus `SRC_COUNTRY = US` shows only US-source TCP.

77 +

78 +There is no negative match. To exclude a value, select all values and remove the unwanted one — works for low-cardinality fields, becomes impractical for high-cardinality ones (the autocomplete cap is 100).

79 +

80 +See [Filters and Facets](/docs/network-flows/visualization/filters-facets.md) for the full mechanics.

81 +

82 +## Choosing aggregation fields

83 +

84 +Some shapes that work well:

85 +

86 +- **Default**: `Source ASN → Protocol → Destination ASN`. The "who, on what, to whom" overview. Good first look.

87 +- **Country flow**: `Source Country → Destination Country`. Cleanest geographic view. Combine with `protocol` for service-level detail.

88 +- **Per-router slice**: `Exporter Name → Input Interface → Destination ASN`. Use when you have per-router questions.

89 +- **Service drill-down**: `Destination Port → Source ASN`. Who's hitting your services.

90 +- **Internal/external split**: `IN_IF_BOUNDARY → DST_COUNTRY → Destination ASN`. After labelling your boundaries via static metadata.

91 +

92 +The order of fields determines the visual flow. Reorder to change which dimension is "left" and "right" in the Sankey.

93 +

94 +## Things to know

95 +

96 +### Doubling

97 +

98 +Without filtering, aggregate volume on a single router is roughly 2× the actual traffic — every packet generates two flow records (one ingress, one egress). To see real volume on a specific link, filter to one exporter and one direction (input interface OR output interface, not both). See [Anti-patterns](/docs/network-flows/anti-patterns.md) for the full framing.

99 +

100 +### Sharing your view

101 +

102 +The dashboard URL preserves all state — time range, filters, aggregation fields, top-N, sort. Copy the URL and share with anyone who has access to the same Netdata Cloud space.

103 +

104 +### Limits

105 +

106 +- `top_n` clamps to one of {25, 50, 100, 200, 500}.

107 +- Maximum 10 group-by fields. More are silently truncated.

108 +- Maximum 50 000 distinct group tuples per query (`query_max_groups`); over that, surplus folds into `__overflow__`.

109 +- The query itself has a 30-second hard timeout.

110 +

111 +## What's next

112 +

113 +- [Filters and Facets](/docs/network-flows/visualization/filters-facets.md) — Filtering mechanics in detail.

114 +- [Time-Series](/docs/network-flows/visualization/time-series.md) — How traffic evolves over the time window.

115 +- [Maps and Globe](/docs/network-flows/visualization/maps-globe.md) — Geographic views.

116 +- [Field Reference](/docs/network-flows/field-reference.md) — Which fields are available for aggregation.

117 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — How to read the numbers correctly.

docs/network-flows/visualization/time-series.md new

+96

@@ -0,0 +1,96 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/visualization/time-series.md"

3 +sidebar_label: "Time-Series"

4 +learn_status: "Published"

5 +learn_rel_path: "Network Flows/Visualization"

6 +keywords: ['time series', 'top-n over time', 'trends', 'visualization']

7 +endmeta-->

8 +

9 +# Time-Series

10 +

11 +The Time-Series view plots traffic over time. Same top-N selection as the Sankey + Table, but rendered as a stacked chart across the time range you've selected.

12 +

13 +Use it for: anomaly detection, trending, comparing now to last week, capacity planning. Use the Sankey/Table view for: "what's the breakdown right now".

14 +

15 +![Time-Series top 25 with table](https://github.com/user-attachments/assets/0bb2637d-632b-4e97-900f-b14155ab0771)

16 +

17 +Stacked chart on top, table at the bottom. Each colored band is one of the 25 top groups, summed into time buckets. The table holds the same 25 rows with totals across the whole window.

18 +

19 +## How it works

20 +

21 +The view runs the same aggregation as the Sankey + Table:

22 +

23 +1. The plugin scans the journal across your time range, aggregating by your group-by fields, summing bytes and packets.

24 +2. It picks the top-N groups by your sort metric (bytes or packets) over the **whole** window.

25 +3. It re-scans the journal and accumulates those top-N groups into time buckets.

26 +4. The result is a stacked chart with one dimension per top-N group.

27 +

28 +The top-N is computed once over the entire window — not per bucket. A flow that's huge for 5 minutes and absent the rest of the time may not make the top-N if a steady mid-volume flow accumulates more total bytes over the same window. If you want to see those bursts, narrow the time range or filter to the conversation you're investigating.

29 +

30 +## Bucket size

31 +

32 +The view auto-picks a bucket size based on the time range:

33 +

34 +| Time range | Tier used | Bucket size |

35 +|---|---|---|

36 +| ≤ ~100 minutes | 1-minute | 60 seconds (the floor) |

37 +| 100 minutes to ~8h20m | 5-minute | 300 seconds |

38 +| ≥ ~8h20m | 1-hour | 3600 seconds |

39 +

40 +The rule: pick the coarsest tier where the time window contains at least 100 buckets, with bucket size `max(tier_bucket, 60)`. For longer ranges, the bucket grows in proportion to keep the chart readable (capped at 500 buckets total).

41 +

42 +The window is **rounded outward** to align with bucket boundaries — your "11:23:00 to 11:48:00" request may render as "11:23:00 to 11:48:30" if the bucket size doesn't divide your range evenly. This is intentional; it ensures every record in your window is reachable.

43 +

44 +### Sub-minute zoom

45 +

46 +The minimum bucket size is **60 seconds**. Zoom in past one minute and the chart silently widens to 60-second buckets. There's no warning — sub-minute jitter just smooths out. For sub-second analysis, flow data is the wrong tool ([microbursts are invisible](/docs/network-flows/anti-patterns.md)).

47 +

48 +## What forces tier 0 (raw)

49 +

50 +Some queries can't use the rollup tiers. They drop to tier 0 and inherit raw-tier retention:

51 +

52 +- Filtering or grouping by `SRC_ADDR`, `DST_ADDR`, `SRC_PORT`, `DST_PORT`, or any geo city / latitude / longitude field

53 +- Any non-empty full-text search

54 +

55 +In those cases the 100-bucket rule still applies, but the source tier is tier 0. Time depth is bounded by raw-tier retention (default: shared 10GB / 7d budget across all tiers — almost always less than 7 days for a busy collector).

56 +

57 +If you've been working at a higher tier and add an IP filter, the time depth on your chart may suddenly shrink — that's the tier switch.

58 +

59 +## "No data" buckets

60 +

61 +Buckets that received no contributing records render as zero. There's no special "missing data" indicator on the chart — the plot is flat at zero in those regions.

62 +

63 +That includes the case where the time range crosses the retention boundary of a tier. Tier 0 (raw) holds the most recent data; older fragments fall back to coarser tiers when available, and emptiness when no tier has the span.

64 +

65 +The dashboard's diagnostic side-panels surface tier coverage in the response stats (`query_tier`, `query_files`, etc.), but the chart itself doesn't visually distinguish "no data" from "zero".

66 +

67 +## Group overflow

68 +

69 +Same overflow semantics as the Sankey + Table view. If your aggregation produces more than `query_max_groups` (default 50 000) distinct group tuples, the surplus is folded into a synthetic `__overflow__` group, which appears as one of the chart's dimensions. Look for the warning in the response stats; narrow your filter or reduce the group-by depth to avoid it.

70 +

71 +## What controls are available

72 +

73 +Same controls as the other views:

74 +

75 +- **Time range** — Netdata's global time picker

76 +- **Filters** — facet selections + autocomplete + full-text search (in the filter ribbon)

77 +- **Top-N** — 25 / 50 / 100 / 200 / 500

78 +- **Sort by** — bytes or packets (determines what "top" means and what units the chart uses)

79 +- **Group-by fields** — same as Sankey, 1-10 fields. The chart shows one stacked dimension per surviving top-N group

80 +

81 +The default group-by is `Source ASN → Protocol → Destination ASN`, same as Sankey + Table.

82 +

83 +## Things that go wrong

84 +

85 +- **Bursty flow not in top-N.** Top-N is over the whole window. Narrow the time range or filter to that conversation.

86 +- **Sub-minute zoom doesn't render finer.** The 60-second floor is hard. For finer detail, use packet capture.

87 +- **Wide range plus IP filter shows less than expected.** IP filter forced tier 0; raw retention is your bound.

88 +- **Window appears to extend slightly beyond what you asked.** Bucket alignment rounds outward.

89 +- **`__overflow__` shows up as the biggest dimension.** Your group-by is producing more distinct tuples than `query_max_groups` (50 000). Narrow the filter or drop a high-cardinality group-by field.

90 +

91 +## What's next

92 +

93 +- [Sankey and Table](/docs/network-flows/visualization/summary-sankey.md) — The default view; same aggregation, point-in-time.

94 +- [Retention and Querying](/docs/network-flows/retention-querying.md) — How tiers map to time ranges.

95 +- [Filters and Facets](/docs/network-flows/visualization/filters-facets.md) — Narrowing the data.

96 +- [Anti-patterns](/docs/network-flows/anti-patterns.md) — Why time-shifted comparison beats absolute thresholds.

integrations/categories.yaml

+20

@@ -63,6 +63,26 @@

63 name: Logs

64 description: "Monitoring logs on your infrastructure"

65 children: []

66 +- id: flows

67 + name: Network Flows

68 + description: "Network flow ingestion, enrichment, and routing-data sources for NetFlow / IPFIX / sFlow analysis"

69 + children:

70 + - id: flows.sources

71 + name: Sources

72 + description: "Flow protocols Netdata receives directly from routers, switches, and software exporters"

73 + children: []

74 + - id: flows.ip-intelligence

75 + name: IP Intelligence

76 + description: "GeoIP and ASN data sources used to enrich flow records with country, city, and AS-number labels"

77 + children: []

78 + - id: flows.bgp-routing

79 + name: BGP Routing

80 + description: "Live BGP route feeds used to enrich flow records with AS path, communities, and next-hop attributes"

81 + children: []

82 + - id: flows.network-identity

83 + name: Network Identity Sources

84 + description: "External feeds (cloud-provider IP ranges, IPAM systems, custom CMDBs) that label network prefixes with names, roles, sites, and tenants"

85 + children: []

86 - id: export

87 name: exporters

88 description: "Exporter Integrations"

integrations/gen_docs_integrations.py

+35

@@ -27,6 +27,7 @@ def cleanup(only_base_paths=None):

27 "src/go/plugin/scripts.d/collector",

28 "src/go/plugin/ibm.d/modules",

29 "src/crates/netdata-otel",

30 + "src/crates/netflow-plugin",

31 "src/collectors",

32 "src/exporting",

33 "integrations/cloud-notifications",

@@ -225,6 +226,33 @@ endmeta-->

226 if integration.get("troubleshooting"):

227 md += f"\n{integration['troubleshooting']}\n"

228

229 + elif mode == "flows":

230 + meta_yaml = integration["edit_link"].replace("blob", "edit")

231 + sidebar_label = integration["meta"]["monitored_instance"]["name"]

232 + learn_rel_path = generate_category_from_name(

233 + integration["meta"]["monitored_instance"]["categories"][0].split("."), categories

234 + )

235 + keywords = integration["meta"]["keywords"] if "keywords" in integration["meta"] else None

236 +

237 + md = f"""<!--startmeta

238 +meta_yaml: "{meta_yaml}"

239 +sidebar_label: "{sidebar_label}"

240 +learn_status: "Published"

241 +learn_rel_path: "{learn_rel_path}"

242 +"""

243 + if keywords:

244 + md += f"keywords: {keywords}\n"

245 +

246 + md += f"""message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

247 +endmeta-->

248 +

249 +{create_overview(integration, integration['meta']['monitored_instance']['icon_filename'])}"""

250 +

251 + if integration.get("setup"):

252 + md += f"\n{integration['setup']}\n"

253 + if integration.get("troubleshooting"):

254 + md += f"\n{integration['troubleshooting']}\n"

255 +

256 elif mode == "exporter":

257 meta_yaml = integration["edit_link"].replace("blob", "edit")

258 sidebar_label = integration["meta"]["name"]

@@ -618,6 +646,13 @@ def main():

646 path = build_path(meta_yaml)

647 write_to_file(path, md, meta_yaml, sidebar_label, community, integration_id=iid)

648

649 + elif itype == "flows" and not args.collector:

650 + meta_yaml, sidebar_label, learn_rel_path, md, community = build_readme_from_integration(

651 + integration, categories, mode="flows"

652 + )

653 + path = build_path(meta_yaml)

654 + write_to_file(path, md, meta_yaml, sidebar_label, community, integration_id=iid)

655 +

656 elif itype == "exporter" and not args.collector:

657 meta_yaml, sidebar_label, learn_rel_path, md, community = build_readme_from_integration(

658 integration, categories, mode="exporter"

integrations/gen_integrations.py

+105 -3

@@ -36,6 +36,10 @@ COLLECTOR_SOURCES = [

36 (AGENT_REPO, REPO_PATH / 'src' / 'crates' / 'netdata-otel', True),

37 ]

39 +FLOWS_SOURCES = [

40 + (AGENT_REPO, REPO_PATH / 'src' / 'crates' / 'netflow-plugin' / 'metadata.yaml', False),

41 +]

42 +

43 DEPLOY_SOURCES = [

44 (AGENT_REPO, INTEGRATIONS_PATH / 'deploy.yaml', False),

45 ]

@@ -78,6 +82,13 @@ COLLECTOR_RENDER_KEYS = [

82 'troubleshooting',

83 ]

85 +FLOWS_RENDER_KEYS = [

86 + 'overview',

87 + 'related_resources',

88 + 'setup',

89 + 'troubleshooting',

90 +]

91 +

92 EXPORTER_RENDER_KEYS = [

93 'overview',

94 'setup',

@@ -208,6 +219,11 @@ COLLECTOR_VALIDATOR = Draft7Validator(

219 registry=registry,

220 )

221

222 +FLOWS_VALIDATOR = Draft7Validator(

223 + {'$ref': './flows.json#'},

224 + registry=registry,

225 +)

226 +

227 SECRETSTORE_VALIDATOR = Draft7Validator(

228 {'$ref': './secretstore.json#'},

229 registry=registry,

@@ -313,7 +329,7 @@ def get_collector_metadata_entries():

329 ret.append((r, item))

330 elif d.exists() and d.is_file() and not m:

331 if d.match(METADATA_PATTERN):

316 - ret.append(d)

332 + ret.append((r, d))

333

334 return ret

335

@@ -388,6 +404,43 @@ def load_collectors():

404 return ret

405

406

407 +def load_flows():

408 + ret = []

409 +

410 + for repo, path, match in FLOWS_SOURCES:

411 + if match and path.exists() and path.is_dir():

412 + files = list(path.glob(METADATA_PATTERN))

413 + elif not match and path.exists() and path.is_file():

414 + files = [path]

415 + else:

416 + files = []

417 +

418 + for file in files:

419 + debug(f'Loading {file}.')

420 + data = load_yaml(file)

421 +

422 + if not data:

423 + continue

424 +

425 + try:

426 + FLOWS_VALIDATOR.validate(data)

427 + except ValidationError as e:

428 + warn(

429 + f'Failed to validate {file} against the schema: {e.message} (path: {"/".join(str(p) for p in e.absolute_path)})',

430 + file)

431 + continue

432 +

433 + for idx, item in enumerate(data['modules']):

434 + item['meta']['plugin_name'] = data['plugin_name']

435 + item['integration_type'] = 'flows'

436 + item['_src_path'] = file

437 + item['_repo'] = repo

438 + item['_index'] = idx

439 + ret.append(item)

440 +

441 + return ret

442 +

443 +

444 def _load_deploy_file(file, repo):

445 ret = []

446 debug(f'Loading {file}.')

@@ -1126,6 +1179,53 @@ def render_cloud_notifications(categories, notifications, ids):

1179 return notifications, clean_notifications, ids

1180

1181

1182 +def render_flows(categories, flows, ids):

1183 + debug('Generating flow IDs.')

1184 +

1185 + for item in flows:

1186 + item['id'] = make_id(item['meta'])

1187 +

1188 + debug('Sorting flows.')

1189 +

1190 + sort_integrations(flows)

1191 +

1192 + debug('Checking flow ids.')

1193 +

1194 + flows, ids = dedupe_integrations(flows, ids)

1195 +

1196 + clean_flows = []

1197 +

1198 + for item in flows:

1199 + item['edit_link'] = make_edit_link(item)

1200 +

1201 + clean_item = deepcopy(item)

1202 +

1203 + for key in FLOWS_RENDER_KEYS:

1204 + if key in item.keys():

1205 + template = get_jinja_env().get_template(get_section_template_name(item, key))

1206 + data = template.render(entry=item, clean=False)

1207 + clean_data = template.render(entry=item, clean=True)

1208 +

1209 + if 'variables' in item['meta']:

1210 + template = get_jinja_env().from_string(data)

1211 + data = template.render(variables=item['meta']['variables'], clean=False)

1212 + template = get_jinja_env().from_string(clean_data)

1213 + clean_data = template.render(variables=item['meta']['variables'], clean=True)

1214 + else:

1215 + data = ''

1216 + clean_data = ''

1217 +

1218 + item[key] = data

1219 + clean_item[key] = clean_data

1220 +

1221 + for k in ['_src_path', '_repo', '_index']:

1222 + del item[k], clean_item[k]

1223 +

1224 + clean_flows.append(clean_item)

1225 +

1226 + return flows, clean_flows, ids

1227 +

1228 +

1229 def render_logs(categories, logs, ids):

1230 debug('Sorting logs.')

1231

@@ -1334,6 +1434,7 @@ def main():

1434 agent_notifications = load_agent_notifications()

1435 cloud_notifications = load_cloud_notifications()

1436 logs = load_logs()

1437 + flows = load_flows()

1438 authentications = load_authentications()

1439 secretstores = load_secretstores()

1440 service_discoveries = load_service_discoveries()

@@ -1346,15 +1447,16 @@ def main():

1447 cloud_notifications, clean_cloud_notifications, ids = render_cloud_notifications(categories, cloud_notifications,

1448 ids)

1449 logs, clean_logs, ids = render_logs(categories, logs, ids)

1450 + flows, clean_flows, ids = render_flows(categories, flows, ids)

1451 authentications, clean_authentications, ids = render_authentications(categories, authentications, ids)

1452 secretstores, clean_secretstores, ids = render_secretstores(categories, secretstores, ids)

1453 service_discoveries, clean_service_discoveries, ids = render_service_discoveries(categories, service_discoveries,

1454 ids)

1455

1354 - integrations = collectors + deploy + exporters + agent_notifications + cloud_notifications + logs + authentications + secretstores + service_discoveries

1456 + integrations = collectors + deploy + exporters + agent_notifications + cloud_notifications + logs + flows + authentications + secretstores + service_discoveries

1457 render_integrations(categories, integrations)

1458

1357 - clean_integrations = clean_collectors + clean_deploy + clean_exporters + clean_agent_notifications + clean_cloud_notifications + clean_logs + clean_authentications + clean_secretstores + clean_service_discoveries

1459 + clean_integrations = clean_collectors + clean_deploy + clean_exporters + clean_agent_notifications + clean_cloud_notifications + clean_logs + clean_flows + clean_authentications + clean_secretstores + clean_service_discoveries

1460 render_json(categories, clean_integrations)

1461

1462 return fail_on_warnings()

integrations/schemas/flows.json new

@@ -0,0 +1,6 @@

1 +{

2 + "$schema": "http://json-schema.org/draft-07/schema#",

3 + "title": "Netdata Flows integrations metadata.",

4 + "description": "Flows integrations share the same shape as collector integrations: identity, overview, setup, troubleshooting. Every entry under integration_type=flows validates against this schema, which currently delegates to collector.json. Forking the schema is acceptable when flow-specific fields diverge.",

5 + "$ref": "./collector.json"

6 +}

integrations/templates/overview.md

@@ -10,6 +10,8 @@

10 [% include 'overview/authentication.md' %]

11 [% elif entry.integration_type == 'logs' %]

12 [% include 'overview/logs.md' %]

13 +[% elif entry.integration_type == 'flows' %]

14 +[% include 'overview/flows.md' %]

15 [% elif entry.integration_type == 'service_discovery' %]

16 [% include 'overview/service_discovery.md' %]

17 [% endif %]

integrations/templates/overview/flows.md new

+70

@@ -0,0 +1,70 @@

1 +# [[ entry.meta.monitored_instance.name ]]

2 +

3 +Plugin: [[ entry.meta.plugin_name ]]

4 +Module: [[ entry.meta.module_name ]]

5 +

6 +## Overview

7 +

8 +[[ entry.overview.data_collection.metrics_description ]]

9 +

10 +[[ entry.overview.data_collection.method_description ]]

11 +

12 +[% if entry.overview.supported_platforms.include %]

13 +This integration is only supported on the following platforms:

14 +

15 +[% for platform in entry.overview.supported_platforms.include %]

16 +- [[ platform ]]

17 +[% endfor %]

18 +[% elif entry.overview.supported_platforms.exclude %]

19 +This integration is supported on all platforms except for the following platforms:

20 +

21 +[% for platform in entry.overview.supported_platforms.exclude %]

22 +- [[ platform ]]

23 +[% endfor %]

24 +[% else %]

25 +This integration is supported on all platforms.

26 +[% endif %]

27 +

28 +[% if entry.overview.multi_instance %]

29 +This integration supports multiple instances configured side-by-side.

30 +[% else %]

31 +This integration runs as a single instance per Netdata Agent.

32 +[% endif %]

33 +

34 +[% if entry.overview.additional_permissions.description %]

35 +[[ entry.overview.additional_permissions.description ]]

36 +[% endif %]

37 +

38 +[% if related %]

39 +[[ entry.meta.monitored_instance.name ]] can be combined with the following other integrations:

40 +

41 +[% for res in related %]

42 +- {% relatedResource id="[[ res.id ]]" %}[[ res.name ]]{% /relatedResource %}

43 +[% endfor %]

44 +

45 +[% endif %]

46 +### Default Behavior

47 +

48 +#### Auto-Detection

49 +

50 +[% if entry.overview.default_behavior.auto_detection.description %]

51 +[[ entry.overview.default_behavior.auto_detection.description ]]

52 +[% else %]

53 +This integration doesn't support auto-detection.

54 +[% endif %]

55 +

56 +#### Limits

57 +

58 +[% if entry.overview.default_behavior.limits.description %]

59 +[[ entry.overview.default_behavior.limits.description ]]

60 +[% else %]

61 +The default configuration for this integration does not impose any limits.

62 +[% endif %]

63 +

64 +#### Performance Impact

65 +

66 +[% if entry.overview.default_behavior.performance_impact.description %]

67 +[[ entry.overview.default_behavior.performance_impact.description ]]

68 +[% else %]

69 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

70 +[% endif %]

src/collectors/COLLECTORS.md

@@ -263,6 +263,7 @@ Need a dedicated integration? [Submit a feature request](https://github.com/netd

263 | [Hitron CODA Cable Modem](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/collector/prometheus/integrations/hitron_coda_cable_modem.md) | Track Hitron CODA cable modem metrics for optimized internet connectivity and performance. |

264 | [InfiniBand](https://github.com/netdata/netdata/blob/master/src/collectors/proc.plugin/integrations/infiniband.md) | This integration monitors InfiniBand network inteface statistics. |

265 | [IP Virtual Server](https://github.com/netdata/netdata/blob/master/src/collectors/proc.plugin/integrations/ip_virtual_server.md) | This integration monitors IP Virtual Server statistics |

266 +| [IPFIX](https://github.com/netdata/netdata/blob/master/src/crates/netflow-plugin/integrations/ipfix.md) | Collects IPFIX (NetFlow v10) records from one or more exporters and stores them in tiered journal files. |

267 | [ipfw](https://github.com/netdata/netdata/blob/master/src/collectors/freebsd.plugin/integrations/ipfw.md) | Collect information about FreeBSD firewall. |

268 | [IPv6 Socket Statistics](https://github.com/netdata/netdata/blob/master/src/collectors/proc.plugin/integrations/ipv6_socket_statistics.md) | This integration provides IPv6 socket statistics. |

269 | [ISC DHCP](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/collector/isc_dhcpd/integrations/isc_dhcp.md) | This collector monitors ISC DHCP lease usage by reading the DHCP client lease database (dhcpd.leases). |

@@ -277,6 +278,7 @@ Need a dedicated integration? [Submit a feature request](https://github.com/netd

278 | [net.inet6.ip6.stats](https://github.com/netdata/netdata/blob/master/src/collectors/freebsd.plugin/integrations/net.inet6.ip6.stats.md) | Collect information abou IPv6 stats. |

279 | [net.isr](https://github.com/netdata/netdata/blob/master/src/collectors/freebsd.plugin/integrations/net.isr.md) | Collect information about system softnet stat. |

280 | [Netfilter](https://github.com/netdata/netdata/blob/master/src/collectors/nfacct.plugin/integrations/netfilter.md) | Monitor Netfilter metrics for optimal packet filtering and manipulation. |

281 +| [NetFlow](https://github.com/netdata/netdata/blob/master/src/crates/netflow-plugin/integrations/netflow.md) | Collects NetFlow v5, v7, and v9 records from one or more exporters (routers, switches, firewalls) and stores them in tiered journal files. |

282 | [Network Connections](https://github.com/netdata/netdata/blob/master/src/collectors/network-viewer.plugin/integrations/network_connections.md) | This plugin reads the system's socket tables to enumerate all active network connections, including TCP and UDP sockets in all states, for both IPv4 and IPv6. |

283 | [Network interfaces](https://github.com/netdata/netdata/blob/master/src/collectors/proc.plugin/integrations/network_interfaces.md) | Monitor network interface metrics about bandwidth, state, errors and more. |

284 | [Network statistics](https://github.com/netdata/netdata/blob/master/src/collectors/proc.plugin/integrations/network_statistics.md) | This integration provides metrics from the `netstat`, `snmp` and `snmp6` modules. |

@@ -294,6 +296,7 @@ Need a dedicated integration? [Submit a feature request](https://github.com/netd

296 | [PowerDNS Recursor](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/collector/powerdns_recursor/integrations/powerdns_recursor.md) | This collector monitors PowerDNS Recursor instances. |

297 | [RIPE Atlas](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/collector/prometheus/integrations/ripe_atlas.md) | Keep tabs on RIPE Atlas Internet measurement platform metrics for efficient network monitoring and performance. |

298 | [SCTP Statistics](https://github.com/netdata/netdata/blob/master/src/collectors/proc.plugin/integrations/sctp_statistics.md) | This integration provides statistics about the Stream Control Transmission Protocol (SCTP). |

299 +| [sFlow](https://github.com/netdata/netdata/blob/master/src/crates/netflow-plugin/integrations/sflow.md) | Collects sFlow v5 datagrams from one or more agents and stores them in tiered journal files. |

300 | [SNMP devices](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/collector/snmp/integrations/snmp_devices.md) | This collector discovers and monitors any SNMP-enabled network device. |

301 | [Socket statistics](https://github.com/netdata/netdata/blob/master/src/collectors/proc.plugin/integrations/socket_statistics.md) | This integration provides socket statistics. |

302 | [SoftEther VPN Server](https://github.com/netdata/netdata/blob/master/src/go/plugin/go.d/collector/prometheus/integrations/softether_vpn_server.md) | Monitor SoftEther VPN Server metrics for efficient virtual private network (VPN) management and performance. |

src/crates/netflow-plugin/README.md

+132 -53

@@ -308,62 +308,141 @@ To make a tier size-only, set `duration_of_journal_files: null`.

308

309 ## Performance benchmarking

310

311 -The plugin now ships manual ingestion benchmarks for both throughput ceilings

312 -and paced resource-envelope measurements:

313 -

314 -- `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release bench_ingestion_protocol_matrix -- --ignored --nocapture`

315 -- `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release bench_ingestion_cardinality_matrix -- --ignored --nocapture`

316 -- `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release ingest::resource_bench_tests::bench_resource_envelope_matrix -- --ignored --nocapture`

317 -

318 -The resource-envelope benchmark is intentionally explicit about its scope:

319 -

320 -- it measures the real ingest hot path after decode

321 -- it uses mixed flow records derived from shipped NetFlow/IPFIX/sFlow fixtures

322 -- it uses disk-backed journals under `src/crates/target/netflow-resource-bench`

323 -- it reports achieved flows/s, CPU utilization, peak/final RSS, and actual disk

324 - write throughput from `/proc/self/io`

325 -

326 -Reference measurements on this workstation:

311 +The plugin ships two complementary benchmarks:

312 +

313 +- `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release ingest::bench_tests::bench_ingestion_protocol_matrix -- --ignored --nocapture`

314 + unpaced full UDP→journal max throughput per protocol, plus decode-only and

315 + post-decode phases

316 +- `cargo test -p netflow-plugin --manifest-path src/crates/Cargo.toml --release ingest::resource_bench_tests::bench_resource_envelope_child -- --ignored --nocapture`

317 + paced post-decode resource envelope at a configurable rate, controlled via

318 + env vars: `NETFLOW_RESOURCE_BENCH_PROTOCOL`, `NETFLOW_RESOURCE_BENCH_PROFILE`,

319 + `NETFLOW_RESOURCE_BENCH_LAYER`, `NETFLOW_RESOURCE_BENCH_FLOWS_PER_SEC`,

320 + `NETFLOW_RESOURCE_BENCH_WARMUP_SECS`, `NETFLOW_RESOURCE_BENCH_MEASURE_SECS`

321 +

322 +The resource-envelope benchmark scope:

323 +

324 +- pre-decoded flow records pushed into the ingest pipeline (UDP receive and

325 + protocol decode are excluded; measure decode separately with the protocol

326 + matrix benchmark)

327 +- full pipeline active: raw journal + 1-minute + 5-minute + 1-hour tier

328 + accumulation, real disk-backed journals

329 +- enrichment is NOT loaded (no GeoIP MMDB, no static metadata, no classifiers,

330 + no static networks); cardinality fields are pre-populated by the harness

331 +- reports achieved flows/s, CPU% of one core, peak/final RSS, real disk read

332 + and write bytes/s from `/proc/self/io`

333 +

334 +`cpu_percent_of_one_core` is the sum of user+system ticks across all threads

335 +of the test process during the measurement window, divided by wall time, as a

336 +percent of one core. 100% means one core's worth of CPU was consumed; values

337 +above 100% are normal for multi-threaded saturation.

338 +

339 +Reference measurements:

340

341 - CPU: `12th Gen Intel(R) Core(TM) i9-12900K`

342 - storage: ext4 on `Seagate FireCuda 530`

330 -- benchmark methodology: release mode, `5s` warmup, `15s` measurement window,

331 - disk-backed journals, post-decode mixed-flow ingest

332 -

333 -Low-cardinality mixed profile (`record_pool_size=256`):

334 -

335 -- offered `5k flows/s`: achieved `5000`, CPU `80.0%` of one core, peak RSS

336 - `283.36 MiB`, write `26453 KiB/s`

337 -- offered `10k flows/s`: achieved `6187`, CPU `98.8%`, peak RSS `345.93 MiB`,

338 - write `35996 KiB/s`

339 -- offered `20k flows/s`: achieved `6305`, CPU `98.6%`, peak RSS `345.93 MiB`,

340 - write `32194 KiB/s`

341 -- offered `30k flows/s`: achieved `6183`, CPU `96.9%`, peak RSS `333.27 MiB`,

342 - write `31989 KiB/s`

343 -

344 -High-cardinality mixed profile (`record_pool_size=4096`):

345 -

346 -- offered `5k flows/s`: achieved `5000`, CPU `83.5%` of one core, peak RSS

347 - `380.39 MiB`, write `24571 KiB/s`

348 -- offered `10k flows/s`: achieved `5843`, CPU `97.5%`, peak RSS `404.55 MiB`,

349 - write `31394 KiB/s`

350 -- offered `20k flows/s`: achieved `5814`, CPU `96.6%`, peak RSS `403.82 MiB`,

351 - write `35888 KiB/s`

352 -- offered `30k flows/s`: achieved `5937`, CPU `98.0%`, peak RSS `410.14 MiB`,

353 - write `33189 KiB/s`

354 -

355 -Interpretation:

356 -

357 -- `5k flows/s` is sustainable on this host for both profiles with headroom left

358 - on one core

359 -- on this host the post-decode ingest path saturates one core around

360 - `5.8k - 6.3k flows/s`

361 -- higher field variability/cardinality mainly raises steady memory usage, not

362 - the one-core throughput ceiling

363 -- disk reads stay near zero in this benchmark because it isolates append-only

364 - ingest, not query/rebuild workloads

365 -- these numbers are host-specific and should be treated as a reference point,

366 - not as a universal guarantee

343 +- methodology: release mode, `5s` warmup, `15s` measurement window,

344 + disk-backed journals, post-decode paced ingest, all-tiers-batched layer

345 +

346 +### Paced post-decode resource envelope (3A)

347 +

348 +Per protocol, per cardinality, at 10 offered rates from 100 to 60 000 flows/s.

349 +Cardinality is synthetic: low-cardinality cycles 256 unique records, high-

350 +cardinality cycles 4 096 unique records. Real exporter data sits between the

351 +two.

352 +

353 +Low cardinality, NetFlow v9:

354 +

356 +|---:|---:|---:|---:|---:|

357 +| 100 | 80 | 0.3% | 95 KiB/s | 13 MiB |

358 +| 1 000 | 1 000 | 1.3% | 804 KiB/s | 23 MiB |

359 +| 10 000 | 10 000 | 12.6% | 7.7 MiB/s | 75 MiB |

360 +| 30 000 | 30 000 | 35.7% | 22.9 MiB/s | 83 MiB |

361 +| 60 000 | 60 000 | 70.3% | 45.6 MiB/s | 98 MiB |

362 +

363 +Low cardinality, IPFIX:

364 +

366 +|---:|---:|---:|---:|---:|

367 +| 100 | 61 | 0.1% | 75 KiB/s | 13 MiB |

368 +| 1 000 | 975 | 1.0% | 730 KiB/s | 24 MiB |

369 +| 10 000 | 9 996 | 11.5% | 6.8 MiB/s | 61 MiB |

370 +| 30 000 | 29 988 | 32.9% | 20.4 MiB/s | 83 MiB |

371 +| 60 000 | 59 977 | 64.1% | 40.8 MiB/s | 78 MiB |

372 +

373 +Low cardinality, sFlow:

374 +

376 +|---:|---:|---:|---:|---:|

377 +| 100 | 99 | 0.2% | 107 KiB/s | 13 MiB |

378 +| 1 000 | 985 | 1.5% | 847 KiB/s | 22 MiB |

379 +| 10 000 | 9 989 | 16.9% | 8.4 MiB/s | 75 MiB |

380 +| 30 000 | 29 967 | 46.2% | 25.2 MiB/s | 84 MiB |

381 +| 60 000 | 59 984 | 87.1% | 50.3 MiB/s | 80 MiB |

382 +

383 +High cardinality, NetFlow v9 (saturates around 30 000 flows/s):

384 +

386 +|---:|---:|---:|---:|---:|

387 +| 100 | 80 | 0.5% | 409 KiB/s | 25 MiB |

388 +| 1 000 | 1 000 | 4.3% | 2.0 MiB/s | 58 MiB |

389 +| 10 000 | 10 000 | 36.8% | 7.2 MiB/s | 104 MiB |

390 +| 30 000 | 29 331 | 98.0% | 24.9 MiB/s | 119 MiB |

391 +| 60 000 | 26 475 | 98.8% | 30.3 MiB/s | 247 MiB |

392 +

393 +High cardinality, IPFIX (saturates around 30-40 000 flows/s):

394 +

396 +|---:|---:|---:|---:|---:|

397 +| 100 | 60 | 0.2% | 214 KiB/s | 22 MiB |

398 +| 1 000 | 961 | 3.2% | 2.0 MiB/s | 61 MiB |

399 +| 10 000 | 9 970 | 28.0% | 7.7 MiB/s | 121 MiB |

400 +| 30 000 | 29 985 | 84.8% | 23.0 MiB/s | 121 MiB |

401 +| 60 000 | 28 835 | 98.5% | 36.8 MiB/s | 193 MiB |

402 +

403 +High cardinality, sFlow (saturates around 30 000 flows/s):

404 +

406 +|---:|---:|---:|---:|---:|

407 +| 100 | 100 | 0.6% | 604 KiB/s | 29 MiB |

408 +| 1 000 | 999 | 4.7% | 3.2 MiB/s | 77 MiB |

409 +| 10 000 | 9 990 | 35.3% | 9.9 MiB/s | 113 MiB |

410 +| 30 000 | 29 257 | 98.3% | 30.9 MiB/s | 129 MiB |

411 +| 60 000 | 30 227 | 98.6% | 29.1 MiB/s | 122 MiB |

412 +

413 +### Unpaced full UDP→journal protocol matrix (3B)

414 +

415 +Single-threaded peak throughput at native fixture cardinality:

416 +

418 +|---|---:|---:|---:|

422 +

423 +### Interpretation

424 +

425 +- The post-decode ingest hot path is currently single-threaded. CPU pins at

426 + ~98-99% of one core at saturation; it does not scale further with more cores.

427 +- Low-cardinality saturation is above 60 000 flows/s for NetFlow v9 and IPFIX,

428 + and around 70 000 flows/s for sFlow on this host. The matrix above does not

429 + reach those ceilings on purpose; extrapolate from the CPU% column.

430 +- High-cardinality saturation is around 30 000 flows/s post-decode for all

431 + three protocols. Above the knee, achieved rate stays at the plateau while

432 + offered rate grows.

433 +- Adding decode (~10 µs/flow) on top of post-decode ingest brings the practical

434 + full-path ceiling to roughly 22-25 000 flows/s at high cardinality on this

435 + host, with the four-tier pipeline running and no enrichment. UDP socket

436 + receive is not measured; at these flow rates packet rate (1-5k pps) is well

437 + below typical socket limits.

438 +- Disk reads stay near zero because the benchmark isolates the ingest path

439 + from query workloads. The journals themselves are indexed and rewrite pages

440 + during normal operation; this benchmark does not exercise that overhead at

441 + steady state because it only runs for the warmup + measurement window.

442 +- Higher cardinality raises steady memory and per-flow encoding cost, lowering

443 + the throughput ceiling.

444 +- These numbers are specific to this host. They do not include GeoIP/MMDB or

445 + any other enrichment; loading enrichment adds per-lookup CPU cost on top.

446

447 ## plugins.d protocol

448

src/crates/netflow-plugin/integrations/aws_ip_ranges.md new

+172

@@ -0,0 +1,172 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/aws_ip_ranges.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "AWS IP Ranges"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Network Identity Sources"

7 +keywords: ['aws', 'amazon', 'cloud', 'ip ranges', 'vpc', 'ec2', 'prefix list']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# AWS IP Ranges

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: aws-ip-ranges

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +AWS publishes a continuously updated JSON file listing every public IP prefix used

25 +by AWS services -- per region, per service. This integration fetches that file

26 +periodically, transforms it via a jq expression, and uses the result to label flow

27 +records destined to / from AWS with `*_NET_TENANT="amazon"` plus a per-region tag.

28 +

29 +The result: traffic to/from AWS shows up clearly in dashboards as "amazon", with

30 +per-region and per-service breakdown if you customize the jq transform.

31 +

32 +For the full network-identity concept (merge order, jq transform, TLS verification),

33 +see [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

34 +

35 +

36 +The plugin issues a periodic GET to `https://ip-ranges.amazonaws.com/ip-ranges.json`,

37 +parses the JSON body, runs the configured jq transform via the [jaq](https://github.com/01mf02/jaq)

38 +library, and merges the resulting prefix-labeled rows into the network-attributes trie.

39 +

40 +

41 +This integration is only supported on the following platforms:

42 +

43 +- Linux

44 +

45 +This integration supports multiple instances configured side-by-side.

46 +

47 +

48 +### Default Behavior

49 +

50 +#### Auto-Detection

51 +

52 +Disabled by default. Add an entry under enrichment.network_sources to enable.

53 +

54 +#### Limits

55 +

56 +The default configuration for this integration does not impose any limits.

57 +

58 +#### Performance Impact

59 +

60 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

61 +

62 +## Setup

63 +

64 +

65 +### Prerequisites

66 +

67 +#### Outbound HTTPS to AWS

68 +

69 +The agent host must be able to reach `https://ip-ranges.amazonaws.com/ip-ranges.json`.

70 +No AWS credentials needed -- the file is public.

71 +

72 +

73 +

74 +### Configuration

75 +

76 +#### Options

77 +

78 +Add a named entry under `enrichment.network_sources`. The `name` you choose appears

79 +in flow records via the `*_NET_TENANT` field (when your jq transform sets it).

80 +

81 +

82 +<details open><summary>Config options</summary>

83 +

84 +

85 +

87 +|:-----|:------------|:--------|:---------:|

88 +| url | AWS publishes the master file at this URL. | https://ip-ranges.amazonaws.com/ip-ranges.json | yes |

90 +| timeout | Per-request timeout. | 60s | no |

91 +| transform | jq expression that converts the AWS response into objects with `prefix` and label fields. | . | yes |

92 +

93 +

94 +</details>

95 +

96 +

97 +

98 +#### via File

99 +

100 +The configuration file name for this integration is `netflow.yaml`.

101 +

102 +

103 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

104 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

105 +

106 +```bash

107 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

108 +sudo ./edit-config netflow.yaml

109 +```

110 +

111 +##### Examples

112 +

113 +###### Tag all AWS prefixes by region and service

114 +

115 +Sets tenant=amazon, region=<aws-region>, role=<service-name>.

116 +

117 +```yaml

118 +enrichment:

119 + network_sources:

120 + aws:

121 + url: "https://ip-ranges.amazonaws.com/ip-ranges.json"

122 + interval: 24h

123 + timeout: 60s

124 + transform: |

125 + (.prefixes + .ipv6_prefixes)[] | {

126 + prefix: (.ip_prefix // .ipv6_prefix),

127 + tenant: "amazon",

128 + region: .region,

129 + role: (.service | ascii_downcase)

130 + }

131 +

132 +```

133 +###### AWS S3 only

134 +

135 +Filter to a single AWS service for narrower tagging.

136 +

137 +<details open><summary>Config</summary>

138 +

139 +```yaml

140 +enrichment:

141 + network_sources:

142 + aws-s3:

143 + url: "https://ip-ranges.amazonaws.com/ip-ranges.json"

144 + interval: 24h

145 + transform: |

146 + (.prefixes + .ipv6_prefixes)[]

147 + | select(.service == "S3")

148 + | {

149 + prefix: (.ip_prefix // .ipv6_prefix),

150 + tenant: "amazon",

151 + role: "s3",

152 + region: .region

153 + }

154 +

155 +```

156 +</details>

157 +

158 +

159 +

160 +### Empty result from the transform is treated as failure

161 +

162 +If the jq filter happens to produce nothing (e.g., AWS responds with no prefixes),

163 +the source backs off as if it errored. Check the journal for `network-sources` warnings.

164 +

165 +

166 +### TLS verification cannot be disabled

167 +

168 +`tls.skip_verify: true` is rejected by validation. Use `tls.ca_file` for

169 +custom-CA paths if needed.

170 +

171 +

172 +

src/crates/netflow-plugin/integrations/azure_ip_ranges.md new

+155

@@ -0,0 +1,155 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/azure_ip_ranges.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "Azure IP Ranges"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Network Identity Sources"

7 +keywords: ['azure', 'microsoft', 'cloud', 'ip ranges', 'service tags']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# Azure IP Ranges

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: azure-ip-ranges

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +Azure publishes "Service Tags" data describing IP ranges per region and per service.

25 +The Azure publication mechanism is **less convenient** than AWS / GCP -- the

26 +authoritative URL contains a date stamp that changes weekly, so you cannot use

27 +a single stable URL.

28 +

29 +For automated fetching, you have two options:

30 +1. Mirror the file in your own infrastructure (a script that resolves the latest

31 + URL via the Azure CLI, downloads, and serves at a stable internal URL).

32 +2. Skip Azure IP Ranges entirely and rely on GeoIP / ASN data for Azure

33 + attribution (Azure ASN is 8075).

34 +

35 +For the full network-identity concept, see

36 +[Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

37 +

38 +

39 +Periodic HTTPS GET against your stable mirror URL, jq transform, merge into

40 +network-attributes trie. The plugin does not handle Azure's date-stamped URL

41 +rotation -- you provide a stable URL via your own mirror.

42 +

43 +

44 +This integration is only supported on the following platforms:

45 +

46 +- Linux

47 +

48 +This integration supports multiple instances configured side-by-side.

49 +

50 +

51 +### Default Behavior

52 +

53 +#### Auto-Detection

54 +

55 +Disabled by default. Set up your own URL mirror, then add an entry under enrichment.network_sources.

56 +

57 +#### Limits

58 +

59 +The default configuration for this integration does not impose any limits.

60 +

61 +#### Performance Impact

62 +

63 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

64 +

65 +## Setup

66 +

67 +

68 +### Prerequisites

69 +

70 +#### A stable URL for the Azure Service Tags JSON

71 +

72 +Azure's authoritative URL embeds a date stamp that changes weekly. A simple

73 +workaround: a cron job that calls

74 +`az network list-service-tags --location global -o json` (Azure CLI) and

75 +writes the result to a stable path on an internal HTTP server. The plugin then

76 +fetches from that stable URL.

77 +

78 +

79 +#### Outbound HTTPS to your mirror

80 +

81 +No Azure credentials needed by the plugin itself; credentials only matter on

82 +the side that does the upstream Azure CLI call.

83 +

84 +

85 +

86 +### Configuration

87 +

88 +#### Options

89 +

90 +Add a named entry under `enrichment.network_sources` pointing at your mirror URL.

91 +

92 +

93 +<details open><summary>Config options</summary>

94 +

95 +

96 +

98 +|:-----|:------------|:--------|:---------:|

99 +| url | Stable URL to your locally-mirrored Azure Service Tags JSON. | | yes |

100 +| transform | jq expression mapping the values[] array to per-prefix objects. | . | yes |

101 +

102 +

103 +</details>

104 +

105 +

106 +

107 +#### via File

108 +

109 +The configuration file name for this integration is `netflow.yaml`.

110 +

111 +

112 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

113 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

114 +

115 +```bash

116 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

117 +sudo ./edit-config netflow.yaml

118 +```

119 +

120 +##### Examples

121 +

122 +###### Internal mirror of Azure Service Tags

123 +

124 +Tag every Azure prefix with tenant=azure plus region/service.

125 +

126 +```yaml

127 +enrichment:

128 + network_sources:

129 + azure:

130 + url: "https://internal.example/azure-service-tags.json"

131 + interval: 24h

132 + transform: |

133 + .values[]

134 + | .id as $id

135 + | .properties.region as $region

136 + | (.properties.systemService // "") as $service

137 + | .properties.addressPrefixes[]

138 + | {

139 + prefix: .,

140 + tenant: "azure",

141 + region: ($region // ""),

142 + role: ($service | ascii_downcase)

143 + }

144 +

145 +```

146 +

147 +

148 +### Empty results

149 +

150 +The Azure Service Tags JSON has nested structure (`values[].properties.addressPrefixes[]`).

151 +If your jq doesn't unwrap correctly, every fetch yields zero rows and the source

152 +backs off. Test the jq locally with `jq < azure-service-tags.json`.

153 +

154 +

155 +

src/crates/netflow-plugin/integrations/bio-rd_-_ripe_ris.md new

+196

@@ -0,0 +1,196 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/bio-rd_-_ripe_ris.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "bio-rd / RIPE RIS"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/BGP Routing"

7 +keywords: ['bioris', 'bio-rd', 'ripe ris', 'bgp', 'grpc', 'route information service']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# bio-rd / RIPE RIS

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: bioris

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +BioRIS lets Netdata consume BGP routing data from a [bio-rd](https://github.com/bio-routing/bio-rd)

25 +`cmd/ris/` daemon over gRPC. bio-rd is a Go-based BGP daemon that can peer with

26 +[RIPE RIS](https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris)

27 +Route Collectors -- or any BGP / BMP source you have access to -- and expose the

28 +resulting RIB through a gRPC interface. Netdata is a client of that interface.

29 +

30 +Use this when you want a third-party view of the BGP routing table (e.g., RIPE

31 +RIS's view) without running a BGP session yourself or deploying BMP across your

32 +network.

33 +

34 +For the full BGP-routing concept and how BMP and BioRIS share the same trie, see

35 +[BGP Routing](https://learn.netdata.cloud/docs/network-flows/enrichment/bgp-routing).

36 +

37 +

38 +The plugin connects to one or more bio-rd `ris` gRPC endpoints. It runs three RPCs:

39 +`GetRouters` to discover what's available, `DumpRIB` to do baseline reconciliation,

40 +and `ObserveRIB` for incremental updates. Multiple instances are additive (not

41 +failover); they all merge into the shared in-memory trie.

42 +

43 +

44 +This integration is only supported on the following platforms:

45 +

46 +- Linux

47 +

48 +This integration runs as a single instance per Netdata Agent.

49 +

50 +

51 +### Default Behavior

52 +

53 +#### Auto-Detection

54 +

55 +Disabled by default. Set enrichment.routing_dynamic.bioris.enabled to true and provide at least one ris_instances entry.

56 +

57 +#### Limits

58 +

59 +The default configuration for this integration does not impose any limits.

60 +

61 +#### Performance Impact

62 +

63 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

64 +

65 +## Setup

66 +

67 +

68 +### Prerequisites

69 +

70 +#### A running bio-rd 'ris' daemon

71 +

72 +bio-rd is a separate project. The plugin only consumes its gRPC interface; it

73 +does not bundle bio-rd. You install it yourself:

74 +

75 +```bash

76 +# Install Go (>=1.20), then:

77 +git clone https://github.com/bio-routing/bio-rd.git

78 +cd bio-rd/cmd/ris

79 +go build -o /usr/local/bin/ris .

80 +```

81 +

82 +Configure `ris` to peer with one or more BGP / BMP sources (RIPE RIS Route

83 +Collectors, your own peers, etc.). Refer to the bio-rd documentation for the

84 +peering setup -- this is bio-rd's configuration, not Netdata's.

85 +

86 +Run the daemon with a gRPC port:

87 +`/usr/local/bin/ris --grpc_port 50051 --config.file /etc/bio-rd.yml`

88 +

89 +

90 +#### Network reachability + no auth

91 +

92 +The gRPC connection is plain HTTP/2 by default (or TLS with system-CA when

93 +`grpc_secure: true`). There is no authentication. Restrict access at the

94 +firewall, or run bio-rd on the same host as the agent and bind it to localhost.

95 +

96 +

97 +

98 +### Configuration

99 +

100 +#### Options

101 +

102 +BioRIS options live under `enrichment.routing_dynamic.bioris`.

103 +

104 +

105 +<details open><summary>Config options</summary>

106 +

107 +

108 +

110 +|:-----|:------------|:--------|:---------:|

112 +| timeout | Connect + per-RPC timeout. Default is aggressive for public RIS over the internet -- raise if you see "deadline exceeded". | 200ms | no |

113 +| refresh | How often to re-dump every router's RIB from scratch. | 30m | no |

114 +| refresh_timeout | Per-DumpRIB request timeout and per-message stream timeout. | 10s | no |

115 +| ris_instances | List of bio-rd endpoints. Each: grpc_addr, grpc_secure, vrf, vrf_id. Multiple instances are additive (not failover) -- routes from all merge. | [] | yes |

116 +

117 +

118 +</details>

119 +

120 +

121 +

122 +#### via File

123 +

124 +The configuration file name for this integration is `netflow.yaml`.

125 +

126 +

127 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

128 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

129 +

130 +```bash

131 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

132 +sudo ./edit-config netflow.yaml

133 +```

134 +

135 +##### Examples

136 +

137 +###### Local bio-rd

138 +

139 +bio-rd running on the same host, plain gRPC.

140 +

141 +```yaml

142 +enrichment:

143 + routing_dynamic:

144 + bioris:

145 + enabled: true

146 + timeout: 2s

147 + refresh: 30m

148 + refresh_timeout: 30s

149 + ris_instances:

150 + - grpc_addr: "127.0.0.1:50051"

151 + grpc_secure: false

152 +

153 +```

154 +###### Remote bio-rd over TLS

155 +

156 +Across a network, system CA bundle. No client cert / mTLS.

157 +

158 +<details open><summary>Config</summary>

159 +

160 +```yaml

161 +enrichment:

162 + routing_dynamic:

163 + bioris:

164 + enabled: true

165 + timeout: 5s

166 + ris_instances:

167 + - grpc_addr: "ris.example.internal:50051"

168 + grpc_secure: true

169 + vrf: "global"

170 +

171 +```

172 +</details>

173 +

174 +

175 +

176 +### Default 200ms timeout too aggressive

177 +

178 +Over the public internet to RIPE RIS, you may need 2-5 seconds. If you see

179 +"deadline exceeded" errors in the journal, raise `timeout`.

180 +

181 +

182 +### Initial dump takes minutes for full feeds

183 +

184 +A full IPv4+IPv6 RIB from a route collector is millions of prefixes. The first

185 +refresh takes time; subsequent observe streams are incremental.

186 +

187 +

188 +### Integration-test gap

189 +

190 +proto and route conversion are unit-tested. The gRPC client path

191 +(connecting, consuming streams, retry/backoff) is NOT integration-tested.

192 +Validate against your specific bio-rd setup before relying on this for

193 +capacity / security decisions.

194 +

195 +

196 +

src/crates/netflow-plugin/integrations/bmp_bgp_monitoring_protocol.md new

+236

@@ -0,0 +1,236 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/bmp_bgp_monitoring_protocol.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "BMP (BGP Monitoring Protocol)"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/BGP Routing"

7 +keywords: ['bmp', 'bgp', 'rfc 7854', 'route monitoring', 'cisco', 'juniper', 'frr']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# BMP (BGP Monitoring Protocol)

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: bmp

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +BMP (BGP Monitoring Protocol, RFC 7854) lets a router push its BGP route updates

25 +to a passive collector. With this integration enabled, Netdata acts as that

26 +collector -- it listens for BMP TCP connections from your routers, parses the BGP

27 +UPDATE messages, and builds an in-memory routing table that flow enrichment then

28 +reads from.

29 +

30 +The result: every flow gets accurate AS numbers, AS paths, communities, and

31 +next-hop information from your real-time BGP table -- not from a stale GeoIP

32 +database or from whatever the exporter happened to send in the flow record.

33 +

34 +For the full BGP-routing concept (shared trie with BioRIS, withdrawal handling,

35 +per-vendor caveats, integration test gap), see

36 +[BGP Routing](https://learn.netdata.cloud/docs/network-flows/enrichment/bgp-routing).

37 +

38 +

39 +The plugin runs a TCP listener on `0.0.0.0:10179` (Akvorado convention -- not the

40 +IANA-registered port 7854). Routers initiate BMP sessions to it. The plugin

41 +processes Initiation, Termination, RouteMonitoring (BGP UPDATE messages), and

42 +PeerDownNotification messages. NLRI types: IPv4/IPv6 unicast, MPLS-labelled, VPNv4,

43 +VPNv6, EVPN IP-prefix.

44 +

45 +BMP and BioRIS share a single in-memory routing trie. Memory grows with the size

46 +of the BGP table; a full IPv4+IPv6 feed is roughly 1.2M prefixes per peer.

47 +

48 +

49 +This integration is only supported on the following platforms:

50 +

51 +- Linux

52 +

53 +This integration runs as a single instance per Netdata Agent.

54 +

55 +

56 +### Default Behavior

57 +

58 +#### Auto-Detection

59 +

60 +Disabled by default. Set enrichment.routing_dynamic.bmp.enabled to true and configure your routers.

61 +

62 +#### Limits

63 +

64 +The default configuration for this integration does not impose any limits.

65 +

66 +#### Performance Impact

67 +

68 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

69 +

70 +## Setup

71 +

72 +

73 +### Prerequisites

74 +

75 +#### BMP-capable routers

76 +

77 +Modern Cisco IOS-XR, Juniper JunOS, Arista EOS, and FRR all support BMP v3.

78 +The plugin parses RFC 7854 BMP v3 specifically. Older versions (v1, v2) are

79 +not supported and will fail to parse.

80 +

81 +

82 +#### TCP reachability between routers and the agent

83 +

84 +Routers initiate the connection -- the plugin is a passive listener. Allow

85 +inbound TCP on the configured port (default 10179) from each BMP-speaking

86 +router to the agent.

87 +

88 +

89 +#### No TLS, no authentication

90 +

91 +The listener accepts plain TCP only. Restrict access at the firewall and on a

92 +dedicated management network -- do not expose 10179 to the public internet.

93 +

94 +

95 +

96 +### Configuration

97 +

98 +#### Options

99 +

100 +All BMP options live under `enrichment.routing_dynamic.bmp` in `netflow.yaml`.

101 +

102 +

103 +<details open><summary>Config options</summary>

104 +

105 +

106 +

108 +|:-----|:------------|:--------|:---------:|

110 +| listen | TCP bind address (host:port). | 0.0.0.0:10179 | no |

111 +| keep | Grace window after a BMP disconnect before purging that session's routes. | 5m | no |

112 +| max_consecutive_decode_errors | Close the session after N consecutive decode errors. | 8 | no |

113 +| receive_buffer | Optional SO_RCVBUF per connection in bytes (0 = OS default). | 0 | no |

117 +| rds | Whitelist of accepted Route Distinguishers. Empty = accept all. Formats: "0", "ASN:idx", "IPv4:idx", or full text. | [] | no |

118 +

119 +

120 +</details>

121 +

122 +

123 +

124 +#### via File

125 +

126 +The configuration file name for this integration is `netflow.yaml`.

127 +

128 +

129 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

130 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

131 +

132 +```bash

133 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

134 +sudo ./edit-config netflow.yaml

135 +```

136 +

137 +##### Examples

138 +

139 +###### Enable BMP listener

140 +

141 +Start the listener on the default port.

142 +

143 +```yaml

144 +enrichment:

145 + routing_dynamic:

146 + bmp:

147 + enabled: true

148 + listen: "0.0.0.0:10179"

149 + keep: 5m

150 +

151 +```

152 +###### Cisco IOS-XR router config

153 +

154 +Vendor-side config to send BMP to Netdata. The bmp server block is global, not under router bgp.

155 +

156 +<details open><summary>Config</summary>

157 +

158 +```yaml

159 +bmp server 1

160 + host 10.0.0.10 port 10179

161 + description "Netdata BMP collector"

162 + initial-delay 5

163 + stats-reporting-period 60

164 + initial-refresh delay 30 spread 2

165 +!

166 +router bgp 65000

167 + neighbor 192.0.2.1

168 + bmp-activate server 1

169 +

170 +```

171 +</details>

172 +

173 +###### Juniper JunOS router config

174 +

175 +Recommended local-address and statistics-timeout for production.

176 +

177 +<details open><summary>Config</summary>

178 +

179 +```yaml

180 +set routing-options bmp station netdata station-address 10.0.0.10

181 +set routing-options bmp station netdata station-port 10179

182 +set routing-options bmp station netdata connection-mode active

183 +set routing-options bmp station netdata local-address 10.0.0.1

184 +set routing-options bmp station netdata statistics-timeout 60

185 +set routing-options bmp station netdata route-monitoring pre-policy

186 +

187 +```

188 +</details>

189 +

190 +###### FRR (bgpd) router config

191 +

192 +Critical -- BMP is a runtime module in FRR. Without "-M bmp" in

193 +/etc/frr/daemons (bgpd_options), every BMP command silently fails.

194 +

195 +

196 +<details open><summary>Config</summary>

197 +

198 +```yaml

199 +# /etc/frr/daemons:

200 +# bgpd_options=" -A 127.0.0.1 -M bmp"

201 +router bgp 65000

202 + bmp targets netdata

203 + bmp connect 10.0.0.10 port 10179 min-retry 5000 max-retry 60000

204 + bmp stats interval 60000

205 + bmp monitor ipv4 unicast pre-policy

206 + bmp monitor ipv6 unicast pre-policy

207 + exit

208 +

209 +```

210 +</details>

211 +

212 +

213 +

214 +### Listener not receiving BMP sessions

215 +

216 +Check `show bmp` (Cisco) / `show bmp connections` (Juniper) / `show bmp targets` (FRR)

217 +to confirm the router has dialed in. The plugin does not initiate -- it listens.

218 +Firewall: allow inbound TCP on 10179.

219 +

220 +

221 +### Memory growth

222 +

223 +A full BGP feed adds ~1.2M prefixes per peer permanently. There is no time-based

224 +eviction. Plan capacity accordingly. After a router disconnect, the routes

225 +for that session are kept for `keep` (default 5 min) before purging.

226 +

227 +

228 +### Integration-test gap

229 +

230 +BMP message parsing has unit tests. The TCP listener path, framed decode loop,

231 +trie apply, and per-router cleanup are NOT integration-tested. Validate against

232 +your specific router firmware before depending on this for capacity / security

233 +decisions.

234 +

235 +

236 +

src/crates/netflow-plugin/integrations/custom_mmdb_database.md new

+162

@@ -0,0 +1,162 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/custom_mmdb_database.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "Custom MMDB Database"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/IP Intelligence"

7 +keywords: ['mmdb', 'custom database', 'bring your own', 'ipinfo', 'ip intelligence']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# Custom MMDB Database

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: custom-mmdb

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +The plugin reads any MMDB file that conforms to the standard schema -- this catch-all

25 +integration covers IPInfo, custom-built internal MMDBs, vendor-specific feeds, or

26 +any provider that publishes MMDB data.

27 +

28 +The plugin reads `country.iso_code`, `city.names.en`, `subdivisions[].iso_code`,

29 +`location.latitude`, `location.longitude`, `autonomous_system_number`, and

30 +`autonomous_system_organization`. Vendor-specific extra fields are ignored.

31 +

32 +For the full IP-intelligence concept, see

33 +[IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

34 +

35 +

36 +You produce or download an MMDB file. Place it on the agent host. Point the

37 +plugin at it via `netflow.yaml`. The plugin reloads on file change every 30

38 +seconds.

39 +

40 +

41 +This integration is only supported on the following platforms:

42 +

43 +- Linux

44 +

45 +This integration runs as a single instance per Netdata Agent.

46 +

47 +

48 +### Default Behavior

49 +

50 +#### Auto-Detection

51 +

52 +Not auto-detected. You must configure paths explicitly.

53 +

54 +#### Limits

55 +

56 +The default configuration for this integration does not impose any limits.

57 +

58 +#### Performance Impact

59 +

60 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

61 +

62 +## Setup

63 +

64 +

65 +### Prerequisites

66 +

67 +#### A standards-compliant MMDB file

68 +

69 +The MMDB file must use the [standard MMDB schema](https://maxmind.github.io/MaxMind-DB/).

70 +Validate with `mmdblookup` from the `libmaxminddb-tools` package before deploying.

71 +

72 +Common sources: IPInfo (`ipinfo.io`), custom internal builds via the `mmdbwriter`

73 +Go tool, or vendor-specific feeds.

74 +

75 +

76 +

77 +### Configuration

78 +

79 +#### Options

80 +

81 +Point `enrichment.geoip.asn_database` and/or `enrichment.geoip.geo_database` at

82 +your MMDB file paths.

83 +

84 +

85 +<details open><summary>Config options</summary>

86 +

87 +

88 +

90 +|:-----|:------------|:--------|:---------:|

91 +| enrichment.geoip.asn_database | List of MMDB paths providing AS data. Multiple files allowed; later entries override on overlap. | [] | no |

92 +| enrichment.geoip.geo_database | List of MMDB paths providing geographic data. | [] | no |

94 +

95 +

96 +</details>

97 +

98 +

99 +

100 +#### via File

101 +

102 +The configuration file name for this integration is `netflow.yaml`.

103 +

104 +

105 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

106 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

107 +

108 +```bash

109 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

110 +sudo ./edit-config netflow.yaml

111 +```

112 +

113 +##### Examples

114 +

115 +###### IPInfo MMDB

116 +

117 +Using IPInfo's MMDB feed (subscription required).

118 +

119 +```yaml

120 +enrichment:

121 + geoip:

122 + asn_database:

123 + - /opt/mmdb/ipinfo-asn.mmdb

124 + geo_database:

125 + - /opt/mmdb/ipinfo-city.mmdb

126 + optional: false

127 +

128 +```

129 +###### Internal custom MMDB

130 +

131 +Built in-house with `mmdbwriter`. Combines public BGP data with internal CIDR labels.

132 +

133 +<details open><summary>Config</summary>

134 +

135 +```yaml

136 +enrichment:

137 + geoip:

138 + asn_database:

139 + - /etc/netdata/internal-asn.mmdb

140 + geo_database:

141 + - /etc/netdata/internal-geo.mmdb

142 + optional: false

143 +

144 +```

145 +</details>

146 +

147 +

148 +

149 +### Lookups silently return empty

150 +

151 +The MMDB schema is non-standard or the IP types don't match (some custom builds

152 +use `string` instead of `array` for ASN). Validate with `mmdblookup -f file.mmdb -i 8.8.8.8`

153 +and confirm the standard fields are present.

154 +

155 +

156 +### Plugin fails to start with optional=false

157 +

158 +File missing or unreadable at the configured path. Check permissions; the netdata

159 +user must be able to read the file.

160 +

161 +

162 +

src/crates/netflow-plugin/integrations/db-ip_ip_intelligence.md new

+169

@@ -0,0 +1,169 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/db-ip_ip_intelligence.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "DB-IP IP Intelligence"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/IP Intelligence"

7 +keywords: ['geoip', 'asn', 'dbip', 'db-ip', 'mmdb', 'ip intelligence', 'flow enrichment']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# DB-IP IP Intelligence

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: dbip

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +DB-IP is the **default** IP intelligence source for the Netdata netflow plugin. Its

25 +MMDB-format databases are bundled with native packages (DEB, RPM) under

26 +`/usr/share/netdata/topology-ip-intel/`. Refreshing pulls newer data from

27 +`download.db-ip.com` via the bundled `topology-ip-intel-downloader`.

28 +

29 +Populates `SRC_COUNTRY`, `DST_COUNTRY`, `SRC_GEO_STATE`, `DST_GEO_STATE`,

30 +`SRC_GEO_CITY`, `DST_GEO_CITY`, `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE`,

31 +`SRC_GEO_LONGITUDE`, `DST_GEO_LONGITUDE`, plus the AS-number and AS-name fields

32 +when included in the resolution chain.

33 +

34 +For the full IP-intelligence concept (MMDB format, lookup priority, internal-IP

35 +handling, hot reload semantics), see

36 +[IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

37 +

38 +

39 +Files are read on plugin start and reloaded automatically every 30 seconds when

40 +their mtime or size changes. Lookups happen in-process; there is no per-flow network

41 +call. Auto-detection scans `${NETDATA_CACHE_DIR}/topology-ip-intel/` first, falling

42 +back to the stock copy under `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/`.

43 +

44 +

45 +This integration is only supported on the following platforms:

46 +

47 +- Linux

48 +

49 +This integration runs as a single instance per Netdata Agent.

50 +

51 +

52 +### Default Behavior

53 +

54 +#### Auto-Detection

55 +

56 +Native packages ship the stock DB-IP MMDB files; the plugin auto-detects them at startup. No configuration required for the default install.

57 +

58 +#### Limits

59 +

60 +The default configuration for this integration does not impose any limits.

61 +

62 +#### Performance Impact

63 +

64 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

65 +

66 +## Setup

67 +

68 +

69 +### Prerequisites

70 +

71 +#### DB-IP MMDB files

72 +

73 +Ships with native packages. For source builds, run the bundled downloader once

74 +to populate `/var/cache/netdata/topology-ip-intel/`:

75 +

76 +```bash

77 +sudo /usr/sbin/topology-ip-intel-downloader

78 +```

79 +

80 +Subsequent refreshes (e.g., monthly cron) re-fetch from db-ip.com.

81 +

82 +

83 +

84 +### Configuration

85 +

86 +#### Options

87 +

88 +Configure DB-IP under `enrichment.geoip` in `netflow.yaml`. Empty `asn_database`

89 +and `geo_database` enable auto-detection.

90 +

91 +

92 +<details open><summary>Config options</summary>

93 +

94 +

95 +

97 +|:-----|:------------|:--------|:---------:|

101 +

102 +

103 +</details>

104 +

105 +

106 +

107 +#### via File

108 +

109 +The configuration file name for this integration is `netflow.yaml`.

110 +

111 +

112 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

113 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

114 +

115 +```bash

116 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

117 +sudo ./edit-config netflow.yaml

118 +```

119 +

120 +##### Examples

121 +

122 +###### Default (auto-detect stock files)

123 +

124 +Native package install. No explicit configuration; the plugin finds the stock or cache copy automatically.

125 +

126 +```yaml

127 +enrichment:

128 + geoip:

129 + asn_database: []

130 + geo_database: []

131 + optional: true

132 +

133 +```

134 +###### Explicit DB-IP paths

135 +

136 +Override auto-detection by pointing to specific DB-IP MMDBs (for example, after running the downloader to a non-standard location).

137 +

138 +<details open><summary>Config</summary>

139 +

140 +```yaml

141 +enrichment:

142 + geoip:

143 + asn_database:

144 + - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb

145 + geo_database:

146 + - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb

147 + optional: false

148 +

149 +```

150 +</details>

151 +

152 +

153 +

154 +### Internal IPs appearing in random countries

155 +

156 +GeoIP databases have no entry for RFC 1918 / private space. The stock DB-IP

157 +build tags private ranges so `*_AS_NAME` renders as "AS0 Private IP Address Space"

158 +with empty country. With third-party MMDBs, results may vary. Declare your

159 +internal CIDRs under `enrichment.networks` to override -- see

160 +[Static metadata](https://learn.netdata.cloud/docs/network-flows/enrichment/static-metadata).

161 +

162 +

163 +### Stale databases

164 +

165 +The plugin does not alert on staleness. Check file mtime: `ls -la /var/cache/netdata/topology-ip-intel/`.

166 +Schedule a weekly cron of `topology-ip-intel-downloader` to keep data fresh.

167 +

168 +

169 +

src/crates/netflow-plugin/integrations/gcp_ip_ranges.md new

+133

@@ -0,0 +1,133 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/gcp_ip_ranges.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "GCP IP Ranges"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Network Identity Sources"

7 +keywords: ['gcp', 'google cloud', 'cloud', 'ip ranges', 'prefix list']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# GCP IP Ranges

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: gcp-ip-ranges

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +Google Cloud publishes its public IP prefixes at `https://www.gstatic.com/ipranges/cloud.json`,

25 +updated periodically. This integration fetches the file and labels flow records

26 +to/from Google Cloud with `*_NET_TENANT="gcp"` plus per-scope and per-service tags.

27 +

28 +For the full network-identity concept, see

29 +[Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

30 +

31 +

32 +Periodic HTTPS GET, jq transform, merge into network-attributes trie. Same mechanism

33 +as AWS IP Ranges, different URL and JSON shape.

34 +

35 +

36 +This integration is only supported on the following platforms:

37 +

38 +- Linux

39 +

40 +This integration supports multiple instances configured side-by-side.

41 +

42 +

43 +### Default Behavior

44 +

45 +#### Auto-Detection

46 +

47 +Disabled by default. Add an entry under enrichment.network_sources to enable.

48 +

49 +#### Limits

50 +

51 +The default configuration for this integration does not impose any limits.

52 +

53 +#### Performance Impact

54 +

55 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

56 +

57 +## Setup

58 +

59 +

60 +### Prerequisites

61 +

62 +#### Outbound HTTPS to Google

63 +

64 +The agent host must be able to reach `https://www.gstatic.com/ipranges/cloud.json`.

65 +No GCP credentials needed -- the file is public.

66 +

67 +

68 +

69 +### Configuration

70 +

71 +#### Options

72 +

73 +Add a named entry under `enrichment.network_sources`.

74 +

75 +

76 +<details open><summary>Config options</summary>

77 +

78 +

79 +

81 +|:-----|:------------|:--------|:---------:|

82 +| url | GCP publishes the master file here. | https://www.gstatic.com/ipranges/cloud.json | yes |

83 +| transform | jq expression mapping `prefixes[]` to `prefix` + label objects. | . | yes |

84 +

85 +

86 +</details>

87 +

88 +

89 +

90 +#### via File

91 +

92 +The configuration file name for this integration is `netflow.yaml`.

93 +

94 +

95 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

96 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

97 +

98 +```bash

99 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

100 +sudo ./edit-config netflow.yaml

101 +```

102 +

103 +##### Examples

104 +

105 +###### Tag all GCP prefixes by service and scope

106 +

107 +Sets tenant=gcp, role=<service>, region=<scope>.

108 +

109 +```yaml

110 +enrichment:

111 + network_sources:

112 + gcp:

113 + url: "https://www.gstatic.com/ipranges/cloud.json"

114 + interval: 24h

115 + transform: |

116 + .prefixes[] | {

117 + prefix: (.ipv4Prefix // .ipv6Prefix),

118 + tenant: "gcp",

119 + role: .service,

120 + region: .scope

121 + }

122 +

123 +```

124 +

125 +

126 +### Customer-only ranges

127 +

128 +GCP also publishes a `goog.json` file (broader: includes Google services beyond

129 +cloud). Use `cloud.json` for compute IP attribution; `goog.json` if you also

130 +want to tag Google's other services.

131 +

132 +

133 +

src/crates/netflow-plugin/integrations/generic_json-over-http_ipam.md new

+222

@@ -0,0 +1,222 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/generic_json-over-http_ipam.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "Generic JSON-over-HTTP IPAM"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Network Identity Sources"

7 +keywords: ['ipam', 'cmdb', 'infoblox', 'bluecat', 'phpipam', 'custom', 'prefix list']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# Generic JSON-over-HTTP IPAM

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: generic-ipam

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +The catch-all integration. Any IPAM, CMDB, or service that exposes prefix metadata

25 +via an HTTP-fetchable JSON endpoint can plug into Netdata's flow enrichment via this

26 +mechanism. Examples: Infoblox WAPI, BlueCat REST API, phpIPAM, internal-built CMDB

27 +endpoints, ServiceNow CMDB queries, custom Lambda functions producing JSON.

28 +

29 +You define the URL, the HTTP method, headers (for auth), and a jq transform that

30 +converts the response into objects with `prefix` + label fields.

31 +

32 +For the full network-identity concept, see

33 +[Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

34 +

35 +

36 +Periodic HTTPS GET (or POST) to a configured URL with optional headers, optional

37 +custom CA / mTLS, jq transform of the response, merge into network-attributes trie.

38 +

39 +

40 +This integration is only supported on the following platforms:

41 +

42 +- Linux

43 +

44 +This integration supports multiple instances configured side-by-side.

45 +

46 +

47 +### Default Behavior

48 +

49 +#### Auto-Detection

50 +

51 +Disabled by default. Add an entry per IPAM source under enrichment.network_sources.

52 +

53 +#### Limits

54 +

55 +The default configuration for this integration does not impose any limits.

56 +

57 +#### Performance Impact

58 +

59 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

60 +

61 +## Setup

62 +

63 +

64 +### Prerequisites

65 +

66 +#### An HTTP/HTTPS endpoint returning JSON

67 +

68 +The endpoint must produce a parseable JSON document. The plugin only supports

69 +GET and POST. There is no pagination, no cursor following, no OAuth flow --

70 +if your IPAM needs those, wrap it in an internal aggregator.

71 +

72 +

73 +#### Authentication via headers

74 +

75 +The plugin has no built-in auth helpers. Set whatever the API needs -- bearer

76 +tokens, basic-auth header, custom API-key headers -- via `headers:`. Store

77 +tokens carefully; they're written into the YAML.

78 +

79 +

80 +

81 +### Configuration

82 +

83 +#### Options

84 +

85 +Add a named entry under `enrichment.network_sources`. The keys below are the

86 +full set of options.

87 +

88 +

89 +<details open><summary>Config options</summary>

90 +

91 +

92 +

94 +|:-----|:------------|:--------|:---------:|

95 +| url | HTTP/HTTPS endpoint. | | yes |

96 +| method | GET or POST. | GET | no |

97 +| interval | Refresh interval (loop floors at 60s). | 60s | no |

98 +| timeout | Per-request timeout. | 60s | no |

99 +| headers | Map of additional HTTP request headers (e.g., authentication). | {} | no |

100 +| transform | jq expression converting response to {prefix, name?, role?, site?, region?, country?, state?, city?, tenant?, asn?, asn_name?} stream. | . | yes |

102 +| tls.ca_file | PEM file with the CA bundle. | | no |

103 +| tls.cert_file | PEM file with the client certificate (mTLS). | | no |

104 +| tls.key_file | PEM file with the client private key. | | no |

105 +

106 +

107 +</details>

108 +

109 +

110 +

111 +#### via File

112 +

113 +The configuration file name for this integration is `netflow.yaml`.

114 +

115 +

116 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

117 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

118 +

119 +```bash

120 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

121 +sudo ./edit-config netflow.yaml

122 +```

123 +

124 +##### Examples

125 +

126 +###### phpIPAM with API token

127 +

128 +phpIPAM exposes /api/<app>/subnets/. Use the standard transform.

129 +

130 +```yaml

131 +enrichment:

132 + network_sources:

133 + phpipam:

134 + url: "https://ipam.example/api/netdata/subnets/"

135 + headers:

136 + token: "abcdef..."

137 + interval: 10m

138 + transform: |

139 + .data[] | {

140 + prefix: (.subnet + "/" + (.mask|tostring)),

141 + name: .description,

142 + tenant: (.custom_tenant // ""),

143 + site: (.location.name // "")

144 + }

145 +

146 +```

147 +###### Custom internal CMDB (POST with body)

148 +

149 +When your CMDB requires POST with a query body. Define `method` and append the body via headers/url. The plugin's body support is limited -- prefer GET endpoints when possible.

150 +

151 +<details open><summary>Config</summary>

152 +

153 +```yaml

154 +enrichment:

155 + network_sources:

156 + cmdb:

157 + url: "https://cmdb.example/query/networks"

158 + method: POST

159 + headers:

160 + Authorization: "Bearer ..."

161 + Content-Type: "application/json"

162 + interval: 30m

163 + transform: |

164 + .results[] | {

165 + prefix: .cidr,

166 + tenant: .organization,

167 + site: .datacenter,

168 + role: .purpose

169 + }

170 +

171 +```

172 +</details>

173 +

174 +###### Internal IPAM with mTLS

175 +

176 +When the IPAM is behind your internal PKI.

177 +

178 +<details open><summary>Config</summary>

179 +

180 +```yaml

181 +enrichment:

182 + network_sources:

183 + corp_ipam:

184 + url: "https://ipam.corp/api/networks"

185 + tls:

186 + enable: true

187 + ca_file: /etc/netdata/ssl/corp-ca.pem

188 + cert_file: /etc/netdata/ssl/netdata.crt

189 + key_file: /etc/netdata/ssl/netdata.key

190 + interval: 10m

191 + transform: |

192 + .[] | {

193 + prefix: .cidr,

194 + name: .label,

195 + tenant: .tenant

196 + }

197 +

198 +```

199 +</details>

200 +

201 +

202 +

203 +### Endpoint requires pagination

204 +

205 +The plugin does not paginate. Either raise the page size to cover your inventory,

206 +or wrap the endpoint with an internal aggregator that returns all results at one URL.

207 +

208 +

209 +### TLS verification cannot be disabled

210 +

211 +`tls.skip_verify` and `tls.verify: false` are rejected by validation. Use

212 +`tls.ca_file` to trust internal CAs.

213 +

214 +

215 +### Empty result back-off

216 +

217 +An empty jq result is treated as a fetch failure. If your IPAM legitimately

218 +returns no prefixes (quiet state), the source backs off as if it errored.

219 +Workaround: have the upstream return at least one synthetic prefix.

220 +

221 +

222 +

src/crates/netflow-plugin/integrations/ipfix.md new

+140

@@ -0,0 +1,140 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/ipfix.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "IPFIX"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Sources"

7 +keywords: ['ipfix', 'netflow v10', 'flows', 'network flows', 'flow collector', 'rfc 7011']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# IPFIX

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: ipfix

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +Collects IPFIX (NetFlow v10) records from one or more exporters and stores them in tiered

25 +journal files. IPFIX extends NetFlow v9 with variable-length fields, vendor-specific

26 +information elements, and template withdrawal. Each record exposes the same core fields

27 +as NetFlow plus any additional IEs the exporter provides.

28 +

29 +For full documentation including vendor configuration examples (Cisco, Juniper, Arista,

30 +ASA NSEL), biflow handling, sampling caveats, and verification steps, see

31 +[IPFIX](https://learn.netdata.cloud/docs/network-flows/sources/ipfix) and the

32 +[Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/).

33 +

34 +

35 +The plugin listens on the same UDP socket as NetFlow. IPFIX messages are identified by

36 +version number 10 and decoded using cached templates. Decoded records are enriched and

37 +appended to disk-backed journal tiers.

38 +

39 +

40 +This integration is only supported on the following platforms:

41 +

42 +- Linux

43 +

44 +This integration runs as a single instance per Netdata Agent.

45 +

46 +

47 +### Default Behavior

48 +

49 +#### Auto-Detection

50 +

51 +The plugin starts when enabled in netflow.yaml and listens on the configured UDP port.

52 +

53 +#### Limits

54 +

55 +The default configuration for this integration does not impose any limits.

56 +

57 +#### Performance Impact

58 +

59 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

60 +

61 +## Setup

62 +

63 +

64 +### Prerequisites

65 +

66 +#### IPFIX-capable exporter

67 +

68 +A router, switch, or firewall configured to export IPFIX datagrams to the

69 +Netdata agent's UDP listener.

70 +

71 +

72 +

73 +### Configuration

74 +

75 +#### Options

76 +

77 +IPFIX shares the same `netflow.yaml` configuration file as NetFlow and sFlow.

78 +Enable IPFIX via the `protocols.ipfix` option.

79 +

80 +

81 +<details open><summary>Config options</summary>

82 +

83 +

84 +

86 +|:-----|:------------|:--------|:---------:|

87 +| listener.listen | UDP endpoint for IPFIX datagrams. | 0.0.0.0:2055 | no |

88 +| protocols.ipfix | Enable IPFIX decoding. | yes | no |

90 +| journal.size_of_journal_files | Maximum total size of all journal files. | 10GB | no |

91 +| journal.duration_of_journal_files | Maximum age of journal files. | 7d | no |

92 +

93 +

94 +</details>

95 +

96 +

97 +

98 +#### via File

99 +

100 +The configuration file name for this integration is `netflow.yaml`.

101 +

102 +

103 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

104 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

105 +

106 +```bash

107 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

108 +sudo ./edit-config netflow.yaml

109 +```

110 +

111 +##### Examples

112 +

113 +###### IPFIX collection

114 +

115 +Listen for IPFIX records on the standard port.

116 +

117 +```yaml

118 +enabled: true

119 +listener:

120 + listen: "0.0.0.0:2055"

121 +protocols:

122 + v5: false

123 + v7: false

124 + v9: false

125 + ipfix: true

126 + sflow: false

127 +

128 +```

129 +

130 +

131 +### Verifying flow data is arriving and diagnosing failures

132 +

133 +See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for

134 +the full diagnostic recipe. For IPFIX specifically, watch the `template_errors` dimension

135 +on `netflow.input_packets` -- IPFIX is template-driven and data records arriving before

136 +their templates are dropped. See also

137 +[Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation).

138 +

139 +

140 +

src/crates/netflow-plugin/integrations/iptoasn.md new

+145

@@ -0,0 +1,145 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/iptoasn.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "IPtoASN"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/IP Intelligence"

7 +keywords: ['iptoasn', 'asn', 'bgp', 'public asn', 'ip intelligence']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# IPtoASN

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: iptoasn

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +[IPtoASN](https://iptoasn.com/) is a free public IP-to-ASN database derived from

25 +BGP RIB snapshots. Daily updates, no license required. Use it as a free, open

26 +alternative to MaxMind ASN data when license cost or terms matter.

27 +

28 +IPtoASN provides ASN data only -- no geographic data. Pair with DB-IP, MaxMind,

29 +or another geo source for country/city enrichment.

30 +

31 +For the full IP-intelligence concept, see

32 +[IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

33 +

34 +

35 +The bundled `topology-ip-intel-downloader` supports IPtoASN as an ASN provider,

36 +fetching the latest TSV and converting it to MMDB format the plugin can read.

37 +Configure the downloader to use IPtoASN with `--asn iptoasn:combined`.

38 +

39 +

40 +This integration is only supported on the following platforms:

41 +

42 +- Linux

43 +

44 +This integration runs as a single instance per Netdata Agent.

45 +

46 +

47 +### Default Behavior

48 +

49 +#### Auto-Detection

50 +

51 +Not auto-detected as the default ASN source -- the plugin auto-detects DB-IP. To use IPtoASN as ASN, run the downloader explicitly.

52 +

53 +#### Limits

54 +

55 +The default configuration for this integration does not impose any limits.

56 +

57 +#### Performance Impact

58 +

59 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

60 +

61 +## Setup

62 +

63 +

64 +### Prerequisites

65 +

66 +#### Run the downloader with IPtoASN as ASN source

67 +

68 +IPtoASN is a TSV file. The bundled downloader knows how to fetch and convert

69 +it to MMDB:

70 +

71 +```bash

72 +sudo /usr/sbin/topology-ip-intel-downloader \

73 + --asn iptoasn:combined \

74 + --geo dbip:city-lite

75 +```

76 +

77 +This produces ASN data from IPtoASN and geographic data from DB-IP. Schedule

78 +this in cron (daily for ASN; weekly is enough for geo).

79 +

80 +

81 +

82 +### Configuration

83 +

84 +#### Options

85 +

86 +Once the downloader has produced MMDB files in the cache directory, the plugin

87 +auto-detects them. To pin the path explicitly, set `enrichment.geoip.asn_database`.

88 +

89 +

90 +<details open><summary>Config options</summary>

91 +

92 +

93 +

95 +|:-----|:------------|:--------|:---------:|

96 +| enrichment.geoip.asn_database | Path to the IPtoASN-derived MMDB. Empty = auto-detect from cache directory. | [] | no |

97 +

98 +

99 +</details>

100 +

101 +

102 +

103 +#### via File

104 +

105 +The configuration file name for this integration is `netflow.yaml`.

106 +

107 +

108 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

109 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

110 +

111 +```bash

112 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

113 +sudo ./edit-config netflow.yaml

114 +```

115 +

116 +##### Examples

117 +

118 +###### After running the downloader with IPtoASN

119 +

120 +Auto-detection picks up the cache copy.

121 +

122 +```yaml

123 +enrichment:

124 + geoip:

125 + asn_database: []

126 + geo_database: []

127 + optional: true

128 +

129 +```

130 +

131 +

132 +### ASN names not appearing

133 +

134 +IPtoASN's data does not always carry a human-readable ASN organization name.

135 +The plugin renders `AS{n}` (without a name) for those records. This is data-source-

136 +level, not a plugin issue. Use MaxMind GeoLite2-ASN if you need richer name data.

137 +

138 +

139 +### Outdated ASN attribution

140 +

141 +IPtoASN is rebuilt daily from BGP. Cron the downloader at least daily to keep

142 +ASN attribution current with real-world routing changes.

143 +

144 +

145 +

src/crates/netflow-plugin/integrations/maxmind_geoip_-_geolite2.md new

+163

@@ -0,0 +1,163 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/maxmind_geoip_-_geolite2.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "MaxMind GeoIP / GeoLite2"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/IP Intelligence"

7 +keywords: ['maxmind', 'geoip2', 'geolite2', 'geoip', 'asn', 'mmdb', 'ip intelligence']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# MaxMind GeoIP / GeoLite2

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: maxmind

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +MaxMind GeoIP2 (commercial) and GeoLite2 (free tier with license key) MMDB databases

25 +are read directly by the netflow plugin. The plugin uses any MMDB-format file that

26 +exposes the standard schema -- it is not tied to MaxMind specifically, but MaxMind

27 +is the canonical source and the format originator.

28 +

29 +Populates the same `SRC_COUNTRY`, `*_GEO_*`, and AS-name fields as DB-IP. Use this

30 +integration when you have a MaxMind license and prefer their data over the bundled

31 +DB-IP defaults.

32 +

33 +For the full IP-intelligence concept, see

34 +[IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

35 +

36 +

37 +You download the MaxMind MMDB files yourself (via `geoipupdate` or manual download),

38 +then point the plugin at their paths in `netflow.yaml`. The plugin reloads on file

39 +change every 30 seconds.

40 +

41 +

42 +This integration is only supported on the following platforms:

43 +

44 +- Linux

45 +

46 +This integration runs as a single instance per Netdata Agent.

47 +

48 +

49 +### Default Behavior

50 +

51 +#### Auto-Detection

52 +

53 +Not auto-detected. You must configure the database paths explicitly.

54 +

55 +#### Limits

56 +

57 +The default configuration for this integration does not impose any limits.

58 +

59 +#### Performance Impact

60 +

61 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

62 +

63 +## Setup

64 +

65 +

66 +### Prerequisites

67 +

68 +#### MaxMind license + downloaded MMDBs

69 +

70 +For GeoLite2 (free): create a MaxMind account, generate a license key, install

71 +`geoipupdate`, and configure it to fetch `GeoLite2-City.mmdb` and

72 +`GeoLite2-ASN.mmdb`. For GeoIP2 (paid): obtain a subscription and use the same

73 +`geoipupdate` mechanism with your paid license key.

74 +

75 +

76 +

77 +### Configuration

78 +

79 +#### Options

80 +

81 +Override the default DB-IP auto-detection by pointing `asn_database` and `geo_database`

82 +at your MaxMind MMDB files.

83 +

84 +

85 +<details open><summary>Config options</summary>

86 +

87 +

88 +

90 +|:-----|:------------|:--------|:---------:|

91 +| enrichment.geoip.asn_database | Paths to MaxMind ASN MMDB files (typically GeoLite2-ASN.mmdb or GeoIP2-ISP.mmdb). | [] | yes |

92 +| enrichment.geoip.geo_database | Paths to MaxMind geographic MMDB files (typically GeoLite2-City.mmdb or GeoIP2-City.mmdb). | [] | yes |

94 +

95 +

96 +</details>

97 +

98 +

99 +

100 +#### via File

101 +

102 +The configuration file name for this integration is `netflow.yaml`.

103 +

104 +

105 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

106 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

107 +

108 +```bash

109 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

110 +sudo ./edit-config netflow.yaml

111 +```

112 +

113 +##### Examples

114 +

115 +###### GeoLite2 (free tier)

116 +

117 +Standard `geoipupdate` install path. Free tier requires a license key.

118 +

119 +```yaml

120 +enrichment:

121 + geoip:

122 + asn_database:

123 + - /usr/share/GeoIP/GeoLite2-ASN.mmdb

124 + geo_database:

125 + - /usr/share/GeoIP/GeoLite2-City.mmdb

126 + optional: false

127 +

128 +```

129 +###### GeoIP2 (paid)

130 +

131 +Commercial subscription. Higher accuracy, more frequent updates.

132 +

133 +<details open><summary>Config</summary>

134 +

135 +```yaml

136 +enrichment:

137 + geoip:

138 + asn_database:

139 + - /usr/share/GeoIP/GeoIP2-ISP.mmdb

140 + geo_database:

141 + - /usr/share/GeoIP/GeoIP2-City.mmdb

142 + optional: false

143 +

144 +```

145 +</details>

146 +

147 +

148 +

149 +### License key missing or expired

150 +

151 +`geoipupdate` fails silently and the MMDB files become stale. Set up a working

152 +`geoipupdate` cron and monitor its exit code.

153 +

154 +

155 +### Schema differences between GeoLite2 and GeoIP2

156 +

157 +Both share the standard MMDB structure but the `Anonymous IP`, `ISP`, and

158 +`Connection Type` databases have GeoIP2-only fields the plugin does not read.

159 +Use `City` for geographic enrichment and `ASN` (GeoLite2) or `ISP` (GeoIP2)

160 +for AS data.

161 +

162 +

163 +

src/crates/netflow-plugin/integrations/netbox.md new

+193

@@ -0,0 +1,193 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/netbox.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "NetBox"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Network Identity Sources"

7 +keywords: ['netbox', 'ipam', 'dcim', 'source of truth', 'prefix list']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# NetBox

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: netbox

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +[NetBox](https://netboxlabs.com/oss/netbox/) is the most widely deployed open-source

25 +IPAM / DCIM. Many networks already curate prefix metadata there -- tenant, site,

26 +role, VRF -- and want flow data to inherit those labels automatically rather than

27 +duplicating them in `netflow.yaml`.

28 +

29 +This integration polls NetBox's Prefixes API at a configurable interval, transforms

30 +the response with jq, and labels flow records with the prefix metadata.

31 +

32 +For the full network-identity concept, see

33 +[Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

34 +

35 +

36 +Periodic HTTPS GET to a NetBox API endpoint with a Bearer token in the

37 +`Authorization` header. jq transform produces per-prefix objects with the labels

38 +you want -- typically `tenant.name`, `site.name`, `role.name`, `description`.

39 +

40 +NetBox paginates results -- there is **no automatic pagination** in this plugin.

41 +For inventories larger than the default page size (50), wrap NetBox with a

42 +server-side aggregator that returns the full list at one URL.

43 +

44 +

45 +This integration is only supported on the following platforms:

46 +

47 +- Linux

48 +

49 +This integration supports multiple instances configured side-by-side.

50 +

51 +

52 +### Default Behavior

53 +

54 +#### Auto-Detection

55 +

56 +Disabled by default. Add an entry under enrichment.network_sources with your NetBox URL and API token.

57 +

58 +#### Limits

59 +

60 +The default configuration for this integration does not impose any limits.

61 +

62 +#### Performance Impact

63 +

64 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

65 +

66 +## Setup

67 +

68 +

69 +### Prerequisites

70 +

71 +#### NetBox API token with read scope on Prefixes

72 +

73 +In NetBox, create or use a service account, generate an API token, scope it

74 +read-only to the Prefixes endpoint. The token goes in the `Authorization` header.

75 +

76 +

77 +#### A bulk endpoint or aggregator

78 +

79 +The plugin does not paginate. If your NetBox has more prefixes than fit in the

80 +default page (`?limit=50`), either raise `limit` (`?limit=10000`) or expose

81 +an internal endpoint that aggregates all pages and serves them at one URL.

82 +

83 +

84 +

85 +### Configuration

86 +

87 +#### Options

88 +

89 +Add a named entry under `enrichment.network_sources` pointing at your NetBox.

90 +

91 +

92 +<details open><summary>Config options</summary>

93 +

94 +

95 +

97 +|:-----|:------------|:--------|:---------:|

98 +| url | NetBox Prefixes API endpoint, with `?limit=` if needed. | | yes |

99 +| headers.Authorization | NetBox API token, prefixed with "Token ". | | yes |

100 +| interval | How often to refresh. NetBox is your source of truth -- 5-15 minutes is typical. | 60s | no |

101 +| transform | jq expression mapping `.results[]` (NetBox's response shape) to per-prefix objects. | . | yes |

102 +

103 +

104 +</details>

105 +

106 +

107 +

108 +#### via File

109 +

110 +The configuration file name for this integration is `netflow.yaml`.

111 +

112 +

113 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

114 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

115 +

116 +```bash

117 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

118 +sudo ./edit-config netflow.yaml

119 +```

120 +

121 +##### Examples

122 +

123 +###### NetBox with API token and standard label set

124 +

125 +Tags prefixes with tenant, site, role, and the NetBox description.

126 +

127 +```yaml

128 +enrichment:

129 + network_sources:

130 + netbox:

131 + url: "https://netbox.example.internal/api/ipam/prefixes/?limit=10000"

132 + headers:

133 + Authorization: "Token abcdef0123456789"

134 + interval: 5m

135 + timeout: 30s

136 + transform: |

137 + .results[] | {

138 + prefix: .prefix,

139 + tenant: (.tenant.name // ""),

140 + site: (.site.name // ""),

141 + role: (.role.name // ""),

142 + name: .description

143 + }

144 +

145 +```

146 +###### NetBox with mTLS to internal CA

147 +

148 +When NetBox is behind your internal PKI; use tls.ca_file plus client cert.

149 +

150 +<details open><summary>Config</summary>

151 +

152 +```yaml

153 +enrichment:

154 + network_sources:

155 + netbox:

156 + url: "https://netbox.example.internal/api/ipam/prefixes/?limit=10000"

157 + headers:

158 + Authorization: "Token abcdef0123456789"

159 + interval: 5m

160 + tls:

161 + enable: true

162 + ca_file: /etc/netdata/ssl/internal-ca.pem

163 + cert_file: /etc/netdata/ssl/netdata.crt

164 + key_file: /etc/netdata/ssl/netdata.key

165 + transform: |

166 + .results[] | {

167 + prefix: .prefix,

168 + tenant: (.tenant.name // ""),

169 + site: (.site.name // ""),

170 + role: (.role.name // ""),

171 + name: .description

172 + }

173 +

174 +```

175 +</details>

176 +

177 +

178 +

179 +### Only first page of results loaded

180 +

181 +NetBox paginates by default at 50 results. The plugin does not follow `next`

182 +links. Use `?limit=10000` (or the actual count) on the URL, or expose an

183 +aggregating endpoint server-side.

184 +

185 +

186 +### Token missing or wrong scope

187 +

188 +NetBox returns 403 silently consumed by the plugin's HTTP error path. Watch

189 +the journal for `network-sources` warnings; verify with curl:

190 +`curl -H "Authorization: Token <tok>" https://netbox/api/ipam/prefixes/`.

191 +

192 +

193 +

src/crates/netflow-plugin/integrations/netflow.md new

+160

@@ -0,0 +1,160 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/netflow.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "NetFlow"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Sources"

7 +keywords: ['netflow', 'netflow v5', 'netflow v7', 'netflow v9', 'cisco', 'flows', 'network flows', 'flow collector']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# NetFlow

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: netflow

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +Collects NetFlow v5, v7, and v9 records from one or more exporters (routers, switches, firewalls)

25 +and stores them in tiered journal files. Each record exposes source and destination IP, ports,

26 +protocol, bytes, packets, ToS, TCP flags, and ingress/egress interface indices.

27 +Enrichment adds GeoIP country/city/ASN, static metadata, and classifier tags.

28 +

29 +For full documentation including vendor configuration examples, sampling caveats, template

30 +handling and verification steps, see [NetFlow](https://learn.netdata.cloud/docs/network-flows/sources/netflow)

31 +and the [Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/).

32 +

33 +

34 +The plugin listens on a configurable UDP socket for NetFlow datagrams.

35 +NetFlow v5 and v7 records are decoded directly. NetFlow v9 records are decoded using

36 +dynamic templates cached from the exporter. Decoded records are enriched in-memory

37 +and appended to disk-backed journal tiers (raw, 1-minute, 5-minute, 1-hour rollups).

38 +

39 +

40 +This integration is only supported on the following platforms:

41 +

42 +- Linux

43 +

44 +This integration runs as a single instance per Netdata Agent.

45 +

46 +

47 +### Default Behavior

48 +

49 +#### Auto-Detection

50 +

51 +The plugin starts when enabled in netflow.yaml and listens on the configured UDP port.

52 +

53 +#### Limits

54 +

55 +The default configuration for this integration does not impose any limits.

56 +

57 +#### Performance Impact

58 +

59 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

60 +

61 +## Setup

62 +

63 +

64 +### Prerequisites

65 +

66 +#### NetFlow-capable exporter

67 +

68 +A router, switch, or firewall configured to export NetFlow v5, v7, or v9 datagrams to the

69 +Netdata agent's UDP listener.

70 +

71 +

72 +

73 +### Configuration

74 +

75 +#### Options

76 +

77 +The plugin is configured via `netflow.yaml` in the Netdata configuration directory.

78 +

79 +

80 +<details open><summary>Config options</summary>

81 +

82 +

83 +

85 +|:-----|:------------|:--------|:---------:|

86 +| listener.listen | UDP endpoint for NetFlow datagrams. | 0.0.0.0:2055 | no |

87 +| protocols.v5 | Enable NetFlow v5 decoding. | yes | no |

88 +| protocols.v7 | Enable NetFlow v7 decoding. | yes | no |

89 +| protocols.v9 | Enable NetFlow v9 decoding. | yes | no |

91 +| journal.size_of_journal_files | Maximum total size of all journal files. | 10GB | no |

92 +| journal.duration_of_journal_files | Maximum age of journal files. | 7d | no |

93 +

94 +

95 +</details>

96 +

97 +

98 +

99 +#### via File

100 +

101 +The configuration file name for this integration is `netflow.yaml`.

102 +

103 +

104 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

105 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

106 +

107 +```bash

108 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

109 +sudo ./edit-config netflow.yaml

110 +```

111 +

112 +##### Examples

113 +

114 +###### Basic NetFlow v5/v9 collection

115 +

116 +Listen on the standard NetFlow port for v5 and v9 records.

117 +

118 +```yaml

119 +enabled: true

120 +listener:

121 + listen: "0.0.0.0:2055"

122 +protocols:

123 + v5: true

124 + v9: true

125 +

126 +```

127 +###### NetFlow v9 only with extended retention

128 +

129 +Accept only v9 records and keep 30 days of journal data.

130 +

131 +<details open><summary>Config</summary>

132 +

133 +```yaml

134 +enabled: true

135 +listener:

136 + listen: "0.0.0.0:2055"

137 +protocols:

138 + v5: false

139 + v7: false

140 + v9: true

141 +journal:

142 + journal_dir: flows

143 + size_of_journal_files: 50GB

144 + duration_of_journal_files: 30d

145 +

146 +```

147 +</details>

148 +

149 +

150 +

151 +### Verifying flow data is arriving and diagnosing failures

152 +

153 +See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for

154 +the full diagnostic recipe -- including UDP path checks, template-error analysis,

155 +and the "looks like a bug but isn't" section (doubling, mirroring, internal-IP geolocation).

156 +See also [Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation)

157 +and [Anti-patterns](https://learn.netdata.cloud/docs/network-flows/anti-patterns).

158 +

159 +

160 +

src/crates/netflow-plugin/integrations/sflow.md new

+143

@@ -0,0 +1,143 @@

1 +<!--startmeta

2 +custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/sflow.md"

3 +meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"

4 +sidebar_label: "sFlow"

5 +learn_status: "Published"

6 +learn_rel_path: "Network Flows/Sources"

7 +keywords: ['sflow', 'sflow v5', 'sampled flows', 'flows', 'network flows', 'flow collector', 'inmon']

8 +message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"

9 +endmeta-->

10 +

11 +# sFlow

12 +

13 +

14 +<img src="https://netdata.cloud/img/network-wired.svg" width="150"/>

15 +

16 +

17 +Plugin: netflow-plugin

18 +Module: sflow

19 +

20 +<img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />

21 +

22 +## Overview

23 +

24 +Collects sFlow v5 datagrams from one or more agents and stores them in tiered journal files.

25 +sFlow provides statistically sampled packet headers, interface counters, or extended

26 +gateway data. Each flow record exposes source and destination IP, ports, protocol, bytes,

27 +packets, and sampling rate information.

28 +

29 +For full documentation including how sFlow differs fundamentally from NetFlow (packet

30 +sampling vs aggregated flows), vendor configuration examples (Arista, Juniper, Aruba CX,

31 +Ruckus, hsflowd), and the limits of sampled data, see

32 +[sFlow](https://learn.netdata.cloud/docs/network-flows/sources/sflow) and the

33 +[Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/).

34 +

35 +

36 +The plugin listens on the same UDP socket as NetFlow. sFlow datagrams are identified by

37 +their distinct header format and decoded per the sFlow v5 specification. Decoded records

38 +are enriched and appended to disk-backed journal tiers.

39 +

40 +

41 +This integration is only supported on the following platforms:

42 +

43 +- Linux

44 +

45 +This integration runs as a single instance per Netdata Agent.

46 +

47 +

48 +### Default Behavior

49 +

50 +#### Auto-Detection

51 +

52 +The plugin starts when enabled in netflow.yaml and listens on the configured UDP port.

53 +

54 +#### Limits

55 +

56 +The default configuration for this integration does not impose any limits.

57 +

58 +#### Performance Impact

59 +

60 +The default configuration for this integration is not expected to impose a significant performance impact on the system.

61 +

62 +## Setup

63 +

64 +

65 +### Prerequisites

66 +

67 +#### sFlow-capable agent

68 +

69 +A switch, router, or host-based sFlow agent (such as Host sFlow) configured to send

70 +sFlow v5 datagrams to the Netdata agent's UDP listener.

71 +

72 +

73 +

74 +### Configuration

75 +

76 +#### Options

77 +

78 +sFlow shares the same `netflow.yaml` configuration file as NetFlow and IPFIX.

79 +Enable sFlow via the `protocols.sflow` option.

80 +

81 +

82 +<details open><summary>Config options</summary>

83 +

84 +

85 +

87 +|:-----|:------------|:--------|:---------:|

88 +| listener.listen | UDP endpoint for sFlow datagrams. | 0.0.0.0:2055 | no |

89 +| protocols.sflow | Enable sFlow decoding. | yes | no |

91 +| journal.size_of_journal_files | Maximum total size of all journal files. | 10GB | no |

92 +| journal.duration_of_journal_files | Maximum age of journal files. | 7d | no |

93 +

94 +

95 +</details>

96 +

97 +

98 +

99 +#### via File

100 +

101 +The configuration file name for this integration is `netflow.yaml`.

102 +

103 +

104 +You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the

105 +Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).

106 +

107 +```bash

108 +cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata

109 +sudo ./edit-config netflow.yaml

110 +```

111 +

112 +##### Examples

113 +

114 +###### sFlow collection

115 +

116 +Listen for sFlow v5 datagrams on the standard port.

117 +

118 +```yaml

119 +enabled: true

120 +listener:

121 + listen: "0.0.0.0:2055"

122 +protocols:

123 + v5: false

124 + v7: false

125 + v9: false

126 + ipfix: false

127 + sflow: true

128 +

129 +```

130 +

131 +

132 +### Verifying sFlow is arriving and diagnosing failures

133 +

134 +See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for

135 +the full diagnostic recipe. sFlow-specific gotchas: counter samples are not surfaced

136 +(only flow samples), bytes/packets are statistical estimates that won't match SNMP

137 +byte-for-byte, and VLAN information comes from `ExtendedSwitch` records only -- not

138 +from 802.1Q tags inside the sampled header. See also

139 +[Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation)

140 +and the sFlow section of [Anti-patterns](https://learn.netdata.cloud/docs/network-flows/anti-patterns).

141 +

142 +

143 +

src/crates/netflow-plugin/metadata.yaml new

+2221

@@ -0,0 +1,2221 @@

1 +plugin_name: netflow-plugin

2 +modules:

3 + - meta:

4 + plugin_name: netflow-plugin

5 + module_name: netflow

6 + monitored_instance:

7 + name: NetFlow

8 + link: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_pi/configuration/15-s/nf-15-s-book/ios-netflow-ov.html

9 + categories:

10 + - flows.sources

11 + icon_filename: network-wired.svg

12 + keywords:

13 + - netflow

14 + - netflow v5

15 + - netflow v7

16 + - netflow v9

17 + - cisco

18 + - flows

19 + - network flows

20 + - flow collector

21 + related_resources:

22 + integrations:

23 + list:

24 + - plugin_name: netflow-plugin

25 + module_name: ipfix

26 + - plugin_name: netflow-plugin

27 + module_name: sflow

28 + info_provided_to_referring_integrations:

29 + description: ""

30 + overview:

31 + data_collection:

32 + metrics_description: |

33 + Collects NetFlow v5, v7, and v9 records from one or more exporters (routers, switches, firewalls)

34 + and stores them in tiered journal files. Each record exposes source and destination IP, ports,

35 + protocol, bytes, packets, ToS, TCP flags, and ingress/egress interface indices.

36 + Enrichment adds GeoIP country/city/ASN, static metadata, and classifier tags.

37 +

38 + For full documentation including vendor configuration examples, sampling caveats, template

39 + handling and verification steps, see [NetFlow](https://learn.netdata.cloud/docs/network-flows/sources/netflow)

40 + and the [Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/).

41 + method_description: |

42 + The plugin listens on a configurable UDP socket for NetFlow datagrams.

43 + NetFlow v5 and v7 records are decoded directly. NetFlow v9 records are decoded using

44 + dynamic templates cached from the exporter. Decoded records are enriched in-memory

45 + and appended to disk-backed journal tiers (raw, 1-minute, 5-minute, 1-hour rollups).

46 + supported_platforms:

47 + include:

48 + - Linux

49 + exclude: []

50 + multi_instance: false

51 + additional_permissions:

52 + description: ""

53 + default_behavior:

54 + auto_detection:

55 + description: "The plugin starts when enabled in netflow.yaml and listens on the configured UDP port."

56 + limits:

57 + description: ""

58 + performance_impact:

59 + description: ""

60 + setup:

61 + prerequisites:

62 + list:

63 + - title: NetFlow-capable exporter

64 + description: |

65 + A router, switch, or firewall configured to export NetFlow v5, v7, or v9 datagrams to the

66 + Netdata agent's UDP listener.

67 + configuration:

68 + file:

69 + name: netflow.yaml

70 + options:

71 + description: |

72 + The plugin is configured via `netflow.yaml` in the Netdata configuration directory.

73 + folding:

74 + title: Config options

75 + enabled: true

76 + list:

77 + - name: listener.listen

78 + description: UDP endpoint for NetFlow datagrams.

79 + default_value: "0.0.0.0:2055"

80 + required: false

81 + - name: protocols.v5

82 + description: Enable NetFlow v5 decoding.

83 + default_value: true

84 + required: false

85 + - name: protocols.v7

86 + description: Enable NetFlow v7 decoding.

87 + default_value: true

88 + required: false

89 + - name: protocols.v9

90 + description: Enable NetFlow v9 decoding.

91 + default_value: true

92 + required: false

93 + - name: journal.journal_dir

94 + description: Directory for journal files (relative to NETDATA_CACHE_DIR).

95 + default_value: flows

96 + required: false

97 + - name: journal.size_of_journal_files

98 + description: Maximum total size of all journal files.

99 + default_value: "10GB"

100 + required: false

101 + - name: journal.duration_of_journal_files

102 + description: Maximum age of journal files.

103 + default_value: "7d"

104 + required: false

105 + examples:

106 + folding:

107 + title: Config

108 + enabled: true

109 + list:

110 + - name: Basic NetFlow v5/v9 collection

111 + folding:

112 + enabled: false

113 + description: Listen on the standard NetFlow port for v5 and v9 records.

114 + config: |

115 + enabled: true

116 + listener:

117 + listen: "0.0.0.0:2055"

118 + protocols:

119 + v5: true

120 + v9: true

121 + - name: NetFlow v9 only with extended retention

122 + folding:

123 + enabled: true

124 + description: Accept only v9 records and keep 30 days of journal data.

125 + config: |

126 + enabled: true

127 + listener:

128 + listen: "0.0.0.0:2055"

129 + protocols:

130 + v5: false

131 + v7: false

132 + v9: true

133 + journal:

134 + journal_dir: flows

135 + size_of_journal_files: 50GB

136 + duration_of_journal_files: 30d

137 + troubleshooting:

138 + problems:

139 + list:

140 + - name: Verifying flow data is arriving and diagnosing failures

141 + description: |

142 + See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for

143 + the full diagnostic recipe -- including UDP path checks, template-error analysis,

144 + and the "looks like a bug but isn't" section (doubling, mirroring, internal-IP geolocation).

145 + See also [Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation)

146 + and [Anti-patterns](https://learn.netdata.cloud/docs/network-flows/anti-patterns).

147 + alerts: []

148 + metrics:

149 + folding:

150 + title: Metrics

151 + enabled: false

152 + description: |

153 + Network flow data is stored as journal records, not as traditional metrics.

154 + Use the "Network Flows" tab in the Netdata dashboard to query and visualize flow data.

155 + For the full list of fields and the per-protocol availability matrix, see the

156 + [Field Reference](https://learn.netdata.cloud/docs/network-flows/field-reference).

157 + For visualisation guidance see [Sankey and Table](https://learn.netdata.cloud/docs/network-flows/visualization/summary-sankey),

158 + [Time-Series](https://learn.netdata.cloud/docs/network-flows/visualization/time-series),

159 + and [Maps and Globe](https://learn.netdata.cloud/docs/network-flows/visualization/maps-globe).

160 + availability: []

161 + scopes: []

162 +

163 + - meta:

164 + plugin_name: netflow-plugin

165 + module_name: ipfix

166 + monitored_instance:

167 + name: IPFIX

168 + link: https://datatracker.ietf.org/doc/html/rfc7011

169 + categories:

170 + - flows.sources

171 + icon_filename: network-wired.svg

172 + keywords:

173 + - ipfix

174 + - netflow v10

175 + - flows

176 + - network flows

177 + - flow collector

178 + - rfc 7011

179 + related_resources:

180 + integrations:

181 + list:

182 + - plugin_name: netflow-plugin

183 + module_name: netflow

184 + - plugin_name: netflow-plugin

185 + module_name: sflow

186 + info_provided_to_referring_integrations:

187 + description: ""

188 + overview:

189 + data_collection:

190 + metrics_description: |

191 + Collects IPFIX (NetFlow v10) records from one or more exporters and stores them in tiered

192 + journal files. IPFIX extends NetFlow v9 with variable-length fields, vendor-specific

193 + information elements, and template withdrawal. Each record exposes the same core fields

194 + as NetFlow plus any additional IEs the exporter provides.

195 +

196 + For full documentation including vendor configuration examples (Cisco, Juniper, Arista,

197 + ASA NSEL), biflow handling, sampling caveats, and verification steps, see

198 + [IPFIX](https://learn.netdata.cloud/docs/network-flows/sources/ipfix) and the

199 + [Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/).

200 + method_description: |

201 + The plugin listens on the same UDP socket as NetFlow. IPFIX messages are identified by

202 + version number 10 and decoded using cached templates. Decoded records are enriched and

203 + appended to disk-backed journal tiers.

204 + supported_platforms:

205 + include:

206 + - Linux

207 + exclude: []

208 + multi_instance: false

209 + additional_permissions:

210 + description: ""

211 + default_behavior:

212 + auto_detection:

213 + description: "The plugin starts when enabled in netflow.yaml and listens on the configured UDP port."

214 + limits:

215 + description: ""

216 + performance_impact:

217 + description: ""

218 + setup:

219 + prerequisites:

220 + list:

221 + - title: IPFIX-capable exporter

222 + description: |

223 + A router, switch, or firewall configured to export IPFIX datagrams to the

224 + Netdata agent's UDP listener.

225 + configuration:

226 + file:

227 + name: netflow.yaml

228 + options:

229 + description: |

230 + IPFIX shares the same `netflow.yaml` configuration file as NetFlow and sFlow.

231 + Enable IPFIX via the `protocols.ipfix` option.

232 + folding:

233 + title: Config options

234 + enabled: true

235 + list:

236 + - name: listener.listen

237 + description: UDP endpoint for IPFIX datagrams.

238 + default_value: "0.0.0.0:2055"

239 + required: false

240 + - name: protocols.ipfix

241 + description: Enable IPFIX decoding.

242 + default_value: true

243 + required: false

244 + - name: journal.journal_dir

245 + description: Directory for journal files (relative to NETDATA_CACHE_DIR).

246 + default_value: flows

247 + required: false

248 + - name: journal.size_of_journal_files

249 + description: Maximum total size of all journal files.

250 + default_value: "10GB"

251 + required: false

252 + - name: journal.duration_of_journal_files

253 + description: Maximum age of journal files.

254 + default_value: "7d"

255 + required: false

256 + examples:

257 + folding:

258 + title: Config

259 + enabled: true

260 + list:

261 + - name: IPFIX collection

262 + folding:

263 + enabled: false

264 + description: Listen for IPFIX records on the standard port.

265 + config: |

266 + enabled: true

267 + listener:

268 + listen: "0.0.0.0:2055"

269 + protocols:

270 + v5: false

271 + v7: false

272 + v9: false

273 + ipfix: true

274 + sflow: false

275 + troubleshooting:

276 + problems:

277 + list:

278 + - name: Verifying flow data is arriving and diagnosing failures

279 + description: |

280 + See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for

281 + the full diagnostic recipe. For IPFIX specifically, watch the `template_errors` dimension

282 + on `netflow.input_packets` -- IPFIX is template-driven and data records arriving before

283 + their templates are dropped. See also

284 + [Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation).

285 + alerts: []

286 + metrics:

287 + folding:

288 + title: Metrics

289 + enabled: false

290 + description: |

291 + Network flow data is stored as journal records, not as traditional metrics.

292 + Use the "Network Flows" tab in the Netdata dashboard to query and visualize flow data.

293 + For the full list of fields and the per-protocol availability matrix, see the

294 + [Field Reference](https://learn.netdata.cloud/docs/network-flows/field-reference).

295 + IPFIX biflow records are emitted as two separate flow records (forward + reverse).

296 + availability: []

297 + scopes: []

298 +

299 + - meta:

300 + plugin_name: netflow-plugin

301 + module_name: sflow

302 + monitored_instance:

303 + name: sFlow

304 + link: https://sflow.org/

305 + categories:

306 + - flows.sources

307 + icon_filename: network-wired.svg

308 + keywords:

309 + - sflow

310 + - sflow v5

311 + - sampled flows

312 + - flows

313 + - network flows

314 + - flow collector

315 + - inmon

316 + related_resources:

317 + integrations:

318 + list:

319 + - plugin_name: netflow-plugin

320 + module_name: netflow

321 + - plugin_name: netflow-plugin

322 + module_name: ipfix

323 + info_provided_to_referring_integrations:

324 + description: ""

325 + overview:

326 + data_collection:

327 + metrics_description: |

328 + Collects sFlow v5 datagrams from one or more agents and stores them in tiered journal files.

329 + sFlow provides statistically sampled packet headers, interface counters, or extended

330 + gateway data. Each flow record exposes source and destination IP, ports, protocol, bytes,

331 + packets, and sampling rate information.

332 +

333 + For full documentation including how sFlow differs fundamentally from NetFlow (packet

334 + sampling vs aggregated flows), vendor configuration examples (Arista, Juniper, Aruba CX,

335 + Ruckus, hsflowd), and the limits of sampled data, see

336 + [sFlow](https://learn.netdata.cloud/docs/network-flows/sources/sflow) and the

337 + [Network Flows Overview](https://learn.netdata.cloud/docs/network-flows/).

338 + method_description: |

339 + The plugin listens on the same UDP socket as NetFlow. sFlow datagrams are identified by

340 + their distinct header format and decoded per the sFlow v5 specification. Decoded records

341 + are enriched and appended to disk-backed journal tiers.

342 + supported_platforms:

343 + include:

344 + - Linux

345 + exclude: []

346 + multi_instance: false

347 + additional_permissions:

348 + description: ""

349 + default_behavior:

350 + auto_detection:

351 + description: "The plugin starts when enabled in netflow.yaml and listens on the configured UDP port."

352 + limits:

353 + description: ""

354 + performance_impact:

355 + description: ""

356 + setup:

357 + prerequisites:

358 + list:

359 + - title: sFlow-capable agent

360 + description: |

361 + A switch, router, or host-based sFlow agent (such as Host sFlow) configured to send

362 + sFlow v5 datagrams to the Netdata agent's UDP listener.

363 + configuration:

364 + file:

365 + name: netflow.yaml

366 + options:

367 + description: |

368 + sFlow shares the same `netflow.yaml` configuration file as NetFlow and IPFIX.

369 + Enable sFlow via the `protocols.sflow` option.

370 + folding:

371 + title: Config options

372 + enabled: true

373 + list:

374 + - name: listener.listen

375 + description: UDP endpoint for sFlow datagrams.

376 + default_value: "0.0.0.0:2055"

377 + required: false

378 + - name: protocols.sflow

379 + description: Enable sFlow decoding.

380 + default_value: true

381 + required: false

382 + - name: journal.journal_dir

383 + description: Directory for journal files (relative to NETDATA_CACHE_DIR).

384 + default_value: flows

385 + required: false

386 + - name: journal.size_of_journal_files

387 + description: Maximum total size of all journal files.

388 + default_value: "10GB"

389 + required: false

390 + - name: journal.duration_of_journal_files

391 + description: Maximum age of journal files.

392 + default_value: "7d"

393 + required: false

394 + examples:

395 + folding:

396 + title: Config

397 + enabled: true

398 + list:

399 + - name: sFlow collection

400 + folding:

401 + enabled: false

402 + description: Listen for sFlow v5 datagrams on the standard port.

403 + config: |

404 + enabled: true

405 + listener:

406 + listen: "0.0.0.0:2055"

407 + protocols:

408 + v5: false

409 + v7: false

410 + v9: false

411 + ipfix: false

412 + sflow: true

413 + troubleshooting:

414 + problems:

415 + list:

416 + - name: Verifying sFlow is arriving and diagnosing failures

417 + description: |

418 + See [Troubleshooting](https://learn.netdata.cloud/docs/network-flows/troubleshooting) for

419 + the full diagnostic recipe. sFlow-specific gotchas: counter samples are not surfaced

420 + (only flow samples), bytes/packets are statistical estimates that won't match SNMP

421 + byte-for-byte, and VLAN information comes from `ExtendedSwitch` records only -- not

422 + from 802.1Q tags inside the sampled header. See also

423 + [Validation and Data Quality](https://learn.netdata.cloud/docs/network-flows/validation)

424 + and the sFlow section of [Anti-patterns](https://learn.netdata.cloud/docs/network-flows/anti-patterns).

425 + alerts: []

426 + metrics:

427 + folding:

428 + title: Metrics

429 + enabled: false

430 + description: |

431 + Network flow data is stored as journal records, not as traditional metrics.

432 + Use the "Network Flows" tab in the Netdata dashboard to query and visualize flow data.

433 + For the full list of fields and the per-protocol availability matrix, see the

434 + [Field Reference](https://learn.netdata.cloud/docs/network-flows/field-reference).

435 + sFlow byte counts are statistical estimates (sampled packets multiplied by sampling rate);

436 + they will not reconcile byte-for-byte with SNMP interface counters.

437 + availability: []

438 + scopes: []

439 +

440 + # =====================================================================

441 + # IP Intelligence: GeoIP / ASN data sources used to enrich flow records

442 + # =====================================================================

443 +

444 + - meta:

445 + plugin_name: netflow-plugin

446 + module_name: dbip

447 + monitored_instance:

448 + name: DB-IP IP Intelligence

449 + link: https://db-ip.com/

450 + categories:

451 + - flows.ip-intelligence

452 + icon_filename: network-wired.svg

453 + keywords:

454 + - geoip

455 + - asn

456 + - dbip

457 + - db-ip

458 + - mmdb

459 + - ip intelligence

460 + - flow enrichment

461 + related_resources:

462 + integrations:

463 + list:

464 + - plugin_name: netflow-plugin

465 + module_name: maxmind

466 + - plugin_name: netflow-plugin

467 + module_name: iptoasn

468 + - plugin_name: netflow-plugin

469 + module_name: custom-mmdb

470 + info_provided_to_referring_integrations:

471 + description: ""

472 + overview:

473 + data_collection:

474 + metrics_description: |

475 + DB-IP is the **default** IP intelligence source for the Netdata netflow plugin. Its

476 + MMDB-format databases are bundled with native packages (DEB, RPM) under

477 + `/usr/share/netdata/topology-ip-intel/`. Refreshing pulls newer data from

478 + `download.db-ip.com` via the bundled `topology-ip-intel-downloader`.

479 +

480 + Populates `SRC_COUNTRY`, `DST_COUNTRY`, `SRC_GEO_STATE`, `DST_GEO_STATE`,

481 + `SRC_GEO_CITY`, `DST_GEO_CITY`, `SRC_GEO_LATITUDE`, `DST_GEO_LATITUDE`,

482 + `SRC_GEO_LONGITUDE`, `DST_GEO_LONGITUDE`, plus the AS-number and AS-name fields

483 + when included in the resolution chain.

484 +

485 + For the full IP-intelligence concept (MMDB format, lookup priority, internal-IP

486 + handling, hot reload semantics), see

487 + [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

488 + method_description: |

489 + Files are read on plugin start and reloaded automatically every 30 seconds when

490 + their mtime or size changes. Lookups happen in-process; there is no per-flow network

491 + call. Auto-detection scans `${NETDATA_CACHE_DIR}/topology-ip-intel/` first, falling

492 + back to the stock copy under `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/`.

493 + supported_platforms:

494 + include:

495 + - Linux

496 + exclude: []

497 + multi_instance: false

498 + additional_permissions:

499 + description: ""

500 + default_behavior:

501 + auto_detection:

502 + description: "Native packages ship the stock DB-IP MMDB files; the plugin auto-detects them at startup. No configuration required for the default install."

503 + limits:

504 + description: ""

505 + performance_impact:

506 + description: ""

507 + setup:

508 + prerequisites:

509 + list:

510 + - title: DB-IP MMDB files

511 + description: |

512 + Ships with native packages. For source builds, run the bundled downloader once

513 + to populate `/var/cache/netdata/topology-ip-intel/`:

514 +

515 + ```bash

516 + sudo /usr/sbin/topology-ip-intel-downloader

517 + ```

518 +

519 + Subsequent refreshes (e.g., monthly cron) re-fetch from db-ip.com.

520 + configuration:

521 + file:

522 + name: netflow.yaml

523 + options:

524 + description: |

525 + Configure DB-IP under `enrichment.geoip` in `netflow.yaml`. Empty `asn_database`

526 + and `geo_database` enable auto-detection.

527 + folding:

528 + title: Config options

529 + enabled: true

530 + list:

531 + - name: enrichment.geoip.asn_database

532 + description: List of MMDB paths providing AS data. Empty = auto-detect under cache/stock dirs.

533 + default_value: "[] (auto-detect)"

534 + required: false

535 + - name: enrichment.geoip.geo_database

536 + description: List of MMDB paths providing geo data. Empty = auto-detect.

537 + default_value: "[] (auto-detect)"

538 + required: false

539 + - name: enrichment.geoip.optional

540 + description: When true, missing or unreadable MMDBs are warnings, not fatal. Auto-detected files default to optional.

541 + default_value: "false (true when auto-detected)"

542 + required: false

543 + examples:

544 + folding:

545 + title: Config

546 + enabled: true

547 + list:

548 + - name: Default (auto-detect stock files)

549 + folding:

550 + enabled: false

551 + description: Native package install. No explicit configuration; the plugin finds the stock or cache copy automatically.

552 + config: |

553 + enrichment:

554 + geoip:

555 + asn_database: []

556 + geo_database: []

557 + optional: true

558 + - name: Explicit DB-IP paths

559 + folding:

560 + enabled: true

561 + description: Override auto-detection by pointing to specific DB-IP MMDBs (for example, after running the downloader to a non-standard location).

562 + config: |

563 + enrichment:

564 + geoip:

565 + asn_database:

566 + - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb

567 + geo_database:

568 + - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb

569 + optional: false

570 + troubleshooting:

571 + problems:

572 + list:

573 + - name: Internal IPs appearing in random countries

574 + description: |

575 + GeoIP databases have no entry for RFC 1918 / private space. The stock DB-IP

576 + build tags private ranges so `*_AS_NAME` renders as "AS0 Private IP Address Space"

577 + with empty country. With third-party MMDBs, results may vary. Declare your

578 + internal CIDRs under `enrichment.networks` to override -- see

579 + [Static metadata](https://learn.netdata.cloud/docs/network-flows/enrichment/static-metadata).

580 + - name: Stale databases

581 + description: |

582 + The plugin does not alert on staleness. Check file mtime: `ls -la /var/cache/netdata/topology-ip-intel/`.

583 + Schedule a weekly cron of `topology-ip-intel-downloader` to keep data fresh.

584 + alerts: []

585 + metrics:

586 + folding:

587 + title: Metrics

588 + enabled: false

589 + description: |

590 + IP intelligence enriches existing flow records; it does not produce metrics of its own.

591 + Verify enrichment is working by querying `SRC_COUNTRY` / `DST_COUNTRY` on the

592 + Network Flows tab and confirming non-empty values for public IPs.

593 + availability: []

594 + scopes: []

595 +

596 + - meta:

597 + plugin_name: netflow-plugin

598 + module_name: maxmind

599 + monitored_instance:

600 + name: MaxMind GeoIP / GeoLite2

601 + link: https://www.maxmind.com/

602 + categories:

603 + - flows.ip-intelligence

604 + icon_filename: network-wired.svg

605 + keywords:

606 + - maxmind

607 + - geoip2

608 + - geolite2

609 + - geoip

610 + - asn

611 + - mmdb

612 + - ip intelligence

613 + related_resources:

614 + integrations:

615 + list:

616 + - plugin_name: netflow-plugin

617 + module_name: dbip

618 + - plugin_name: netflow-plugin

619 + module_name: iptoasn

620 + - plugin_name: netflow-plugin

621 + module_name: custom-mmdb

622 + info_provided_to_referring_integrations:

623 + description: ""

624 + overview:

625 + data_collection:

626 + metrics_description: |

627 + MaxMind GeoIP2 (commercial) and GeoLite2 (free tier with license key) MMDB databases

628 + are read directly by the netflow plugin. The plugin uses any MMDB-format file that

629 + exposes the standard schema -- it is not tied to MaxMind specifically, but MaxMind

630 + is the canonical source and the format originator.

631 +

632 + Populates the same `SRC_COUNTRY`, `*_GEO_*`, and AS-name fields as DB-IP. Use this

633 + integration when you have a MaxMind license and prefer their data over the bundled

634 + DB-IP defaults.

635 +

636 + For the full IP-intelligence concept, see

637 + [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

638 + method_description: |

639 + You download the MaxMind MMDB files yourself (via `geoipupdate` or manual download),

640 + then point the plugin at their paths in `netflow.yaml`. The plugin reloads on file

641 + change every 30 seconds.

642 + supported_platforms:

643 + include:

644 + - Linux

645 + exclude: []

646 + multi_instance: false

647 + additional_permissions:

648 + description: ""

649 + default_behavior:

650 + auto_detection:

651 + description: "Not auto-detected. You must configure the database paths explicitly."

652 + limits:

653 + description: ""

654 + performance_impact:

655 + description: ""

656 + setup:

657 + prerequisites:

658 + list:

659 + - title: MaxMind license + downloaded MMDBs

660 + description: |

661 + For GeoLite2 (free): create a MaxMind account, generate a license key, install

662 + `geoipupdate`, and configure it to fetch `GeoLite2-City.mmdb` and

663 + `GeoLite2-ASN.mmdb`. For GeoIP2 (paid): obtain a subscription and use the same

664 + `geoipupdate` mechanism with your paid license key.

665 + configuration:

666 + file:

667 + name: netflow.yaml

668 + options:

669 + description: |

670 + Override the default DB-IP auto-detection by pointing `asn_database` and `geo_database`

671 + at your MaxMind MMDB files.

672 + folding:

673 + title: Config options

674 + enabled: true

675 + list:

676 + - name: enrichment.geoip.asn_database

677 + description: Paths to MaxMind ASN MMDB files (typically GeoLite2-ASN.mmdb or GeoIP2-ISP.mmdb).

678 + default_value: "[]"

679 + required: true

680 + - name: enrichment.geoip.geo_database

681 + description: Paths to MaxMind geographic MMDB files (typically GeoLite2-City.mmdb or GeoIP2-City.mmdb).

682 + default_value: "[]"

683 + required: true

684 + - name: enrichment.geoip.optional

685 + description: When true, missing or unreadable MMDBs are warnings, not fatal at startup.

686 + default_value: "false"

687 + required: false

688 + examples:

689 + folding:

690 + title: Config

691 + enabled: true

692 + list:

693 + - name: GeoLite2 (free tier)

694 + folding:

695 + enabled: false

696 + description: Standard `geoipupdate` install path. Free tier requires a license key.

697 + config: |

698 + enrichment:

699 + geoip:

700 + asn_database:

701 + - /usr/share/GeoIP/GeoLite2-ASN.mmdb

702 + geo_database:

703 + - /usr/share/GeoIP/GeoLite2-City.mmdb

704 + optional: false

705 + - name: GeoIP2 (paid)

706 + folding:

707 + enabled: true

708 + description: Commercial subscription. Higher accuracy, more frequent updates.

709 + config: |

710 + enrichment:

711 + geoip:

712 + asn_database:

713 + - /usr/share/GeoIP/GeoIP2-ISP.mmdb

714 + geo_database:

715 + - /usr/share/GeoIP/GeoIP2-City.mmdb

716 + optional: false

717 + troubleshooting:

718 + problems:

719 + list:

720 + - name: License key missing or expired

721 + description: |

722 + `geoipupdate` fails silently and the MMDB files become stale. Set up a working

723 + `geoipupdate` cron and monitor its exit code.

724 + - name: Schema differences between GeoLite2 and GeoIP2

725 + description: |

726 + Both share the standard MMDB structure but the `Anonymous IP`, `ISP`, and

727 + `Connection Type` databases have GeoIP2-only fields the plugin does not read.

728 + Use `City` for geographic enrichment and `ASN` (GeoLite2) or `ISP` (GeoIP2)

729 + for AS data.

730 + alerts: []

731 + metrics:

732 + folding:

733 + title: Metrics

734 + enabled: false

735 + description: |

736 + Enriches flow records; produces no metrics of its own. Verify on the Network Flows

737 + tab via `SRC_COUNTRY`, `DST_COUNTRY`, and `*_AS_NAME` columns.

738 + availability: []

739 + scopes: []

740 +

741 + - meta:

742 + plugin_name: netflow-plugin

743 + module_name: iptoasn

744 + monitored_instance:

745 + name: IPtoASN

746 + link: https://iptoasn.com/

747 + categories:

748 + - flows.ip-intelligence

749 + icon_filename: network-wired.svg

750 + keywords:

751 + - iptoasn

752 + - asn

753 + - bgp

754 + - public asn

755 + - ip intelligence

756 + related_resources:

757 + integrations:

758 + list:

759 + - plugin_name: netflow-plugin

760 + module_name: dbip

761 + - plugin_name: netflow-plugin

762 + module_name: maxmind

763 + - plugin_name: netflow-plugin

764 + module_name: custom-mmdb

765 + info_provided_to_referring_integrations:

766 + description: ""

767 + overview:

768 + data_collection:

769 + metrics_description: |

770 + [IPtoASN](https://iptoasn.com/) is a free public IP-to-ASN database derived from

771 + BGP RIB snapshots. Daily updates, no license required. Use it as a free, open

772 + alternative to MaxMind ASN data when license cost or terms matter.

773 +

774 + IPtoASN provides ASN data only -- no geographic data. Pair with DB-IP, MaxMind,

775 + or another geo source for country/city enrichment.

776 +

777 + For the full IP-intelligence concept, see

778 + [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

779 + method_description: |

780 + The bundled `topology-ip-intel-downloader` supports IPtoASN as an ASN provider,

781 + fetching the latest TSV and converting it to MMDB format the plugin can read.

782 + Configure the downloader to use IPtoASN with `--asn iptoasn:combined`.

783 + supported_platforms:

784 + include:

785 + - Linux

786 + exclude: []

787 + multi_instance: false

788 + additional_permissions:

789 + description: ""

790 + default_behavior:

791 + auto_detection:

792 + description: "Not auto-detected as the default ASN source -- the plugin auto-detects DB-IP. To use IPtoASN as ASN, run the downloader explicitly."

793 + limits:

794 + description: ""

795 + performance_impact:

796 + description: ""

797 + setup:

798 + prerequisites:

799 + list:

800 + - title: Run the downloader with IPtoASN as ASN source

801 + description: |

802 + IPtoASN is a TSV file. The bundled downloader knows how to fetch and convert

803 + it to MMDB:

804 +

805 + ```bash

806 + sudo /usr/sbin/topology-ip-intel-downloader \

807 + --asn iptoasn:combined \

808 + --geo dbip:city-lite

809 + ```

810 +

811 + This produces ASN data from IPtoASN and geographic data from DB-IP. Schedule

812 + this in cron (daily for ASN; weekly is enough for geo).

813 + configuration:

814 + file:

815 + name: netflow.yaml

816 + options:

817 + description: |

818 + Once the downloader has produced MMDB files in the cache directory, the plugin

819 + auto-detects them. To pin the path explicitly, set `enrichment.geoip.asn_database`.

820 + folding:

821 + title: Config options

822 + enabled: true

823 + list:

824 + - name: enrichment.geoip.asn_database

825 + description: Path to the IPtoASN-derived MMDB. Empty = auto-detect from cache directory.

826 + default_value: "[]"

827 + required: false

828 + examples:

829 + folding:

830 + title: Config

831 + enabled: true

832 + list:

833 + - name: After running the downloader with IPtoASN

834 + folding:

835 + enabled: false

836 + description: Auto-detection picks up the cache copy.

837 + config: |

838 + enrichment:

839 + geoip:

840 + asn_database: []

841 + geo_database: []

842 + optional: true

843 + troubleshooting:

844 + problems:

845 + list:

846 + - name: ASN names not appearing

847 + description: |

848 + IPtoASN's data does not always carry a human-readable ASN organization name.

849 + The plugin renders `AS{n}` (without a name) for those records. This is data-source-

850 + level, not a plugin issue. Use MaxMind GeoLite2-ASN if you need richer name data.

851 + - name: Outdated ASN attribution

852 + description: |

853 + IPtoASN is rebuilt daily from BGP. Cron the downloader at least daily to keep

854 + ASN attribution current with real-world routing changes.

855 + alerts: []

856 + metrics:

857 + folding:

858 + title: Metrics

859 + enabled: false

860 + description: |

861 + Enriches flow records with AS numbers; provides no metrics of its own.

862 + availability: []

863 + scopes: []

864 +

865 + - meta:

866 + plugin_name: netflow-plugin

867 + module_name: custom-mmdb

868 + monitored_instance:

869 + name: Custom MMDB Database

870 + link: https://maxmind.github.io/MaxMind-DB/

871 + categories:

872 + - flows.ip-intelligence

873 + icon_filename: network-wired.svg

874 + keywords:

875 + - mmdb

876 + - custom database

877 + - bring your own

878 + - ipinfo

879 + - ip intelligence

880 + related_resources:

881 + integrations:

882 + list:

883 + - plugin_name: netflow-plugin

884 + module_name: dbip

885 + - plugin_name: netflow-plugin

886 + module_name: maxmind

887 + - plugin_name: netflow-plugin

888 + module_name: iptoasn

889 + info_provided_to_referring_integrations:

890 + description: ""

891 + overview:

892 + data_collection:

893 + metrics_description: |

894 + The plugin reads any MMDB file that conforms to the standard schema -- this catch-all

895 + integration covers IPInfo, custom-built internal MMDBs, vendor-specific feeds, or

896 + any provider that publishes MMDB data.

897 +

898 + The plugin reads `country.iso_code`, `city.names.en`, `subdivisions[].iso_code`,

899 + `location.latitude`, `location.longitude`, `autonomous_system_number`, and

900 + `autonomous_system_organization`. Vendor-specific extra fields are ignored.

901 +

902 + For the full IP-intelligence concept, see

903 + [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment/ip-intelligence).

904 + method_description: |

905 + You produce or download an MMDB file. Place it on the agent host. Point the

906 + plugin at it via `netflow.yaml`. The plugin reloads on file change every 30

907 + seconds.

908 + supported_platforms:

909 + include:

910 + - Linux

911 + exclude: []

912 + multi_instance: false

913 + additional_permissions:

914 + description: ""

915 + default_behavior:

916 + auto_detection:

917 + description: "Not auto-detected. You must configure paths explicitly."

918 + limits:

919 + description: ""

920 + performance_impact:

921 + description: ""

922 + setup:

923 + prerequisites:

924 + list:

925 + - title: A standards-compliant MMDB file

926 + description: |

927 + The MMDB file must use the [standard MMDB schema](https://maxmind.github.io/MaxMind-DB/).

928 + Validate with `mmdblookup` from the `libmaxminddb-tools` package before deploying.

929 +

930 + Common sources: IPInfo (`ipinfo.io`), custom internal builds via the `mmdbwriter`

931 + Go tool, or vendor-specific feeds.

932 + configuration:

933 + file:

934 + name: netflow.yaml

935 + options:

936 + description: |

937 + Point `enrichment.geoip.asn_database` and/or `enrichment.geoip.geo_database` at

938 + your MMDB file paths.

939 + folding:

940 + title: Config options

941 + enabled: true

942 + list:

943 + - name: enrichment.geoip.asn_database

944 + description: List of MMDB paths providing AS data. Multiple files allowed; later entries override on overlap.

945 + default_value: "[]"

946 + required: false

947 + - name: enrichment.geoip.geo_database

948 + description: List of MMDB paths providing geographic data.

949 + default_value: "[]"

950 + required: false

951 + - name: enrichment.geoip.optional

952 + description: When true, missing files become startup warnings instead of fatal errors.

953 + default_value: "false"

954 + required: false

955 + examples:

956 + folding:

957 + title: Config

958 + enabled: true

959 + list:

960 + - name: IPInfo MMDB

961 + folding:

962 + enabled: false

963 + description: Using IPInfo's MMDB feed (subscription required).

964 + config: |

965 + enrichment:

966 + geoip:

967 + asn_database:

968 + - /opt/mmdb/ipinfo-asn.mmdb

969 + geo_database:

970 + - /opt/mmdb/ipinfo-city.mmdb

971 + optional: false

972 + - name: Internal custom MMDB

973 + folding:

974 + enabled: true

975 + description: Built in-house with `mmdbwriter`. Combines public BGP data with internal CIDR labels.

976 + config: |

977 + enrichment:

978 + geoip:

979 + asn_database:

980 + - /etc/netdata/internal-asn.mmdb

981 + geo_database:

982 + - /etc/netdata/internal-geo.mmdb

983 + optional: false

984 + troubleshooting:

985 + problems:

986 + list:

987 + - name: Lookups silently return empty

988 + description: |

989 + The MMDB schema is non-standard or the IP types don't match (some custom builds

990 + use `string` instead of `array` for ASN). Validate with `mmdblookup -f file.mmdb -i 8.8.8.8`

991 + and confirm the standard fields are present.

992 + - name: Plugin fails to start with optional=false

993 + description: |

994 + File missing or unreadable at the configured path. Check permissions; the netdata

995 + user must be able to read the file.

996 + alerts: []

997 + metrics:

998 + folding:

999 + title: Metrics

1000 + enabled: false

1001 + description: |

1002 + Enriches flow records; produces no metrics of its own.

1003 + availability: []

1004 + scopes: []

1005 +

1006 + # =====================================================================

1007 + # BGP Routing: live route feeds (BMP, BioRIS) used to enrich flow records

1008 + # =====================================================================

1009 +

1010 + - meta:

1011 + plugin_name: netflow-plugin

1012 + module_name: bmp

1013 + monitored_instance:

1014 + name: BMP (BGP Monitoring Protocol)

1015 + link: https://www.rfc-editor.org/rfc/rfc7854

1016 + categories:

1017 + - flows.bgp-routing

1018 + icon_filename: network-wired.svg

1019 + keywords:

1020 + - bmp

1021 + - bgp

1022 + - rfc 7854

1023 + - route monitoring

1024 + - cisco

1025 + - juniper

1026 + - frr

1027 + related_resources:

1028 + integrations:

1029 + list:

1030 + - plugin_name: netflow-plugin

1031 + module_name: bioris

1032 + info_provided_to_referring_integrations:

1033 + description: ""

1034 + overview:

1035 + data_collection:

1036 + metrics_description: |

1037 + BMP (BGP Monitoring Protocol, RFC 7854) lets a router push its BGP route updates

1038 + to a passive collector. With this integration enabled, Netdata acts as that

1039 + collector -- it listens for BMP TCP connections from your routers, parses the BGP

1040 + UPDATE messages, and builds an in-memory routing table that flow enrichment then

1041 + reads from.

1042 +

1043 + The result: every flow gets accurate AS numbers, AS paths, communities, and

1044 + next-hop information from your real-time BGP table -- not from a stale GeoIP

1045 + database or from whatever the exporter happened to send in the flow record.

1046 +

1047 + For the full BGP-routing concept (shared trie with BioRIS, withdrawal handling,

1048 + per-vendor caveats, integration test gap), see

1049 + [BGP Routing](https://learn.netdata.cloud/docs/network-flows/enrichment/bgp-routing).

1050 + method_description: |

1051 + The plugin runs a TCP listener on `0.0.0.0:10179` (Akvorado convention -- not the

1052 + IANA-registered port 7854). Routers initiate BMP sessions to it. The plugin

1053 + processes Initiation, Termination, RouteMonitoring (BGP UPDATE messages), and

1054 + PeerDownNotification messages. NLRI types: IPv4/IPv6 unicast, MPLS-labelled, VPNv4,

1055 + VPNv6, EVPN IP-prefix.

1056 +

1057 + BMP and BioRIS share a single in-memory routing trie. Memory grows with the size

1058 + of the BGP table; a full IPv4+IPv6 feed is roughly 1.2M prefixes per peer.

1059 + supported_platforms:

1060 + include:

1061 + - Linux

1062 + exclude: []

1063 + multi_instance: false

1064 + additional_permissions:

1065 + description: ""

1066 + default_behavior:

1067 + auto_detection:

1068 + description: "Disabled by default. Set enrichment.routing_dynamic.bmp.enabled to true and configure your routers."

1069 + limits:

1070 + description: ""

1071 + performance_impact:

1072 + description: ""

1073 + setup:

1074 + prerequisites:

1075 + list:

1076 + - title: BMP-capable routers

1077 + description: |

1078 + Modern Cisco IOS-XR, Juniper JunOS, Arista EOS, and FRR all support BMP v3.

1079 + The plugin parses RFC 7854 BMP v3 specifically. Older versions (v1, v2) are

1080 + not supported and will fail to parse.

1081 + - title: TCP reachability between routers and the agent

1082 + description: |

1083 + Routers initiate the connection -- the plugin is a passive listener. Allow

1084 + inbound TCP on the configured port (default 10179) from each BMP-speaking

1085 + router to the agent.

1086 + - title: No TLS, no authentication

1087 + description: |

1088 + The listener accepts plain TCP only. Restrict access at the firewall and on a

1089 + dedicated management network -- do not expose 10179 to the public internet.

1090 + configuration:

1091 + file:

1092 + name: netflow.yaml

1093 + options:

1094 + description: |

1095 + All BMP options live under `enrichment.routing_dynamic.bmp` in `netflow.yaml`.

1096 + folding:

1097 + title: Config options

1098 + enabled: true

1099 + list:

1100 + - name: enabled

1101 + description: Master switch. Set to true to start the listener.

1102 + default_value: "false"

1103 + required: false

1104 + - name: listen

1105 + description: TCP bind address (host:port).

1106 + default_value: "0.0.0.0:10179"

1107 + required: false

1108 + - name: keep

1109 + description: Grace window after a BMP disconnect before purging that session's routes.

1110 + default_value: "5m"

1111 + required: false

1112 + - name: max_consecutive_decode_errors

1113 + description: Close the session after N consecutive decode errors.

1114 + default_value: "8"

1115 + required: false

1116 + - name: receive_buffer

1117 + description: Optional SO_RCVBUF per connection in bytes (0 = OS default).

1118 + default_value: "0"

1119 + required: false

1120 + - name: collect_asns

1121 + description: When false, AS numbers from BMP are forced to 0.

1122 + default_value: "true"

1123 + required: false

1124 + - name: collect_as_paths

1125 + description: When false, AS paths are dropped before storage.

1126 + default_value: "true"

1127 + required: false

1128 + - name: collect_communities

1129 + description: When false, communities and large communities are dropped.

1130 + default_value: "true"

1131 + required: false

1132 + - name: rds

1133 + description: |

1134 + Whitelist of accepted Route Distinguishers. Empty = accept all.

1135 + Formats: "0", "ASN:idx", "IPv4:idx", or full text.

1136 + default_value: "[]"

1137 + required: false

1138 + examples:

1139 + folding:

1140 + title: Config

1141 + enabled: true

1142 + list:

1143 + - name: Enable BMP listener

1144 + folding:

1145 + enabled: false

1146 + description: Start the listener on the default port.

1147 + config: |

1148 + enrichment:

1149 + routing_dynamic:

1150 + bmp:

1151 + enabled: true

1152 + listen: "0.0.0.0:10179"

1153 + keep: 5m

1154 + - name: Cisco IOS-XR router config

1155 + folding:

1156 + enabled: true

1157 + description: Vendor-side config to send BMP to Netdata. The bmp server block is global, not under router bgp.

1158 + config: |

1159 + bmp server 1

1160 + host 10.0.0.10 port 10179

1161 + description "Netdata BMP collector"

1162 + initial-delay 5

1163 + stats-reporting-period 60

1164 + initial-refresh delay 30 spread 2

1165 + !

1166 + router bgp 65000

1167 + neighbor 192.0.2.1

1168 + bmp-activate server 1

1169 + - name: Juniper JunOS router config

1170 + folding:

1171 + enabled: true

1172 + description: Recommended local-address and statistics-timeout for production.

1173 + config: |

1174 + set routing-options bmp station netdata station-address 10.0.0.10

1175 + set routing-options bmp station netdata station-port 10179

1176 + set routing-options bmp station netdata connection-mode active

1177 + set routing-options bmp station netdata local-address 10.0.0.1

1178 + set routing-options bmp station netdata statistics-timeout 60

1179 + set routing-options bmp station netdata route-monitoring pre-policy

1180 + - name: FRR (bgpd) router config

1181 + folding:

1182 + enabled: true

1183 + description: |

1184 + Critical -- BMP is a runtime module in FRR. Without "-M bmp" in

1185 + /etc/frr/daemons (bgpd_options), every BMP command silently fails.

1186 + config: |

1187 + # /etc/frr/daemons:

1188 + # bgpd_options=" -A 127.0.0.1 -M bmp"

1189 + router bgp 65000

1190 + bmp targets netdata

1191 + bmp connect 10.0.0.10 port 10179 min-retry 5000 max-retry 60000

1192 + bmp stats interval 60000

1193 + bmp monitor ipv4 unicast pre-policy

1194 + bmp monitor ipv6 unicast pre-policy

1195 + exit

1196 + troubleshooting:

1197 + problems:

1198 + list:

1199 + - name: Listener not receiving BMP sessions

1200 + description: |

1201 + Check `show bmp` (Cisco) / `show bmp connections` (Juniper) / `show bmp targets` (FRR)

1202 + to confirm the router has dialed in. The plugin does not initiate -- it listens.

1203 + Firewall: allow inbound TCP on 10179.

1204 + - name: Memory growth

1205 + description: |

1206 + A full BGP feed adds ~1.2M prefixes per peer permanently. There is no time-based

1207 + eviction. Plan capacity accordingly. After a router disconnect, the routes

1208 + for that session are kept for `keep` (default 5 min) before purging.

1209 + - name: Integration-test gap

1210 + description: |

1211 + BMP message parsing has unit tests. The TCP listener path, framed decode loop,

1212 + trie apply, and per-router cleanup are NOT integration-tested. Validate against

1213 + your specific router firmware before depending on this for capacity / security

1214 + decisions.

1215 + alerts: []

1216 + metrics:

1217 + folding:

1218 + title: Metrics

1219 + enabled: false

1220 + description: |

1221 + Enriches flow records with `SRC_AS`, `DST_AS`, `SRC_MASK`, `DST_MASK`, `NEXT_HOP`,

1222 + `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES`. Source-side AS path and

1223 + communities are not surfaced. AS names come from the GeoIP/ASN MMDB, not from BMP.

1224 + Verify by querying the AS-path columns on the Network Flows tab.

1225 + availability: []

1226 + scopes: []

1227 +

1228 + - meta:

1229 + plugin_name: netflow-plugin

1230 + module_name: bioris

1231 + monitored_instance:

1232 + name: bio-rd / RIPE RIS

1233 + link: https://github.com/bio-routing/bio-rd

1234 + categories:

1235 + - flows.bgp-routing

1236 + icon_filename: network-wired.svg

1237 + keywords:

1238 + - bioris

1239 + - bio-rd

1240 + - ripe ris

1241 + - bgp

1242 + - grpc

1243 + - route information service

1244 + related_resources:

1245 + integrations:

1246 + list:

1247 + - plugin_name: netflow-plugin

1248 + module_name: bmp

1249 + info_provided_to_referring_integrations:

1250 + description: ""

1251 + overview:

1252 + data_collection:

1253 + metrics_description: |

1254 + BioRIS lets Netdata consume BGP routing data from a [bio-rd](https://github.com/bio-routing/bio-rd)

1255 + `cmd/ris/` daemon over gRPC. bio-rd is a Go-based BGP daemon that can peer with

1256 + [RIPE RIS](https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris)

1257 + Route Collectors -- or any BGP / BMP source you have access to -- and expose the

1258 + resulting RIB through a gRPC interface. Netdata is a client of that interface.

1259 +

1260 + Use this when you want a third-party view of the BGP routing table (e.g., RIPE

1261 + RIS's view) without running a BGP session yourself or deploying BMP across your

1262 + network.

1263 +

1264 + For the full BGP-routing concept and how BMP and BioRIS share the same trie, see

1265 + [BGP Routing](https://learn.netdata.cloud/docs/network-flows/enrichment/bgp-routing).

1266 + method_description: |

1267 + The plugin connects to one or more bio-rd `ris` gRPC endpoints. It runs three RPCs:

1268 + `GetRouters` to discover what's available, `DumpRIB` to do baseline reconciliation,

1269 + and `ObserveRIB` for incremental updates. Multiple instances are additive (not

1270 + failover); they all merge into the shared in-memory trie.

1271 + supported_platforms:

1272 + include:

1273 + - Linux

1274 + exclude: []

1275 + multi_instance: false

1276 + additional_permissions:

1277 + description: ""

1278 + default_behavior:

1279 + auto_detection:

1280 + description: "Disabled by default. Set enrichment.routing_dynamic.bioris.enabled to true and provide at least one ris_instances entry."

1281 + limits:

1282 + description: ""

1283 + performance_impact:

1284 + description: ""

1285 + setup:

1286 + prerequisites:

1287 + list:

1288 + - title: A running bio-rd 'ris' daemon

1289 + description: |

1290 + bio-rd is a separate project. The plugin only consumes its gRPC interface; it

1291 + does not bundle bio-rd. You install it yourself:

1292 +

1293 + ```bash

1294 + # Install Go (>=1.20), then:

1295 + git clone https://github.com/bio-routing/bio-rd.git

1296 + cd bio-rd/cmd/ris

1297 + go build -o /usr/local/bin/ris .

1298 + ```

1299 +

1300 + Configure `ris` to peer with one or more BGP / BMP sources (RIPE RIS Route

1301 + Collectors, your own peers, etc.). Refer to the bio-rd documentation for the

1302 + peering setup -- this is bio-rd's configuration, not Netdata's.

1303 +

1304 + Run the daemon with a gRPC port:

1305 + `/usr/local/bin/ris --grpc_port 50051 --config.file /etc/bio-rd.yml`

1306 + - title: Network reachability + no auth

1307 + description: |

1308 + The gRPC connection is plain HTTP/2 by default (or TLS with system-CA when

1309 + `grpc_secure: true`). There is no authentication. Restrict access at the

1310 + firewall, or run bio-rd on the same host as the agent and bind it to localhost.

1311 + configuration:

1312 + file:

1313 + name: netflow.yaml

1314 + options:

1315 + description: |

1316 + BioRIS options live under `enrichment.routing_dynamic.bioris`.

1317 + folding:

1318 + title: Config options

1319 + enabled: true

1320 + list:

1321 + - name: enabled

1322 + description: Master switch.

1323 + default_value: "false"

1324 + required: false

1325 + - name: timeout

1326 + description: Connect + per-RPC timeout. Default is aggressive for public RIS over the internet -- raise if you see "deadline exceeded".

1327 + default_value: "200ms"

1328 + required: false

1329 + - name: refresh

1330 + description: How often to re-dump every router's RIB from scratch.

1331 + default_value: "30m"

1332 + required: false

1333 + - name: refresh_timeout

1334 + description: Per-DumpRIB request timeout and per-message stream timeout.

1335 + default_value: "10s"

1336 + required: false

1337 + - name: ris_instances

1338 + description: |

1339 + List of bio-rd endpoints. Each: grpc_addr, grpc_secure, vrf, vrf_id.

1340 + Multiple instances are additive (not failover) -- routes from all merge.

1341 + default_value: "[]"

1342 + required: true

1343 + examples:

1344 + folding:

1345 + title: Config

1346 + enabled: true

1347 + list:

1348 + - name: Local bio-rd

1349 + folding:

1350 + enabled: false

1351 + description: bio-rd running on the same host, plain gRPC.

1352 + config: |

1353 + enrichment:

1354 + routing_dynamic:

1355 + bioris:

1356 + enabled: true

1357 + timeout: 2s

1358 + refresh: 30m

1359 + refresh_timeout: 30s

1360 + ris_instances:

1361 + - grpc_addr: "127.0.0.1:50051"

1362 + grpc_secure: false

1363 + - name: Remote bio-rd over TLS

1364 + folding:

1365 + enabled: true

1366 + description: Across a network, system CA bundle. No client cert / mTLS.

1367 + config: |

1368 + enrichment:

1369 + routing_dynamic:

1370 + bioris:

1371 + enabled: true

1372 + timeout: 5s

1373 + ris_instances:

1374 + - grpc_addr: "ris.example.internal:50051"

1375 + grpc_secure: true

1376 + vrf: "global"

1377 + troubleshooting:

1378 + problems:

1379 + list:

1380 + - name: Default 200ms timeout too aggressive

1381 + description: |

1382 + Over the public internet to RIPE RIS, you may need 2-5 seconds. If you see

1383 + "deadline exceeded" errors in the journal, raise `timeout`.

1384 + - name: Initial dump takes minutes for full feeds

1385 + description: |

1386 + A full IPv4+IPv6 RIB from a route collector is millions of prefixes. The first

1387 + refresh takes time; subsequent observe streams are incremental.

1388 + - name: Integration-test gap

1389 + description: |

1390 + proto and route conversion are unit-tested. The gRPC client path

1391 + (connecting, consuming streams, retry/backoff) is NOT integration-tested.

1392 + Validate against your specific bio-rd setup before relying on this for

1393 + capacity / security decisions.

1394 + alerts: []

1395 + metrics:

1396 + folding:

1397 + title: Metrics

1398 + enabled: false

1399 + description: |

1400 + Enriches flow records with the same fields as BMP (`SRC_AS`, `DST_AS`, masks,

1401 + `NEXT_HOP`, `DST_AS_PATH`, `DST_COMMUNITIES`, `DST_LARGE_COMMUNITIES`). Both

1402 + sources contribute to the same shared trie -- if both are enabled, lookups

1403 + prefer the best-matching route across both.

1404 + availability: []

1405 + scopes: []

1406 +

1407 + # =====================================================================

1408 + # Network Identity: external feeds that label your network prefixes

1409 + # =====================================================================

1410 +

1411 + - meta:

1412 + plugin_name: netflow-plugin

1413 + module_name: aws-ip-ranges

1414 + monitored_instance:

1415 + name: AWS IP Ranges

1416 + link: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

1417 + categories:

1418 + - flows.network-identity

1419 + icon_filename: network-wired.svg

1420 + keywords:

1421 + - aws

1422 + - amazon

1423 + - cloud

1424 + - ip ranges

1425 + - vpc

1426 + - ec2

1427 + - prefix list

1428 + related_resources:

1429 + integrations:

1430 + list:

1431 + - plugin_name: netflow-plugin

1432 + module_name: gcp-ip-ranges

1433 + - plugin_name: netflow-plugin

1434 + module_name: azure-ip-ranges

1435 + - plugin_name: netflow-plugin

1436 + module_name: netbox

1437 + - plugin_name: netflow-plugin

1438 + module_name: generic-ipam

1439 + info_provided_to_referring_integrations:

1440 + description: ""

1441 + overview:

1442 + data_collection:

1443 + metrics_description: |

1444 + AWS publishes a continuously updated JSON file listing every public IP prefix used

1445 + by AWS services -- per region, per service. This integration fetches that file

1446 + periodically, transforms it via a jq expression, and uses the result to label flow

1447 + records destined to / from AWS with `*_NET_TENANT="amazon"` plus a per-region tag.

1448 +

1449 + The result: traffic to/from AWS shows up clearly in dashboards as "amazon", with

1450 + per-region and per-service breakdown if you customize the jq transform.

1451 +

1452 + For the full network-identity concept (merge order, jq transform, TLS verification),

1453 + see [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

1454 + method_description: |

1455 + The plugin issues a periodic GET to `https://ip-ranges.amazonaws.com/ip-ranges.json`,

1456 + parses the JSON body, runs the configured jq transform via the [jaq](https://github.com/01mf02/jaq)

1457 + library, and merges the resulting prefix-labeled rows into the network-attributes trie.

1458 + supported_platforms:

1459 + include:

1460 + - Linux

1461 + exclude: []

1462 + multi_instance: true

1463 + additional_permissions:

1464 + description: ""

1465 + default_behavior:

1466 + auto_detection:

1467 + description: "Disabled by default. Add an entry under enrichment.network_sources to enable."

1468 + limits:

1469 + description: ""

1470 + performance_impact:

1471 + description: ""

1472 + setup:

1473 + prerequisites:

1474 + list:

1475 + - title: Outbound HTTPS to AWS

1476 + description: |

1477 + The agent host must be able to reach `https://ip-ranges.amazonaws.com/ip-ranges.json`.

1478 + No AWS credentials needed -- the file is public.

1479 + configuration:

1480 + file:

1481 + name: netflow.yaml

1482 + options:

1483 + description: |

1484 + Add a named entry under `enrichment.network_sources`. The `name` you choose appears

1485 + in flow records via the `*_NET_TENANT` field (when your jq transform sets it).

1486 + folding:

1487 + title: Config options

1488 + enabled: true

1489 + list:

1490 + - name: url

1491 + description: AWS publishes the master file at this URL.

1492 + default_value: "https://ip-ranges.amazonaws.com/ip-ranges.json"

1493 + required: true

1494 + - name: interval

1495 + description: How often to fetch. AWS updates the file roughly every 15 minutes; daily is enough for most uses.

1496 + default_value: "60s (loop floor)"

1497 + required: false

1498 + - name: timeout

1499 + description: Per-request timeout.

1500 + default_value: "60s"

1501 + required: false

1502 + - name: transform

1503 + description: jq expression that converts the AWS response into objects with `prefix` and label fields.

1504 + default_value: "."

1505 + required: true

1506 + examples:

1507 + folding:

1508 + title: Config

1509 + enabled: true

1510 + list:

1511 + - name: Tag all AWS prefixes by region and service

1512 + folding:

1513 + enabled: false

1514 + description: Sets tenant=amazon, region=<aws-region>, role=<service-name>.

1515 + config: |

1516 + enrichment:

1517 + network_sources:

1518 + aws:

1519 + url: "https://ip-ranges.amazonaws.com/ip-ranges.json"

1520 + interval: 24h

1521 + timeout: 60s

1522 + transform: |

1523 + (.prefixes + .ipv6_prefixes)[] | {

1524 + prefix: (.ip_prefix // .ipv6_prefix),

1525 + tenant: "amazon",

1526 + region: .region,

1527 + role: (.service | ascii_downcase)

1528 + }

1529 + - name: AWS S3 only

1530 + folding:

1531 + enabled: true

1532 + description: Filter to a single AWS service for narrower tagging.

1533 + config: |

1534 + enrichment:

1535 + network_sources:

1536 + aws-s3:

1537 + url: "https://ip-ranges.amazonaws.com/ip-ranges.json"

1538 + interval: 24h

1539 + transform: |

1540 + (.prefixes + .ipv6_prefixes)[]

1541 + | select(.service == "S3")

1542 + | {

1543 + prefix: (.ip_prefix // .ipv6_prefix),

1544 + tenant: "amazon",

1545 + role: "s3",

1546 + region: .region

1547 + }

1548 + troubleshooting:

1549 + problems:

1550 + list:

1551 + - name: Empty result from the transform is treated as failure

1552 + description: |

1553 + If the jq filter happens to produce nothing (e.g., AWS responds with no prefixes),

1554 + the source backs off as if it errored. Check the journal for `network-sources` warnings.

1555 + - name: TLS verification cannot be disabled

1556 + description: |

1557 + `tls.skip_verify: true` is rejected by validation. Use `tls.ca_file` for

1558 + custom-CA paths if needed.

1559 + alerts: []

1560 + metrics:

1561 + folding:

1562 + title: Metrics

1563 + enabled: false

1564 + description: |

1565 + Tags flow records with `SRC_NET_TENANT`, `*_NET_REGION`, `*_NET_ROLE`, etc. when

1566 + the source or destination IP matches an AWS prefix. Verify on the Network Flows

1567 + tab via the `*_NET_*` columns.

1568 + availability: []

1569 + scopes: []

1570 +

1571 + - meta:

1572 + plugin_name: netflow-plugin

1573 + module_name: gcp-ip-ranges

1574 + monitored_instance:

1575 + name: GCP IP Ranges

1576 + link: https://cloud.google.com/compute/docs/faq#find_ip_range

1577 + categories:

1578 + - flows.network-identity

1579 + icon_filename: network-wired.svg

1580 + keywords:

1581 + - gcp

1582 + - google cloud

1583 + - cloud

1584 + - ip ranges

1585 + - prefix list

1586 + related_resources:

1587 + integrations:

1588 + list:

1589 + - plugin_name: netflow-plugin

1590 + module_name: aws-ip-ranges

1591 + - plugin_name: netflow-plugin

1592 + module_name: azure-ip-ranges

1593 + - plugin_name: netflow-plugin

1594 + module_name: netbox

1595 + - plugin_name: netflow-plugin

1596 + module_name: generic-ipam

1597 + info_provided_to_referring_integrations:

1598 + description: ""

1599 + overview:

1600 + data_collection:

1601 + metrics_description: |

1602 + Google Cloud publishes its public IP prefixes at `https://www.gstatic.com/ipranges/cloud.json`,

1603 + updated periodically. This integration fetches the file and labels flow records

1604 + to/from Google Cloud with `*_NET_TENANT="gcp"` plus per-scope and per-service tags.

1605 +

1606 + For the full network-identity concept, see

1607 + [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

1608 + method_description: |

1609 + Periodic HTTPS GET, jq transform, merge into network-attributes trie. Same mechanism

1610 + as AWS IP Ranges, different URL and JSON shape.

1611 + supported_platforms:

1612 + include:

1613 + - Linux

1614 + exclude: []

1615 + multi_instance: true

1616 + additional_permissions:

1617 + description: ""

1618 + default_behavior:

1619 + auto_detection:

1620 + description: "Disabled by default. Add an entry under enrichment.network_sources to enable."

1621 + limits:

1622 + description: ""

1623 + performance_impact:

1624 + description: ""

1625 + setup:

1626 + prerequisites:

1627 + list:

1628 + - title: Outbound HTTPS to Google

1629 + description: |

1630 + The agent host must be able to reach `https://www.gstatic.com/ipranges/cloud.json`.

1631 + No GCP credentials needed -- the file is public.

1632 + configuration:

1633 + file:

1634 + name: netflow.yaml

1635 + options:

1636 + description: |

1637 + Add a named entry under `enrichment.network_sources`.

1638 + folding:

1639 + title: Config options

1640 + enabled: true

1641 + list:

1642 + - name: url

1643 + description: GCP publishes the master file here.

1644 + default_value: "https://www.gstatic.com/ipranges/cloud.json"

1645 + required: true

1646 + - name: transform

1647 + description: jq expression mapping `prefixes[]` to `prefix` + label objects.

1648 + default_value: "."

1649 + required: true

1650 + examples:

1651 + folding:

1652 + title: Config

1653 + enabled: true

1654 + list:

1655 + - name: Tag all GCP prefixes by service and scope

1656 + folding:

1657 + enabled: false

1658 + description: Sets tenant=gcp, role=<service>, region=<scope>.

1659 + config: |

1660 + enrichment:

1661 + network_sources:

1662 + gcp:

1663 + url: "https://www.gstatic.com/ipranges/cloud.json"

1664 + interval: 24h

1665 + transform: |

1666 + .prefixes[] | {

1667 + prefix: (.ipv4Prefix // .ipv6Prefix),

1668 + tenant: "gcp",

1669 + role: .service,

1670 + region: .scope

1671 + }

1672 + troubleshooting:

1673 + problems:

1674 + list:

1675 + - name: Customer-only ranges

1676 + description: |

1677 + GCP also publishes a `goog.json` file (broader: includes Google services beyond

1678 + cloud). Use `cloud.json` for compute IP attribution; `goog.json` if you also

1679 + want to tag Google's other services.

1680 + alerts: []

1681 + metrics:

1682 + folding:

1683 + title: Metrics

1684 + enabled: false

1685 + description: |

1686 + Tags flow records with `*_NET_TENANT`, `*_NET_REGION`, `*_NET_ROLE`. Verify on the

1687 + Network Flows tab via the `*_NET_*` columns.

1688 + availability: []

1689 + scopes: []

1690 +

1691 + - meta:

1692 + plugin_name: netflow-plugin

1693 + module_name: azure-ip-ranges

1694 + monitored_instance:

1695 + name: Azure IP Ranges

1696 + link: https://www.microsoft.com/en-us/download/details.aspx?id=56519

1697 + categories:

1698 + - flows.network-identity

1699 + icon_filename: network-wired.svg

1700 + keywords:

1701 + - azure

1702 + - microsoft

1703 + - cloud

1704 + - ip ranges

1705 + - service tags

1706 + related_resources:

1707 + integrations:

1708 + list:

1709 + - plugin_name: netflow-plugin

1710 + module_name: aws-ip-ranges

1711 + - plugin_name: netflow-plugin

1712 + module_name: gcp-ip-ranges

1713 + - plugin_name: netflow-plugin

1714 + module_name: netbox

1715 + - plugin_name: netflow-plugin

1716 + module_name: generic-ipam

1717 + info_provided_to_referring_integrations:

1718 + description: ""

1719 + overview:

1720 + data_collection:

1721 + metrics_description: |

1722 + Azure publishes "Service Tags" data describing IP ranges per region and per service.

1723 + The Azure publication mechanism is **less convenient** than AWS / GCP -- the

1724 + authoritative URL contains a date stamp that changes weekly, so you cannot use

1725 + a single stable URL.

1726 +

1727 + For automated fetching, you have two options:

1728 + 1. Mirror the file in your own infrastructure (a script that resolves the latest

1729 + URL via the Azure CLI, downloads, and serves at a stable internal URL).

1730 + 2. Skip Azure IP Ranges entirely and rely on GeoIP / ASN data for Azure

1731 + attribution (Azure ASN is 8075).

1732 +

1733 + For the full network-identity concept, see

1734 + [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

1735 + method_description: |

1736 + Periodic HTTPS GET against your stable mirror URL, jq transform, merge into

1737 + network-attributes trie. The plugin does not handle Azure's date-stamped URL

1738 + rotation -- you provide a stable URL via your own mirror.

1739 + supported_platforms:

1740 + include:

1741 + - Linux

1742 + exclude: []

1743 + multi_instance: true

1744 + additional_permissions:

1745 + description: ""

1746 + default_behavior:

1747 + auto_detection:

1748 + description: "Disabled by default. Set up your own URL mirror, then add an entry under enrichment.network_sources."

1749 + limits:

1750 + description: ""

1751 + performance_impact:

1752 + description: ""

1753 + setup:

1754 + prerequisites:

1755 + list:

1756 + - title: A stable URL for the Azure Service Tags JSON

1757 + description: |

1758 + Azure's authoritative URL embeds a date stamp that changes weekly. A simple

1759 + workaround: a cron job that calls

1760 + `az network list-service-tags --location global -o json` (Azure CLI) and

1761 + writes the result to a stable path on an internal HTTP server. The plugin then

1762 + fetches from that stable URL.

1763 + - title: Outbound HTTPS to your mirror

1764 + description: |

1765 + No Azure credentials needed by the plugin itself; credentials only matter on

1766 + the side that does the upstream Azure CLI call.

1767 + configuration:

1768 + file:

1769 + name: netflow.yaml

1770 + options:

1771 + description: |

1772 + Add a named entry under `enrichment.network_sources` pointing at your mirror URL.

1773 + folding:

1774 + title: Config options

1775 + enabled: true

1776 + list:

1777 + - name: url

1778 + description: Stable URL to your locally-mirrored Azure Service Tags JSON.

1779 + default_value: ""

1780 + required: true

1781 + - name: transform

1782 + description: jq expression mapping the values[] array to per-prefix objects.

1783 + default_value: "."

1784 + required: true

1785 + examples:

1786 + folding:

1787 + title: Config

1788 + enabled: true

1789 + list:

1790 + - name: Internal mirror of Azure Service Tags

1791 + folding:

1792 + enabled: false

1793 + description: Tag every Azure prefix with tenant=azure plus region/service.

1794 + config: |

1795 + enrichment:

1796 + network_sources:

1797 + azure:

1798 + url: "https://internal.example/azure-service-tags.json"

1799 + interval: 24h

1800 + transform: |

1801 + .values[]

1802 + | .id as $id

1803 + | .properties.region as $region

1804 + | (.properties.systemService // "") as $service

1805 + | .properties.addressPrefixes[]

1806 + | {

1807 + prefix: .,

1808 + tenant: "azure",

1809 + region: ($region // ""),

1810 + role: ($service | ascii_downcase)

1811 + }

1812 + troubleshooting:

1813 + problems:

1814 + list:

1815 + - name: Empty results

1816 + description: |

1817 + The Azure Service Tags JSON has nested structure (`values[].properties.addressPrefixes[]`).

1818 + If your jq doesn't unwrap correctly, every fetch yields zero rows and the source

1819 + backs off. Test the jq locally with `jq < azure-service-tags.json`.

1820 + alerts: []

1821 + metrics:

1822 + folding:

1823 + title: Metrics

1824 + enabled: false

1825 + description: |

1826 + Tags flow records to/from Azure with `*_NET_TENANT`, `*_NET_REGION`, `*_NET_ROLE`.

1827 + availability: []

1828 + scopes: []

1829 +

1830 + - meta:

1831 + plugin_name: netflow-plugin

1832 + module_name: netbox

1833 + monitored_instance:

1834 + name: NetBox

1835 + link: https://netboxlabs.com/oss/netbox/

1836 + categories:

1837 + - flows.network-identity

1838 + icon_filename: network-wired.svg

1839 + keywords:

1840 + - netbox

1841 + - ipam

1842 + - dcim

1843 + - source of truth

1844 + - prefix list

1845 + related_resources:

1846 + integrations:

1847 + list:

1848 + - plugin_name: netflow-plugin

1849 + module_name: aws-ip-ranges

1850 + - plugin_name: netflow-plugin

1851 + module_name: gcp-ip-ranges

1852 + - plugin_name: netflow-plugin

1853 + module_name: azure-ip-ranges

1854 + - plugin_name: netflow-plugin

1855 + module_name: generic-ipam

1856 + info_provided_to_referring_integrations:

1857 + description: ""

1858 + overview:

1859 + data_collection:

1860 + metrics_description: |

1861 + [NetBox](https://netboxlabs.com/oss/netbox/) is the most widely deployed open-source

1862 + IPAM / DCIM. Many networks already curate prefix metadata there -- tenant, site,

1863 + role, VRF -- and want flow data to inherit those labels automatically rather than

1864 + duplicating them in `netflow.yaml`.

1865 +

1866 + This integration polls NetBox's Prefixes API at a configurable interval, transforms

1867 + the response with jq, and labels flow records with the prefix metadata.

1868 +

1869 + For the full network-identity concept, see

1870 + [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

1871 + method_description: |

1872 + Periodic HTTPS GET to a NetBox API endpoint with a Bearer token in the

1873 + `Authorization` header. jq transform produces per-prefix objects with the labels

1874 + you want -- typically `tenant.name`, `site.name`, `role.name`, `description`.

1875 +

1876 + NetBox paginates results -- there is **no automatic pagination** in this plugin.

1877 + For inventories larger than the default page size (50), wrap NetBox with a

1878 + server-side aggregator that returns the full list at one URL.

1879 + supported_platforms:

1880 + include:

1881 + - Linux

1882 + exclude: []

1883 + multi_instance: true

1884 + additional_permissions:

1885 + description: ""

1886 + default_behavior:

1887 + auto_detection:

1888 + description: "Disabled by default. Add an entry under enrichment.network_sources with your NetBox URL and API token."

1889 + limits:

1890 + description: ""

1891 + performance_impact:

1892 + description: ""

1893 + setup:

1894 + prerequisites:

1895 + list:

1896 + - title: NetBox API token with read scope on Prefixes

1897 + description: |

1898 + In NetBox, create or use a service account, generate an API token, scope it

1899 + read-only to the Prefixes endpoint. The token goes in the `Authorization` header.

1900 + - title: A bulk endpoint or aggregator

1901 + description: |

1902 + The plugin does not paginate. If your NetBox has more prefixes than fit in the

1903 + default page (`?limit=50`), either raise `limit` (`?limit=10000`) or expose

1904 + an internal endpoint that aggregates all pages and serves them at one URL.

1905 + configuration:

1906 + file:

1907 + name: netflow.yaml

1908 + options:

1909 + description: |

1910 + Add a named entry under `enrichment.network_sources` pointing at your NetBox.

1911 + folding:

1912 + title: Config options

1913 + enabled: true

1914 + list:

1915 + - name: url

1916 + description: NetBox Prefixes API endpoint, with `?limit=` if needed.

1917 + default_value: ""

1918 + required: true

1919 + - name: headers.Authorization

1920 + description: NetBox API token, prefixed with "Token ".

1921 + default_value: ""

1922 + required: true

1923 + - name: interval

1924 + description: How often to refresh. NetBox is your source of truth -- 5-15 minutes is typical.

1925 + default_value: "60s"

1926 + required: false

1927 + - name: transform

1928 + description: jq expression mapping `.results[]` (NetBox's response shape) to per-prefix objects.

1929 + default_value: "."

1930 + required: true

1931 + examples:

1932 + folding:

1933 + title: Config

1934 + enabled: true

1935 + list:

1936 + - name: NetBox with API token and standard label set

1937 + folding:

1938 + enabled: false

1939 + description: Tags prefixes with tenant, site, role, and the NetBox description.

1940 + config: |

1941 + enrichment:

1942 + network_sources:

1943 + netbox:

1944 + url: "https://netbox.example.internal/api/ipam/prefixes/?limit=10000"

1945 + headers:

1946 + Authorization: "Token abcdef0123456789"

1947 + interval: 5m

1948 + timeout: 30s

1949 + transform: |

1950 + .results[] | {

1951 + prefix: .prefix,

1952 + tenant: (.tenant.name // ""),

1953 + site: (.site.name // ""),

1954 + role: (.role.name // ""),

1955 + name: .description

1956 + }

1957 + - name: NetBox with mTLS to internal CA

1958 + folding:

1959 + enabled: true

1960 + description: When NetBox is behind your internal PKI; use tls.ca_file plus client cert.

1961 + config: |

1962 + enrichment:

1963 + network_sources:

1964 + netbox:

1965 + url: "https://netbox.example.internal/api/ipam/prefixes/?limit=10000"

1966 + headers:

1967 + Authorization: "Token abcdef0123456789"

1968 + interval: 5m

1969 + tls:

1970 + enable: true

1971 + ca_file: /etc/netdata/ssl/internal-ca.pem

1972 + cert_file: /etc/netdata/ssl/netdata.crt

1973 + key_file: /etc/netdata/ssl/netdata.key

1974 + transform: |

1975 + .results[] | {

1976 + prefix: .prefix,

1977 + tenant: (.tenant.name // ""),

1978 + site: (.site.name // ""),

1979 + role: (.role.name // ""),

1980 + name: .description

1981 + }

1982 + troubleshooting:

1983 + problems:

1984 + list:

1985 + - name: Only first page of results loaded

1986 + description: |

1987 + NetBox paginates by default at 50 results. The plugin does not follow `next`

1988 + links. Use `?limit=10000` (or the actual count) on the URL, or expose an

1989 + aggregating endpoint server-side.

1990 + - name: Token missing or wrong scope

1991 + description: |

1992 + NetBox returns 403 silently consumed by the plugin's HTTP error path. Watch

1993 + the journal for `network-sources` warnings; verify with curl:

1994 + `curl -H "Authorization: Token <tok>" https://netbox/api/ipam/prefixes/`.

1995 + alerts: []

1996 + metrics:

1997 + folding:

1998 + title: Metrics

1999 + enabled: false

2000 + description: |

2001 + Tags flow records with the labels you mapped from NetBox -- typically `*_NET_TENANT`,

2002 + `*_NET_SITE`, `*_NET_ROLE`, `*_NET_NAME`. Verify on the Network Flows tab.

2003 + availability: []

2004 + scopes: []

2005 +

2006 + - meta:

2007 + plugin_name: netflow-plugin

2008 + module_name: generic-ipam

2009 + monitored_instance:

2010 + name: Generic JSON-over-HTTP IPAM

2011 + link: https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity

2012 + categories:

2013 + - flows.network-identity

2014 + icon_filename: network-wired.svg

2015 + keywords:

2016 + - ipam

2017 + - cmdb

2018 + - infoblox

2019 + - bluecat

2020 + - phpipam

2021 + - custom

2022 + - prefix list

2023 + related_resources:

2024 + integrations:

2025 + list:

2026 + - plugin_name: netflow-plugin

2027 + module_name: aws-ip-ranges

2028 + - plugin_name: netflow-plugin

2029 + module_name: gcp-ip-ranges

2030 + - plugin_name: netflow-plugin

2031 + module_name: azure-ip-ranges

2032 + - plugin_name: netflow-plugin

2033 + module_name: netbox

2034 + info_provided_to_referring_integrations:

2035 + description: ""

2036 + overview:

2037 + data_collection:

2038 + metrics_description: |

2039 + The catch-all integration. Any IPAM, CMDB, or service that exposes prefix metadata

2040 + via an HTTP-fetchable JSON endpoint can plug into Netdata's flow enrichment via this

2041 + mechanism. Examples: Infoblox WAPI, BlueCat REST API, phpIPAM, internal-built CMDB

2042 + endpoints, ServiceNow CMDB queries, custom Lambda functions producing JSON.

2043 +

2044 + You define the URL, the HTTP method, headers (for auth), and a jq transform that

2045 + converts the response into objects with `prefix` + label fields.

2046 +

2047 + For the full network-identity concept, see

2048 + [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment/network-identity).

2049 + method_description: |

2050 + Periodic HTTPS GET (or POST) to a configured URL with optional headers, optional

2051 + custom CA / mTLS, jq transform of the response, merge into network-attributes trie.

2052 + supported_platforms:

2053 + include:

2054 + - Linux

2055 + exclude: []

2056 + multi_instance: true

2057 + additional_permissions:

2058 + description: ""

2059 + default_behavior:

2060 + auto_detection:

2061 + description: "Disabled by default. Add an entry per IPAM source under enrichment.network_sources."

2062 + limits:

2063 + description: ""

2064 + performance_impact:

2065 + description: ""

2066 + setup:

2067 + prerequisites:

2068 + list:

2069 + - title: An HTTP/HTTPS endpoint returning JSON

2070 + description: |

2071 + The endpoint must produce a parseable JSON document. The plugin only supports

2072 + GET and POST. There is no pagination, no cursor following, no OAuth flow --

2073 + if your IPAM needs those, wrap it in an internal aggregator.

2074 + - title: Authentication via headers

2075 + description: |

2076 + The plugin has no built-in auth helpers. Set whatever the API needs -- bearer

2077 + tokens, basic-auth header, custom API-key headers -- via `headers:`. Store

2078 + tokens carefully; they're written into the YAML.

2079 + configuration:

2080 + file:

2081 + name: netflow.yaml

2082 + options:

2083 + description: |

2084 + Add a named entry under `enrichment.network_sources`. The keys below are the

2085 + full set of options.

2086 + folding:

2087 + title: Config options

2088 + enabled: true

2089 + list:

2090 + - name: url

2091 + description: HTTP/HTTPS endpoint.

2092 + default_value: ""

2093 + required: true

2094 + - name: method

2095 + description: GET or POST.

2096 + default_value: "GET"

2097 + required: false

2098 + - name: interval

2099 + description: Refresh interval (loop floors at 60s).

2100 + default_value: "60s"

2101 + required: false

2102 + - name: timeout

2103 + description: Per-request timeout.

2104 + default_value: "60s"

2105 + required: false

2106 + - name: headers

2107 + description: Map of additional HTTP request headers (e.g., authentication).

2108 + default_value: "{}"

2109 + required: false

2110 + - name: transform

2111 + description: jq expression converting response to {prefix, name?, role?, site?, region?, country?, state?, city?, tenant?, asn?, asn_name?} stream.

2112 + default_value: "."

2113 + required: true

2114 + - name: tls.enable

2115 + description: Use custom TLS settings (custom CA, mTLS).

2116 + default_value: "false"

2117 + required: false

2118 + - name: tls.ca_file

2119 + description: PEM file with the CA bundle.

2120 + default_value: ""

2121 + required: false

2122 + - name: tls.cert_file

2123 + description: PEM file with the client certificate (mTLS).

2124 + default_value: ""

2125 + required: false

2126 + - name: tls.key_file

2127 + description: PEM file with the client private key.

2128 + default_value: ""

2129 + required: false

2130 + examples:

2131 + folding:

2132 + title: Config

2133 + enabled: true

2134 + list:

2135 + - name: phpIPAM with API token

2136 + folding:

2137 + enabled: false

2138 + description: phpIPAM exposes /api/<app>/subnets/. Use the standard transform.

2139 + config: |

2140 + enrichment:

2141 + network_sources:

2142 + phpipam:

2143 + url: "https://ipam.example/api/netdata/subnets/"

2144 + headers:

2145 + token: "abcdef..."

2146 + interval: 10m

2147 + transform: |

2148 + .data[] | {

2149 + prefix: (.subnet + "/" + (.mask|tostring)),

2150 + name: .description,

2151 + tenant: (.custom_tenant // ""),

2152 + site: (.location.name // "")

2153 + }

2154 + - name: Custom internal CMDB (POST with body)

2155 + folding:

2156 + enabled: true

2157 + description: When your CMDB requires POST with a query body. Define `method` and append the body via headers/url. The plugin's body support is limited -- prefer GET endpoints when possible.

2158 + config: |

2159 + enrichment:

2160 + network_sources:

2161 + cmdb:

2162 + url: "https://cmdb.example/query/networks"

2163 + method: POST

2164 + headers:

2165 + Authorization: "Bearer ..."

2166 + Content-Type: "application/json"

2167 + interval: 30m

2168 + transform: |

2169 + .results[] | {

2170 + prefix: .cidr,

2171 + tenant: .organization,

2172 + site: .datacenter,

2173 + role: .purpose

2174 + }

2175 + - name: Internal IPAM with mTLS

2176 + folding:

2177 + enabled: true

2178 + description: When the IPAM is behind your internal PKI.

2179 + config: |

2180 + enrichment:

2181 + network_sources:

2182 + corp_ipam:

2183 + url: "https://ipam.corp/api/networks"

2184 + tls:

2185 + enable: true

2186 + ca_file: /etc/netdata/ssl/corp-ca.pem

2187 + cert_file: /etc/netdata/ssl/netdata.crt

2188 + key_file: /etc/netdata/ssl/netdata.key

2189 + interval: 10m

2190 + transform: |

2191 + .[] | {

2192 + prefix: .cidr,

2193 + name: .label,

2194 + tenant: .tenant

2195 + }

2196 + troubleshooting:

2197 + problems:

2198 + list:

2199 + - name: Endpoint requires pagination

2200 + description: |

2201 + The plugin does not paginate. Either raise the page size to cover your inventory,

2202 + or wrap the endpoint with an internal aggregator that returns all results at one URL.

2203 + - name: TLS verification cannot be disabled

2204 + description: |

2205 + `tls.skip_verify` and `tls.verify: false` are rejected by validation. Use

2206 + `tls.ca_file` to trust internal CAs.

2207 + - name: Empty result back-off

2208 + description: |

2209 + An empty jq result is treated as a fetch failure. If your IPAM legitimately

2210 + returns no prefixes (quiet state), the source backs off as if it errored.

2211 + Workaround: have the upstream return at least one synthetic prefix.

2212 + alerts: []

2213 + metrics:

2214 + folding:

2215 + title: Metrics

2216 + enabled: false

2217 + description: |

2218 + Tags flow records with whatever labels your jq transform emits. Verify on the

2219 + Network Flows tab via the `*_NET_*` columns.

2220 + availability: []

2221 + scopes: []

src/crates/netflow-plugin/src/api/flows/handler.rs

+15 -6

@@ -42,12 +42,21 @@ impl NetflowFlowsHandler {

42 request: query::FlowsRequest,

43 ) -> Result<FlowsFunctionResponse> {

44 if request.is_autocomplete_mode() {

45 - let query_output = self

46 - .query

47 - .autocomplete_field_values(&request)

48 - .map_err(|err| NetdataPluginError::Other {

49 - message: format!("failed to autocomplete facet values: {err:#}"),

50 - })?;

45 + // Substring autocomplete on text fields can stream a full FST

46 + // sidecar; keep it off the async runtime so it does not stall

47 + // tokio workers handling unrelated traffic.

48 + let request_for_query = request.clone();

49 + let query = Arc::clone(&self.query);

50 + let query_output = task::spawn_blocking(move || {

51 + query.autocomplete_field_values(&request_for_query)

52 + })

53 + .await

54 + .map_err(|err| NetdataPluginError::Other {

55 + message: format!("autocomplete task join failed: {err}"),

56 + })?

57 + .map_err(|err| NetdataPluginError::Other {

58 + message: format!("failed to autocomplete facet values: {err:#}"),

59 + })?;

60 let mut stats = self.metrics.snapshot();

61 stats.extend(query_output.stats);

src/crates/netflow-plugin/src/facet_catalog.rs

+24

@@ -12,12 +12,28 @@ pub(crate) enum FacetValueKind {

12 IpAddr,

13 }

15 +/// How the autocomplete dropdown matches user input against stored values.

16 +///

17 +/// This is exclusively about the autocomplete/dropdown UX. Regular facet

18 +/// matching (selections / `key in [values]`) is always exact equality and

19 +/// uses indexes — never a substring scan.

20 +#[derive(Debug, Clone, Copy, PartialEq, Eq)]

21 +pub(crate) enum AutocompleteMatchKind {

22 + /// `value.starts_with(term)`. Cheap on FST sidecars (automaton-driven).

23 + /// Right for IPs (typing `10.0.` to narrow), short numeric values.

24 + Prefix,

25 + /// `value.contains(term)`. Linear scan. Right for free-form labels

26 + /// (e.g. `AS20940 Akamai International` matching a search for `Akamai`).

27 + Substring,

28 +}

29 +

30 #[derive(Debug, Clone, Copy, PartialEq, Eq)]

31 pub(crate) struct FacetFieldSpec {

32 pub(crate) name: &'static str,

33 pub(crate) kind: FacetValueKind,

34 pub(crate) supports_autocomplete: bool,

35 pub(crate) uses_sidecar: bool,

36 + pub(crate) autocomplete_match: AutocompleteMatchKind,

37 }

39 const VIRTUAL_FACET_FIELDS: &[FacetFieldSpec] = &[

@@ -26,12 +42,14 @@ const VIRTUAL_FACET_FIELDS: &[FacetFieldSpec] = &[

42 kind: FacetValueKind::Text,

43 supports_autocomplete: true,

44 uses_sidecar: false,

45 + autocomplete_match: AutocompleteMatchKind::Substring,

46 },

47 FacetFieldSpec {

48 name: "ICMPV6",

49 kind: FacetValueKind::Text,

50 supports_autocomplete: true,

51 uses_sidecar: false,

52 + autocomplete_match: AutocompleteMatchKind::Substring,

53 },

54 ];

@@ -126,6 +144,11 @@ fn facet_field_spec_for_name(field: &'static str) -> Option<FacetFieldSpec> {

144 _ => FacetValueKind::Text,

145 };

146

147 + let autocomplete_match = match kind {

148 + FacetValueKind::Text => AutocompleteMatchKind::Substring,

149 + _ => AutocompleteMatchKind::Prefix,

150 + };

151 +

152 Some(FacetFieldSpec {

153 name: field,

154 kind,

@@ -138,6 +161,7 @@ fn facet_field_spec_for_name(field: &'static str) -> Option<FacetFieldSpec> {

161 | FacetValueKind::SparseU64

162 | FacetValueKind::DenseU16

163 ),

164 + autocomplete_match,

165 })

166 }

167

src/crates/netflow-plugin/src/facet_runtime.rs

+268 -8

@@ -3,8 +3,8 @@ mod sidecar;

3 mod store;

5 use crate::facet_catalog::{

6 - FACET_ALLOWED_OPTIONS, FACET_FIELD_SPECS, FacetFieldSpec, facet_field_spec,

7 - facet_field_spec_static,

6 + AutocompleteMatchKind, FACET_ALLOWED_OPTIONS, FACET_FIELD_SPECS, FacetFieldSpec,

7 + facet_field_spec, facet_field_spec_static,

8 };

9 use crate::flow::FlowRecord;

10 use crate::query::{

@@ -362,6 +362,7 @@ impl FacetRuntime {

362 let Some(spec) = facet_field_spec(&normalized) else {

363 return Ok(Vec::new());

364 };

365 + let match_kind = spec.autocomplete_match;

366

367 let (promoted, mut matches, archived_paths) = {

368 let state = self

@@ -371,13 +372,19 @@ impl FacetRuntime {

372 let Some(published) = state.published.fields.get(normalized.as_str()) else {

373 return Ok(Vec::new());

374 };

374 - let active_matches =

375 - active_autocomplete_matches(&state.active_contributions, normalized.as_str(), term);

375 + let active_matches = active_autocomplete_matches(

376 + &state.active_contributions,

377 + normalized.as_str(),

378 + term,

379 + match_kind,

380 + );

381 let archived_matches = if !spec.uses_sidecar || !published.autocomplete {

382 state

383 .archived_fields

384 .get(normalized.as_str())

380 - .map(|store| store.prefix_matches(term, FACET_AUTOCOMPLETE_LIMIT))

385 + .map(|store| {

386 + store.autocomplete_matches(term, FACET_AUTOCOMPLETE_LIMIT, match_kind)

387 + })

388 .unwrap_or_default()

389 } else {

390 Vec::new()

@@ -400,8 +407,13 @@ impl FacetRuntime {

407 if needed == 0 {

408 break;

409 }

403 - let sidecar_matches =

404 - search_sidecar(Path::new(&path), normalized.as_str(), term, needed)?;

410 + let sidecar_matches = search_sidecar(

411 + Path::new(&path),

412 + normalized.as_str(),

413 + term,

414 + needed,

415 + match_kind,

416 + )?;

417 matches = merge_autocomplete_values(matches, sidecar_matches);

418 if matches.len() >= FACET_AUTOCOMPLETE_LIMIT {

419 break;

@@ -885,6 +897,7 @@ fn active_autocomplete_matches(

897 active_contributions: &BTreeMap<String, FacetFileContribution>,

898 field: &str,

899 term: &str,

900 + match_kind: AutocompleteMatchKind,

901 ) -> Vec<String> {

902 let mut matches = Vec::new();

903 for contribution in active_contributions.values() {

@@ -893,7 +906,7 @@ fn active_autocomplete_matches(

906 };

907 matches = merge_autocomplete_values(

908 matches,

896 - store.prefix_matches(term, FACET_AUTOCOMPLETE_LIMIT),

909 + store.autocomplete_matches(term, FACET_AUTOCOMPLETE_LIMIT, match_kind),

910 );

911 if matches.len() >= FACET_AUTOCOMPLETE_LIMIT {

912 break;

@@ -1204,6 +1217,253 @@ mod tests {

1217 );

1218 }

1219

1220 + #[test]

1221 + fn runtime_autocomplete_text_field_uses_substring_matching() {

1222 + let tmp = tempfile::tempdir().expect("create temp dir");

1223 + let runtime = FacetRuntime::new(tmp.path());

1224 +

1225 + let mut fields = FlowFields::new();

1226 + fields.insert("SRC_AS_NAME", "AS20940 Akamai International".to_string());

1227 + runtime

1228 + .observe_active_contribution(

1229 + Path::new("/tmp/flows-substring-active.journal"),

1230 + &facet_contribution_from_flow_fields(&fields),

1231 + )

1232 + .expect("observe substring contribution");

1233 +

1234 + let mid_token = runtime

1235 + .autocomplete("SRC_AS_NAME", "Akamai")

1236 + .expect("autocomplete by org name");

1237 + assert!(

1238 + mid_token.iter().any(|v| v == "AS20940 Akamai International"),

1239 + "substring search on AS_NAME must match the organisation token; got {mid_token:?}"

1240 + );

1241 +

1242 + let inner_token = runtime

1243 + .autocomplete("SRC_AS_NAME", "kamai")

1244 + .expect("autocomplete by inner substring");

1245 + assert!(

1246 + inner_token

1247 + .iter()

1248 + .any(|v| v == "AS20940 Akamai International"),

1249 + "substring search must match a non-prefix substring; got {inner_token:?}"

1250 + );

1251 +

1252 + let prefix_still_works = runtime

1253 + .autocomplete("SRC_AS_NAME", "AS20940")

1254 + .expect("autocomplete by AS prefix");

1255 + assert!(

1256 + prefix_still_works

1257 + .iter()

1258 + .any(|v| v == "AS20940 Akamai International"),

1259 + "AS-number prefix must still match; got {prefix_still_works:?}"

1260 + );

1261 + }

1262 +

1263 + #[test]

1264 + fn runtime_autocomplete_text_substring_survives_archive_promotion() {

1265 + let tmp = tempfile::tempdir().expect("create temp dir");

1266 + let runtime = FacetRuntime::new(tmp.path());

1267 + let archived_path = Path::new("/tmp/flows-substring-archived.journal");

1268 +

1269 + for asn in 0..120u32 {

1270 + let mut fields = FlowFields::new();

1271 + fields.insert("SRC_AS_NAME", format!("AS{asn:05} Akamai-{asn:03}"));

1272 + runtime

1273 + .observe_active_contribution(

1274 + archived_path,

1275 + &facet_contribution_from_flow_fields(&fields),

1276 + )

1277 + .expect("observe contribution");

1278 + }

1279 +

1280 + runtime

1281 + .observe_rotation(

1282 + archived_path,

1283 + Path::new("/tmp/flows-substring-archived-next.journal"),

1284 + )

1285 + .expect("rotate file");

1286 +

1287 + let results = runtime

1288 + .autocomplete("SRC_AS_NAME", "Akamai")

1289 + .expect("autocomplete via promoted sidecar");

1290 +

1291 + assert!(

1292 + !results.is_empty(),

1293 + "promoted sidecar must return substring matches"

1294 + );

1295 + assert!(

1296 + results.iter().all(|v| v.contains("Akamai")),

1297 + "every result must contain the search term; got {results:?}"

1298 + );

1299 + }

1300 +

1301 + #[test]

1302 + fn runtime_autocomplete_text_substring_matches_across_multiple_sidecars() {

1303 + let tmp = tempfile::tempdir().expect("create temp dir");

1304 + let runtime = FacetRuntime::new(tmp.path());

1305 +

1306 + // Two journals, each promoted to sidecar by inserting >FACET_VALUE_LIMIT

1307 + // distinct values. Each sidecar has only 30 entries that match the

1308 + // search term, so neither alone can fill FACET_AUTOCOMPLETE_LIMIT (100)

1309 + // and the loop must reach both sidecars.

1310 + let paths = [

1311 + Path::new("/tmp/flows-multi-a.journal"),

1312 + Path::new("/tmp/flows-multi-b.journal"),

1313 + ];

1314 + for (idx, journal_path) in paths.iter().enumerate() {

1315 + for asn in 0..120u32 {

1316 + let mut fields = FlowFields::new();

1317 + let unique_asn = asn + (idx as u32) * 1000;

1318 + let label = if asn < 30 {

1319 + format!("AS{unique_asn:05} Provider-{idx}-{asn:03}")

1320 + } else {

1321 + format!("AS{unique_asn:05} Filler-{idx}-{asn:03}")

1322 + };

1323 + fields.insert("SRC_AS_NAME", label);

1324 + runtime

1325 + .observe_active_contribution(

1326 + journal_path,

1327 + &facet_contribution_from_flow_fields(&fields),

1328 + )

1329 + .expect("observe contribution");

1330 + }

1331 + runtime

1332 + .observe_rotation(

1333 + journal_path,

1334 + Path::new(&format!("/tmp/flows-multi-{idx}-rot.journal")),

1335 + )

1336 + .expect("rotate file");

1337 + }

1338 +

1339 + let results = runtime

1340 + .autocomplete("SRC_AS_NAME", "Provider-")

1341 + .expect("autocomplete across sidecars");

1342 +

1343 + let from_a = results.iter().any(|v| v.contains("Provider-0-"));

1344 + let from_b = results.iter().any(|v| v.contains("Provider-1-"));

1345 + assert!(

1346 + from_a && from_b,

1347 + "expected hits from both sidecars; got {results:?}"

1348 + );

1349 + assert!(

1350 + results.iter().all(|v| v.contains("Provider-")),

1351 + "every result must match the search term; got {results:?}"

1352 + );

1353 + }

1354 +

1355 + #[test]

1356 + fn runtime_autocomplete_text_empty_term_returns_some_values() {

1357 + let tmp = tempfile::tempdir().expect("create temp dir");

1358 + let runtime = FacetRuntime::new(tmp.path());

1359 +

1360 + for asn in 0..5u32 {

1361 + let mut fields = FlowFields::new();

1362 + fields.insert("SRC_AS_NAME", format!("AS{asn:05} EXAMPLE-{asn}"));

1363 + runtime

1364 + .observe_active_contribution(

1365 + Path::new("/tmp/flows-empty-term.journal"),

1366 + &facet_contribution_from_flow_fields(&fields),

1367 + )

1368 + .expect("observe contribution");

1369 + }

1370 +

1371 + let results = runtime

1372 + .autocomplete("SRC_AS_NAME", "")

1373 + .expect("autocomplete with empty term");

1374 + assert_eq!(

1375 + results.len(),

1376 + 5,

1377 + "empty term must surface every value (limited by FACET_AUTOCOMPLETE_LIMIT); got {results:?}"

1378 + );

1379 + }

1380 +

1381 + #[test]

1382 + fn runtime_autocomplete_text_long_term_returns_no_match_without_panicking() {

1383 + let tmp = tempfile::tempdir().expect("create temp dir");

1384 + let runtime = FacetRuntime::new(tmp.path());

1385 +

1386 + let mut fields = FlowFields::new();

1387 + fields.insert("SRC_AS_NAME", "AS20940 Akamai International".to_string());

1388 + runtime

1389 + .observe_active_contribution(

1390 + Path::new("/tmp/flows-long-term.journal"),

1391 + &facet_contribution_from_flow_fields(&fields),

1392 + )

1393 + .expect("observe contribution");

1394 +

1395 + let term = "x".repeat(200);

1396 + let results = runtime

1397 + .autocomplete("SRC_AS_NAME", &term)

1398 + .expect("autocomplete with long term");

1399 + assert!(results.is_empty(), "got {results:?}");

1400 + }

1401 +

1402 + #[test]

1403 + fn runtime_autocomplete_promotion_threshold_at_exact_limit() {

1404 + let tmp = tempfile::tempdir().expect("create temp dir");

1405 + let runtime = FacetRuntime::new(tmp.path());

1406 + let archived_path = Path::new("/tmp/flows-threshold.journal");

1407 +

1408 + // FACET_VALUE_LIMIT == 100; promotion fires only when total_values > 100.

1409 + // Insert exactly 100 values: should NOT promote, archived in-memory store wins.

1410 + for asn in 0..100u32 {

1411 + let mut fields = FlowFields::new();

1412 + fields.insert("SRC_AS_NAME", format!("AS{asn:05} Provider-{asn:03}"));

1413 + runtime

1414 + .observe_active_contribution(

1415 + archived_path,

1416 + &facet_contribution_from_flow_fields(&fields),

1417 + )

1418 + .expect("observe contribution");

1419 + }

1420 + runtime

1421 + .observe_rotation(archived_path, Path::new("/tmp/flows-threshold-rot.journal"))

1422 + .expect("rotate file");

1423 +

1424 + let results = runtime

1425 + .autocomplete("SRC_AS_NAME", "Provider-")

1426 + .expect("autocomplete at threshold");

1427 + assert!(

1428 + !results.is_empty(),

1429 + "exact-threshold archived-in-memory path must still match"

1430 + );

1431 + assert!(

1432 + results.iter().all(|v| v.contains("Provider-")),

1433 + "got {results:?}"

1434 + );

1435 + }

1436 +

1437 + #[test]

1438 + fn runtime_autocomplete_ip_field_keeps_prefix_matching() {

1439 + let tmp = tempfile::tempdir().expect("create temp dir");

1440 + let runtime = FacetRuntime::new(tmp.path());

1441 +

1442 + for last in 0..3u32 {

1443 + let mut fields = FlowFields::new();

1444 + fields.insert("SRC_ADDR", format!("10.0.0.{last}"));

1445 + runtime

1446 + .observe_active_contribution(

1447 + Path::new("/tmp/flows-ip-prefix.journal"),

1448 + &facet_contribution_from_flow_fields(&fields),

1449 + )

1450 + .expect("observe ip contribution");

1451 + }

1452 +

1453 + let prefix_hit = runtime

1454 + .autocomplete("SRC_ADDR", "10.0.")

1455 + .expect("autocomplete by ip prefix");

1456 + assert_eq!(prefix_hit.len(), 3, "got {prefix_hit:?}");

1457 +

1458 + let middle_octet_miss = runtime

1459 + .autocomplete("SRC_ADDR", "0.0.0")

1460 + .expect("autocomplete by middle substring");

1461 + assert!(

1462 + middle_octet_miss.is_empty(),

1463 + "IP autocomplete must remain prefix-only; got {middle_octet_miss:?}"

1464 + );

1465 + }

1466 +

1467 #[test]

1468 fn incremental_active_updates_match_full_published_rebuild() {

1469 let mut state = FacetState::new();

src/crates/netflow-plugin/src/facet_runtime/sidecar.rs

+29 -9

@@ -1,7 +1,8 @@

1 use super::FacetFileContribution;

2 -use crate::facet_catalog::FACET_FIELD_SPECS;

2 +use crate::facet_catalog::{AutocompleteMatchKind, FACET_FIELD_SPECS};

3 use anyhow::{Context, Result};

4 use fst::{Automaton, IntoStreamer, Set, SetBuilder, Streamer, automaton::Str};

5 +use memchr::memmem::Finder;

6 use memmap2::Mmap;

7 use std::fs::{self, File};

8 use std::io::BufWriter;

@@ -42,6 +43,7 @@ pub(crate) fn search_sidecar(

43 field: &str,

44 term: &str,

45 limit: usize,

46 + match_kind: AutocompleteMatchKind,

47 ) -> Result<Vec<String>> {

48 let sidecar = sidecar_path(journal_path, field);

49 if !sidecar.exists() {

@@ -54,15 +56,33 @@ pub(crate) fn search_sidecar(

56 .with_context(|| format!("failed to mmap facet sidecar {}", sidecar.display()))?;

57 let set = Set::new(mmap)

58 .with_context(|| format!("failed to load facet sidecar {}", sidecar.display()))?;

57 - let matcher = Str::new(term).starts_with();

58 - let mut stream = set.search(&matcher).into_stream();

59 - let mut out = Vec::new();

61 - while let Some(key) = stream.next() {

62 - let rendered = String::from_utf8_lossy(key).into_owned();

63 - out.push(rendered);

64 - if out.len() >= limit {

65 - break;

60 + let mut out = Vec::new();

61 + match match_kind {

62 + AutocompleteMatchKind::Prefix => {

63 + // FST automaton can prune the trie; cheap.

64 + let matcher = Str::new(term).starts_with();

65 + let mut stream = set.search(&matcher).into_stream();

66 + while let Some(key) = stream.next() {

67 + out.push(String::from_utf8_lossy(key).into_owned());

68 + if out.len() >= limit {

69 + break;

70 + }

71 + }

72 + }

73 + AutocompleteMatchKind::Substring => {

74 + // FST has no substring automaton; stream every key and use a

75 + // SIMD-accelerated finder. Bounded by `limit` early-stop.

76 + let finder = Finder::new(term.as_bytes());

77 + let mut stream = set.stream();

78 + while let Some(key) = stream.next() {

79 + if finder.find(key).is_some() {

80 + out.push(String::from_utf8_lossy(key).into_owned());

81 + if out.len() >= limit {

82 + break;

83 + }

84 + }

85 + }

86 }

87 }

src/crates/netflow-plugin/src/facet_runtime/store.rs

+77 -22

@@ -1,4 +1,4 @@

1 -use crate::facet_catalog::FacetValueKind;

1 +use crate::facet_catalog::{AutocompleteMatchKind, FacetValueKind};

2 use allocative::{Allocative, Key, Visitor};

3 use bitvec::prelude::*;

4 use hashbrown::HashTable;

@@ -270,14 +270,22 @@ impl FacetStore {

270 }

271 }

272

273 - pub(super) fn prefix_matches(&self, prefix: &str, limit: usize) -> Vec<String> {

273 + /// Autocomplete (dropdown) matching, parameterised by the per-field

274 + /// policy in `FacetFieldSpec::autocomplete_match`. Regular facet

275 + /// matching (selections) is exact equality and never reaches this path.

276 + pub(super) fn autocomplete_matches(

277 + &self,

278 + term: &str,

279 + limit: usize,

280 + match_kind: AutocompleteMatchKind,

281 + ) -> Vec<String> {

282 match self {

275 - Self::Text(store) => store.prefix_matches(prefix, limit),

276 - Self::DenseU8(store) => store.prefix_matches(prefix, limit),

277 - Self::DenseU16(store) => store.prefix_matches(prefix, limit),

278 - Self::SparseU32(store) => prefix_match_roaring(store, prefix, limit),

279 - Self::SparseU64(store) => prefix_match_roaring(store, prefix, limit),

280 - Self::IpAddr(store) => store.prefix_matches(prefix, limit),

283 + Self::Text(store) => store.autocomplete_matches(term, limit, match_kind),

284 + Self::DenseU8(store) => store.autocomplete_matches(term, limit, match_kind),

285 + Self::DenseU16(store) => store.autocomplete_matches(term, limit, match_kind),

286 + Self::SparseU32(store) => autocomplete_match_roaring(store, term, limit, match_kind),

287 + Self::SparseU64(store) => autocomplete_match_roaring(store, term, limit, match_kind),

288 + Self::IpAddr(store) => store.autocomplete_matches(term, limit, match_kind),

289 }

290 }

291

@@ -387,11 +395,20 @@ impl<const N: usize> DenseBitSet<N> {

395 values

396 }

397

390 - fn prefix_matches(&self, prefix: &str, limit: usize) -> Vec<String> {

398 + fn autocomplete_matches(

399 + &self,

400 + term: &str,

401 + limit: usize,

402 + match_kind: AutocompleteMatchKind,

403 + ) -> Vec<String> {

404 let mut values = Vec::new();

405 for value in self.iter_indices() {

406 let rendered = value.to_string();

394 - if rendered.starts_with(prefix) {

407 + let hit = match match_kind {

408 + AutocompleteMatchKind::Prefix => rendered.starts_with(term),

409 + AutocompleteMatchKind::Substring => rendered.contains(term),

410 + };

411 + if hit {

412 values.push(rendered);

413 if values.len() >= limit {

414 break;

@@ -500,14 +517,34 @@ impl TextValueStore {

517 }

518 }

519

503 - fn prefix_matches(&self, prefix: &str, limit: usize) -> Vec<String> {

520 + fn autocomplete_matches(

521 + &self,

522 + term: &str,

523 + limit: usize,

524 + match_kind: AutocompleteMatchKind,

525 + ) -> Vec<String> {

526 let mut values = Vec::new();

505 - for field_id in 0..self.entries.len() {

506 - let Some(value) = self.value_str(field_id as u32) else {

507 - continue;

508 - };

509 - if value.starts_with(prefix) {

510 - values.push(value.to_string());

527 + match match_kind {

528 + AutocompleteMatchKind::Prefix => {

529 + for field_id in 0..self.entries.len() {

530 + let Some(value) = self.value_str(field_id as u32) else {

531 + continue;

532 + };

533 + if value.starts_with(term) {

534 + values.push(value.to_string());

535 + }

536 + }

537 + }

538 + AutocompleteMatchKind::Substring => {

539 + let finder = memchr::memmem::Finder::new(term.as_bytes());

540 + for field_id in 0..self.entries.len() {

541 + let Some(value) = self.value_str(field_id as u32) else {

542 + continue;

543 + };

544 + if finder.find(value.as_bytes()).is_some() {

545 + values.push(value.to_string());

546 + }

547 + }

548 }

549 }

550 values.sort_unstable();

@@ -645,17 +682,26 @@ impl IpValueStore {

682 values

683 }

684

648 - fn prefix_matches(&self, prefix: &str, limit: usize) -> Vec<String> {

685 + fn autocomplete_matches(

686 + &self,

687 + term: &str,

688 + limit: usize,

689 + match_kind: AutocompleteMatchKind,

690 + ) -> Vec<String> {

691 let mut values = Vec::new();

692 + let predicate = |rendered: &str| match match_kind {

693 + AutocompleteMatchKind::Prefix => rendered.starts_with(term),

694 + AutocompleteMatchKind::Substring => rendered.contains(term),

695 + };

696 for value in self.v4_values.iter() {

697 let rendered = IpAddr::V4(Ipv4Addr::from((value as u32).to_be_bytes())).to_string();

652 - if rendered.starts_with(prefix) {

698 + if predicate(&rendered) {

699 values.push(rendered);

700 }

701 }

702 for value in &self.v6_values {

703 let rendered = IpAddr::V6(Ipv6Addr::from(*value)).to_string();

658 - if rendered.starts_with(prefix) {

704 + if predicate(&rendered) {

705 values.push(rendered);

706 }

707 }

@@ -745,11 +791,20 @@ fn collect_roaring_strings(bitmap: &RoaringTreemap, limit: Option<usize>) -> Vec

791 values

792 }

793

748 -fn prefix_match_roaring(bitmap: &RoaringTreemap, prefix: &str, limit: usize) -> Vec<String> {

794 +fn autocomplete_match_roaring(

795 + bitmap: &RoaringTreemap,

796 + term: &str,

797 + limit: usize,

798 + match_kind: AutocompleteMatchKind,

799 +) -> Vec<String> {

800 let mut values = Vec::new();

801 for value in bitmap.iter() {

802 let rendered = value.to_string();

752 - if rendered.starts_with(prefix) {

803 + let hit = match match_kind {

804 + AutocompleteMatchKind::Prefix => rendered.starts_with(term),

805 + AutocompleteMatchKind::Substring => rendered.contains(term),

806 + };

807 + if hit {

808 values.push(rendered);

809 if values.len() >= limit {

810 break;

src/crates/netflow-plugin/src/ingest_resource_bench_support.rs

+47 -2

@@ -19,6 +19,7 @@ pub(super) struct ResourceEnvelopeReport {

19 pub(super) methodology: String,

20 pub(super) layer: String,

21 pub(super) profile: String,

22 + pub(super) protocol: String,

23 pub(super) requested_flows_per_sec: u64,

24 pub(super) achieved_flows_per_sec: f64,

25 pub(super) cpu_percent_of_one_core: f64,

@@ -72,6 +73,7 @@ pub(super) fn print_resource_report(report: &ResourceEnvelopeReport) {

73 eprintln!();

74 eprintln!("Layer: {}", report.layer);

75 eprintln!("Profile: {}", report.profile);

76 + eprintln!("Protocol: {}", report.protocol);

77 eprintln!(

78 " offered load: {} flows/s",

79 report.requested_flows_per_sec

@@ -170,16 +172,59 @@ fn bytes_to_mib(bytes: u64) -> f64 {

172 bytes as f64 / (1024.0 * 1024.0)

173 }

174

175 +#[derive(Debug, Clone, Serialize, Deserialize)]

176 +pub(super) struct StorageFootprintSample {

177 + pub(super) elapsed_secs: u64,

178 + pub(super) raw_dir_bytes: u64,

179 + pub(super) minute_1_dir_bytes: u64,

180 + pub(super) minute_5_dir_bytes: u64,

181 + pub(super) hour_1_dir_bytes: u64,

182 + pub(super) total_disk_bytes: u64,

183 + pub(super) cumulative_io_write_bytes: u64,

184 + pub(super) cumulative_logical_bytes: u64,

185 + pub(super) cumulative_flows_ingested: u64,

186 + pub(super) rss_bytes: u64,

187 +}

188 +

189 +#[derive(Debug, Clone, Serialize, Deserialize)]

190 +pub(super) struct StorageFootprintReport {

191 + pub(super) protocol: String,

192 + pub(super) profile: String,

193 + pub(super) flows_per_sec: u64,

194 + pub(super) duration_secs: u64,

195 + pub(super) sample_interval_secs: u64,

196 + pub(super) samples: Vec<StorageFootprintSample>,

197 + pub(super) final_total_flows: u64,

198 + pub(super) final_disk_bytes: u64,

199 + pub(super) final_logical_bytes: u64,

200 + pub(super) final_io_write_bytes: u64,

201 +}

202 +

203 #[allow(dead_code)]

204 +pub(super) fn parse_storage_child_report(output: &std::process::Output) -> StorageFootprintReport {

205 + let stdout = String::from_utf8_lossy(&output.stdout);

206 + let stderr = String::from_utf8_lossy(&output.stderr);

207 + let combined = format!("{stdout}\n{stderr}");

208 + let json = combined

209 + .lines()

210 + .find_map(|line| {

211 + line.split_once("STORAGE_BENCH_RESULT:")

212 + .map(|(_, json)| json)

213 + })

214 + .unwrap_or_else(|| panic!("storage bench child did not emit result\n{combined}"));

215 + serde_json::from_str(json)

216 + .unwrap_or_else(|err| panic!("parse storage bench result JSON: {err}\n{combined}"))

217 +}

218 +

219 pub(super) fn journal_dir_size_bytes(path: &Path) -> u64 {

220 let mut total = 0_u64;

221 if let Ok(entries) = fs::read_dir(path) {

222 for entry in entries.flatten() {

223 let path = entry.path();

224 if path.is_dir() {

180 - total += journal_dir_size_bytes(&path);

225 + total = total.saturating_add(journal_dir_size_bytes(&path));

226 } else if let Ok(meta) = entry.metadata() {

182 - total += meta.len();

227 + total = total.saturating_add(meta.len());

228 }

229 }

230 }

src/crates/netflow-plugin/src/ingest_resource_bench_tests.rs

+138 -9

@@ -1,9 +1,10 @@

1 use super::bench_support::{

2 - CARDINALITY_SOURCE_SCENARIO, CardinalityMode, build_cardinality_record_batches,

3 - collect_decoded_record_batches,

2 + CARDINALITY_SOURCE_SCENARIO, CardinalityMode, PROTOCOL_SCENARIOS, ProtocolScenario,

3 + build_cardinality_record_batches, collect_decoded_record_batches,

4 };

5 use super::resource_bench_support::{

6 - ResourceEnvelopeReport, cpu_percent_of_one_core, parse_child_report, print_resource_report,

6 + ResourceEnvelopeReport, StorageFootprintReport, StorageFootprintSample,

7 + cpu_percent_of_one_core, journal_dir_size_bytes, parse_child_report, print_resource_report,

8 take_proc_snapshot,

9 };

10 use super::test_support::{new_disk_benchmark_ingest_service, new_disk_benchmark_raw_log};

@@ -15,11 +16,17 @@ use std::time::{Duration, Instant};

16 const CHILD_ENV: &str = "NETFLOW_RESOURCE_BENCH_CHILD";

17 const LAYER_ENV: &str = "NETFLOW_RESOURCE_BENCH_LAYER";

18 const PROFILE_ENV: &str = "NETFLOW_RESOURCE_BENCH_PROFILE";

19 +const PROTOCOL_ENV: &str = "NETFLOW_RESOURCE_BENCH_PROTOCOL";

20 const RATE_ENV: &str = "NETFLOW_RESOURCE_BENCH_FLOWS_PER_SEC";

21 const WARMUP_ENV: &str = "NETFLOW_RESOURCE_BENCH_WARMUP_SECS";

22 const MEASURE_ENV: &str = "NETFLOW_RESOURCE_BENCH_MEASURE_SECS";

23 const HIGH_POOL_ENV: &str = "NETFLOW_RESOURCE_BENCH_HIGH_POOL_FLOWS";

24 const LOW_POOL_ENV: &str = "NETFLOW_RESOURCE_BENCH_LOW_POOL_FLOWS";

25 +const STORAGE_DURATION_ENV: &str = "NETFLOW_STORAGE_BENCH_DURATION_SECS";

26 +const STORAGE_SAMPLE_ENV: &str = "NETFLOW_STORAGE_BENCH_SAMPLE_INTERVAL_SECS";

27 +const DEFAULT_STORAGE_DURATION_SECS: u64 = 900;

28 +const DEFAULT_STORAGE_SAMPLE_INTERVAL_SECS: u64 = 30;

29 +const DEFAULT_STORAGE_FLOWS_PER_SEC: u64 = 10_000;

31 const DEFAULT_WARMUP_SECS: u64 = 5;

32 const DEFAULT_MEASURE_SECS: u64 = 15;

@@ -147,6 +154,103 @@ fn bench_resource_envelope_child() {

154 );

155 }

156

157 +#[test]

158 +#[ignore = "manual storage footprint benchmark child helper"]

159 +fn bench_storage_footprint_child() {

160 + if std::env::var_os(CHILD_ENV).is_none() {

161 + return;

162 + }

163 +

164 + let report = run_storage_footprint_child();

165 + println!(

166 + "STORAGE_BENCH_RESULT:{}",

167 + serde_json::to_string(&report).expect("serialize storage benchmark result")

168 + );

169 +}

170 +

171 +fn run_storage_footprint_child() -> StorageFootprintReport {

172 + let profile = ResourceProfile::from_env();

173 + let flows_per_sec = env_u64(RATE_ENV, DEFAULT_STORAGE_FLOWS_PER_SEC);

174 + let duration_secs = env_u64(STORAGE_DURATION_ENV, DEFAULT_STORAGE_DURATION_SECS);

175 + let sample_interval_secs = env_u64(

176 + STORAGE_SAMPLE_ENV,

177 + DEFAULT_STORAGE_SAMPLE_INTERVAL_SECS,

178 + )

179 + .max(1);

180 +

181 + let (record_batches, protocol_name) = build_record_batches(profile);

182 + let (_tmp, mut service) = new_disk_benchmark_ingest_service(ConfigDecapsulationMode::None);

183 + let layer = ResourceLayer::AllTiersBatched;

184 + configure_service_for_layer(&mut service, layer);

185 +

186 + let raw_dir = service.cfg.journal.raw_tier_dir();

187 + let m1_dir = service.cfg.journal.minute_1_tier_dir();

188 + let m5_dir = service.cfg.journal.minute_5_tier_dir();

189 + let h1_dir = service.cfg.journal.hour_1_tier_dir();

190 +

191 + let proc_initial = take_proc_snapshot();

192 + let metrics_initial = service.metrics.snapshot();

193 + let started = Instant::now();

194 +

195 + let mut samples = Vec::new();

196 + let mut entries_since_sync = 0_usize;

197 + let mut total_flows_ingested = 0_u64;

198 + let total_intervals = duration_secs.div_ceil(sample_interval_secs);

199 +

200 + for _ in 0..total_intervals {

201 + let segment = run_paced_plugin_loop(

202 + &mut service,

203 + &record_batches,

204 + flows_per_sec,

205 + Duration::from_secs(sample_interval_secs),

206 + entries_since_sync,

207 + false,

208 + );

209 + entries_since_sync = segment.entries_since_sync;

210 + total_flows_ingested =

211 + total_flows_ingested.saturating_add(segment.ingested_flows as u64);

212 +

213 + let elapsed = started.elapsed().as_secs();

214 + let proc_now = take_proc_snapshot();

215 + let metrics_now = service.metrics.snapshot();

216 + let raw_bytes = journal_dir_size_bytes(&raw_dir);

217 + let m1_bytes = journal_dir_size_bytes(&m1_dir);

218 + let m5_bytes = journal_dir_size_bytes(&m5_dir);

219 + let h1_bytes = journal_dir_size_bytes(&h1_dir);

220 +

221 + samples.push(StorageFootprintSample {

222 + elapsed_secs: elapsed,

223 + raw_dir_bytes: raw_bytes,

224 + minute_1_dir_bytes: m1_bytes,

225 + minute_5_dir_bytes: m5_bytes,

226 + hour_1_dir_bytes: h1_bytes,

227 + total_disk_bytes: raw_bytes + m1_bytes + m5_bytes + h1_bytes,

228 + cumulative_io_write_bytes: proc_now

229 + .write_bytes

230 + .saturating_sub(proc_initial.write_bytes),

231 + cumulative_logical_bytes: total_logical_bytes_delta(&metrics_initial, &metrics_now),

232 + cumulative_flows_ingested: total_flows_ingested,

233 + rss_bytes: proc_now.rss_bytes,

234 + });

235 + }

236 +

237 + service.finish_shutdown_for_test(entries_since_sync);

238 +

239 + let last = samples.last().expect("at least one storage sample");

240 + StorageFootprintReport {

241 + protocol: protocol_name.to_string(),

242 + profile: profile.label().to_string(),

243 + flows_per_sec,

244 + duration_secs,

245 + sample_interval_secs,

246 + final_total_flows: last.cumulative_flows_ingested,

247 + final_disk_bytes: last.total_disk_bytes,

248 + final_logical_bytes: last.cumulative_logical_bytes,

249 + final_io_write_bytes: last.cumulative_io_write_bytes,

250 + samples: samples.clone(),

251 + }

252 +}

253 +

254 fn run_resource_envelope_case(

255 layer: ResourceLayer,

256 profile: ResourceProfile,

@@ -177,6 +281,10 @@ fn run_resource_envelope_case(

281 },

282 )

283 .env(RATE_ENV, flows_per_sec.to_string())

284 + .env(

285 + PROTOCOL_ENV,

286 + std::env::var(PROTOCOL_ENV).unwrap_or_else(|_| "mixed".to_string()),

287 + )

288 .env(

289 WARMUP_ENV,

290 env_u64(WARMUP_ENV, DEFAULT_WARMUP_SECS).to_string(),

@@ -229,13 +337,32 @@ fn run_resource_envelope_child() -> ResourceEnvelopeReport {

337 }

338 }

339

232 -fn build_record_batches(profile: ResourceProfile) -> Vec<Vec<crate::flow::FlowRecord>> {

233 - let source_batches = collect_decoded_record_batches(&CARDINALITY_SOURCE_SCENARIO);

234 - build_cardinality_record_batches(

340 +fn build_record_batches(

341 + profile: ResourceProfile,

342 +) -> (Vec<Vec<crate::flow::FlowRecord>>, &'static str) {

343 + let scenario = resolve_source_scenario();

344 + let source_batches = collect_decoded_record_batches(scenario);

345 + let batches = build_cardinality_record_batches(

346 &source_batches,

347 profile.record_pool_size(),

348 profile.cardinality_mode(),

238 - )

349 + );

350 + (batches, scenario.name)

351 +}

352 +

353 +fn resolve_source_scenario() -> &'static ProtocolScenario {

354 + match std::env::var(PROTOCOL_ENV) {

355 + Err(_) => &CARDINALITY_SOURCE_SCENARIO,

356 + Ok(value) if value.is_empty() || value == "mixed" => &CARDINALITY_SOURCE_SCENARIO,

357 + Ok(value) => PROTOCOL_SCENARIOS

358 + .iter()

359 + .find(|scenario| scenario.name == value)

360 + .unwrap_or_else(|| {

361 + panic!(

362 + "unsupported {PROTOCOL_ENV}={value} (expected: mixed, netflow-v5, netflow-v9, ipfix, sflow)"

363 + )

364 + }),

365 + }

366 }

367

368 fn run_writer_only_resource_envelope(

@@ -244,7 +371,7 @@ fn run_writer_only_resource_envelope(

371 warmup_secs: u64,

372 measurement_secs: u64,

373 ) -> ResourceEnvelopeReport {

247 - let record_batches = build_record_batches(profile);

374 + let (record_batches, protocol_name) = build_record_batches(profile);

375 let (_tmp, mut log) = new_disk_benchmark_raw_log();

376 let mut encode_buf = JournalEncodeBuffer::new();

377

@@ -274,6 +401,7 @@ fn run_writer_only_resource_envelope(

401 .to_string(),

402 layer: ResourceLayer::WriterOnly.label().to_string(),

403 profile: profile.label().to_string(),

404 + protocol: protocol_name.to_string(),

405 requested_flows_per_sec: flows_per_sec,

406 achieved_flows_per_sec: measurement_result.ingested_flows as f64 / elapsed.as_secs_f64(),

407 cpu_percent_of_one_core: cpu_percent_of_one_core(proc_before, proc_after, elapsed),

@@ -308,7 +436,7 @@ fn run_plugin_resource_envelope(

436 warmup_secs: u64,

437 measurement_secs: u64,

438 ) -> ResourceEnvelopeReport {

311 - let record_batches = build_record_batches(profile);

439 + let (record_batches, protocol_name) = build_record_batches(profile);

440 let (_tmp, mut service) = new_disk_benchmark_ingest_service(ConfigDecapsulationMode::None);

441 configure_service_for_layer(&mut service, layer);

442

@@ -369,6 +497,7 @@ fn run_plugin_resource_envelope(

497 .to_string(),

498 layer: layer.label().to_string(),

499 profile: profile.label().to_string(),

500 + protocol: protocol_name.to_string(),

501 requested_flows_per_sec: flows_per_sec,

502 achieved_flows_per_sec: measurement_result.ingested_flows as f64 / elapsed.as_secs_f64(),

503 cpu_percent_of_one_core: cpu_percent_of_one_core(proc_before, proc_after, elapsed),

src/crates/netflow-plugin/src/query/request/constants.rs

@@ -8,6 +8,10 @@ pub(crate) const MAX_GROUP_BY_FIELDS: usize = 10;

8 #[cfg(test)]

9 pub(crate) const DEFAULT_GROUP_ACCUMULATOR_MAX_GROUPS: usize = 50_000;

10 pub(crate) const FACET_VALUE_LIMIT: usize = 100;

11 +/// Hard cap on the autocomplete `term` to prevent runaway substring scans

12 +/// on large vocabularies. AS names, MAC addresses, IPv6 strings, CIDRs all

13 +/// fit comfortably under this; longer terms get rejected at deserialize.

14 +pub(crate) const MAX_AUTOCOMPLETE_TERM_LEN: usize = 256;

15 pub(crate) const FACET_CACHE_JOURNAL_WINDOW_SIZE: u64 = 8 * 1024 * 1024;

16 #[cfg(test)]

17 pub(crate) const DEFAULT_FACET_ACCUMULATOR_MAX_VALUES_PER_FIELD: usize = 5_000;

src/crates/netflow-plugin/src/query/request/model/deserialize.rs

+9 -1

@@ -1,6 +1,7 @@

1 use super::super::*;

2 use super::{FlowsRequest, RawFlowsRequest};

3 use crate::facet_catalog::facet_field_enabled;

4 +use crate::query::request::constants::MAX_AUTOCOMPLETE_TERM_LEN;

6 impl<'de> Deserialize<'de> for FlowsRequest {

7 fn deserialize<D>(deserializer: D) -> std::result::Result<Self, D::Error>

@@ -54,6 +55,7 @@ impl<'de> Deserialize<'de> for FlowsRequest {

55 (!normalized.is_empty()).then_some(normalized)

56 });

58 + let trimmed_term = raw.term.trim();

59 if matches!(mode, super::RequestMode::Autocomplete) {

60 let Some(field) = field.as_deref() else {

61 return Err(D::Error::custom(

@@ -65,7 +67,13 @@ impl<'de> Deserialize<'de> for FlowsRequest {

67 "unsupported autocomplete field `{field}`"

68 )));

69 }

70 + if trimmed_term.len() > MAX_AUTOCOMPLETE_TERM_LEN {

71 + return Err(D::Error::custom(format!(

72 + "autocomplete `term` exceeds {MAX_AUTOCOMPLETE_TERM_LEN}-byte limit"

73 + )));

74 + }

75 }

76 + let term = trimmed_term.to_string();

78 validate_selection_fields(&raw.selections).map_err(D::Error::custom)?;

@@ -81,7 +89,7 @@ impl<'de> Deserialize<'de> for FlowsRequest {

89 sort_by,

90 top_n,

91 field,

84 - term: raw.term.trim().to_string(),

92 + term,

93 })

94 }

95 }

src/crates/netflow-plugin/src/query/tests.rs

+28

@@ -540,6 +540,34 @@ fn request_deserialization_supports_autocomplete_mode() {

540 assert_eq!(request.normalized_autocomplete_term(), "10.0.");

541 }

542

543 +#[test]

544 +fn request_deserialization_rejects_oversized_autocomplete_term() {

545 + let term = "x".repeat(257);

546 + let payload = format!(

547 + r#"{{"mode":"autocomplete","field":"src_as_name","term":"{term}"}}"#

548 + );

549 + let error = serde_json::from_str::<FlowsRequest>(&payload)

550 + .expect_err("oversized autocomplete term should be rejected");

551 + assert!(

552 + error.to_string().contains("autocomplete `term` exceeds"),

553 + "unexpected error: {error}"

554 + );

555 +}

556 +

557 +#[test]

558 +fn request_deserialization_accepts_long_term_for_non_autocomplete_mode() {

559 + // The 256-byte cap is autocomplete-only. Regular flows/timeseries requests

560 + // may carry an ignored `term` of any length and must not be rejected.

561 + let term = "x".repeat(1024);

562 + let payload = format!(

563 + r#"{{"mode":"flows","view":"table-sankey","term":"{term}"}}"#

564 + );

565 + let request = serde_json::from_str::<FlowsRequest>(&payload)

566 + .expect("non-autocomplete request with long term must deserialize");

567 + assert!(!request.is_autocomplete_mode());

568 + assert_eq!(request.normalized_autocomplete_term().len(), 1024);

569 +}

570 +

571 #[test]

572 fn request_deserialization_rejects_removed_internal_timestamp_autocomplete_field() {

573 let error = serde_json::from_str::<FlowsRequest>(