@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 233 lines 8.2 KB
Rendered Raw
1 <!--startmeta
2 custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/custom_mmdb_database.md"
3 meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"
4 sidebar_label: "Custom MMDB Database"
5 learn_status: "Published"
6 learn_rel_path: "Network Flows/Enrichment Methods"
7 keywords: ['mmdb', 'custom database', 'bring your own', 'mmdbwriter', 'internal asn', 'internal ipam', 'ip intelligence']
8 message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"
9 endmeta-->
10
11 <!-- markdownlint-disable-file -->
12
13 # Custom MMDB Database
14
15
16 <img src="https://netdata.cloud/img/network-wired.svg" width="150"/>
17
18
19 Plugin: netflow-plugin
20 Module: custom-mmdb
21
22 <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
23
24 ## Overview
25
26 Enrich network flows with ASN, geographic, or internal network context from
27 operator-built MMDB databases. Use Custom MMDB Database when you want the netflow
28 plugin to read a MaxMind-format binary database (`.mmdb`) that your
29 own tooling produces -- typical use cases are joining the upstream DB-IP /
30 GeoLite2 data with internal AS labels, or producing a CIDR-tagged custom file
31 directly from an internal IPAM. The plugin only requires the file to conform to
32 the [MaxMind DB binary format](https://maxmind.github.io/MaxMind-DB/); whatever
33 fields it exposes are picked up by the same decoder path used for DB-IP and
34 MaxMind.
35
36 The decoder reads `autonomous_system_number`, `autonomous_system_organization`,
37 and (optionally) a string `asn` field from the ASN database, and `country.iso_code`,
38 `subdivisions[].iso_code`, `city.names.en`, `location.latitude`,
39 `location.longitude`, plus the Netdata-specific `netdata.ip_class` flag from the
40 geo database. Any other vendor-specific keys are ignored.
41
42 Note that `*_GEO_CITY`, `*_GEO_LATITUDE`, and `*_GEO_LONGITUDE` are written to
43 the raw journal tier only -- the rollup tiers drop these high-cardinality fields.
44 State and country survive into all four tiers.
45
46 For the full IP-intelligence concept (composition rule, ASN provider chain,
47 auto-detect path order, dual-stack handling, hot reload semantics), see
48 [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment).
49
50
51 You build (or download) the MMDB file with your own tooling, place it on the
52 agent host, and point `enrichment.geoip.asn_database` and / or
53 `enrichment.geoip.geo_database` at it in `netflow.yaml`. The plugin reloads on
54 file change automatically (mechanism documented on the IP Intelligence page).
55
56
57 This integration is only supported on the following platforms:
58
59 - Linux
60
61 This integration runs as a single instance per Netdata Agent.
62
63
64 ### Default Behavior
65
66 #### Auto-Detection
67
68 Not auto-detected. You must configure paths explicitly.
69
70 #### Limits
71
72 Coverage, schema quality, and freshness are entirely controlled by the custom MMDB build you provide. Invalid or missing files fail startup unless marked optional.
73
74 #### Performance Impact
75
76 Lookups are local MMDB reads with no per-flow network call. Memory use depends on the size and number of custom databases you configure.
77
78 ## Setup
79
80
81 ### Prerequisites
82
83 #### Build or obtain a standards-compliant MMDB file
84
85 The file must conform to the
86 [MaxMind DB binary format spec](https://maxmind.github.io/MaxMind-DB/) --
87 a binary search tree over IP prefixes plus a data section of
88 type-length-value records. Validate with `mmdblookup` from the
89 `libmaxminddb-tools` package before deploying:
90
91 ```bash
92 mmdblookup --file your-custom.mmdb --ip 8.8.8.8
93 ```
94
95 Common ways to produce a file:
96
97 - **Go**: [`github.com/maxmind/mmdbwriter`](https://github.com/maxmind/mmdbwriter)
98 -- MaxMind's official Go writer. Lets you start from an upstream MMDB
99 (DB-IP, GeoLite2) and overlay extra records, then write a new file.
100 Pairs well with the read-side library
101 [`github.com/oschwald/maxminddb-golang`](https://github.com/oschwald/maxminddb-golang).
102 - **Perl**: [`MaxMind::DB::Writer`](https://metacpan.org/pod/MaxMind::DB::Writer)
103 -- the original reference writer.
104 - **Python**: community ports such as `mmdb_writer` (PyPI) wrap the same
105 format.
106
107 Whatever fields you encode are read by the plugin only if they match the
108 standard schema names listed in the metrics description above.
109
110
111
112 ### Configuration
113
114 #### Options
115
116 Point `enrichment.geoip.asn_database` and / or `enrichment.geoip.geo_database`
117 at your custom MMDB file paths. Set `optional: true` while you iterate so a
118 missing or malformed file does not block plugin startup.
119
120
121 <details open><summary>Config options</summary>
122
123
124
125 | Option | Description | Default | Required |
126 |:-----|:------------|:--------|:---------:|
127 | enrichment.geoip.asn_database | List of MMDB paths providing AS data. Multiple files compose -- per field, the last database returning a non-empty value wins. | [] | no |
128 | enrichment.geoip.geo_database | List of MMDB paths providing geographic data. Same composition rule as the ASN list. | [] | no |
129 | enrichment.geoip.optional | When true, missing or unreadable files at startup are tolerated (the resolver starts with no databases). Recommended while testing a custom file. | false | no |
130
131
132 </details>
133
134
135
136 #### via File
137
138 The configuration file name for this integration is `netflow.yaml`.
139
140
141 You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the
142 Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).
143
144 ```bash
145 cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
146 sudo ./edit-config netflow.yaml
147 ```
148
149 ##### Examples
150
151 ###### Internal MMDB built with mmdbwriter
152
153 A custom build that combines public BGP data with internal CIDR labels, written atomically into the plugin's lookup directory by the operator's own pipeline.
154
155 ```yaml
156 enrichment:
157 geoip:
158 asn_database:
159 - /etc/netdata/internal-asn.mmdb
160 geo_database:
161 - /etc/netdata/internal-geo.mmdb
162 optional: false
163
164 ```
165 ###### Overlay a custom ASN file on top of the bundled DB-IP geo
166
167 Keep DB-IP for geographic data and use a custom MMDB only for AS labels (your IPAM-derived prefixes win because they appear last in the list -- the resolver merges per field, last non-empty wins).
168
169 <details open><summary>Config</summary>
170
171 ```yaml
172 enrichment:
173 geoip:
174 asn_database:
175 - /var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb
176 - /etc/netdata/internal-asn.mmdb
177 geo_database:
178 - /var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb
179 optional: true
180
181 ```
182 </details>
183
184 ###### Iterating on a new custom MMDB
185
186 While you tune the build pipeline, mark the database optional so an absent or malformed file becomes a startup warning instead of a fatal error.
187
188 <details open><summary>Config</summary>
189
190 ```yaml
191 enrichment:
192 geoip:
193 asn_database:
194 - /etc/netdata/wip-asn.mmdb
195 geo_database: []
196 optional: true
197
198 ```
199 </details>
200
201
202
203 ### Lookups silently return empty
204
205 The custom file's schema is non-standard (for example, the ASN is encoded
206 as a string instead of `autonomous_system_number: uint32`, or the country
207 is at a non-standard path). The plugin only reads the standard MaxMind
208 field names listed in the data-collection description. Validate with
209 `mmdblookup --file your-custom.mmdb --ip <known-public-ip>` and confirm the
210 standard fields are present at the expected paths.
211
212
213 ### Plugin fails to start with optional=false
214
215 File missing, unreadable, or not a valid MMDB at the configured path.
216 Check permissions (the netdata user must be able to read the file) and
217 re-validate with `mmdblookup`. Set `optional: true` while iterating.
218
219
220 ### Custom file refresh did not take effect
221
222 The plugin reloads custom databases the same way it reloads stock ones
223 (signature-driven, in place). If the file was rewritten in place the
224 plugin will pick it up; if it was edited byte-by-byte (rare with MMDB
225 builders) verify size or mtime actually changed. The reload mechanism
226 and its 30-second cadence are documented in
227 [IP Intelligence](https://learn.netdata.cloud/docs/network-flows/enrichment).
228 Operational practice: have your build pipeline write the new MMDB to a
229 temp file in the same directory and `rename(2)` it over the live path so
230 the swap is atomic.
231
232
233