| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/intel-downloader.md" |
| 3 | sidebar_label: "Enrichment Intel Downloader" |
| 4 | learn_status: "Published" |
| 5 | learn_rel_path: "Network Flows" |
| 6 | keywords: ['ip intelligence', 'mmdb', 'downloader', 'db-ip', 'iptoasn', 'topology-ip-intel-downloader', 'enrichment', 'refresh'] |
| 7 | endmeta--> |
| 8 | |
| 9 | <!-- markdownlint-disable-file --> |
| 10 | |
| 11 | # Enrichment Intel Downloader |
| 12 | |
| 13 | `topology-ip-intel-downloader` is a small Netdata-supplied tool that keeps the IP intelligence MMDB databases used by the netflow plugin (and the topology subsystem) up to date. It fetches the upstream payloads, normalises them into a fixed Netdata MMDB layout, applies CIDR classification policy, and atomically replaces the files on disk. The netflow plugin's resolver picks up the new files within 30 seconds — no plugin restart required. |
| 14 | |
| 15 | Packaged 32-bit installs ship the stock MMDB payload but do not include the downloader binary. Source builds from a Git checkout also do not include the generated stock MMDB payload by default. |
| 16 | |
| 17 | The downloader is a separate executable so you can run it on whatever schedule fits your environment without coupling it to the agent's lifecycle. |
| 18 | |
| 19 | ## What it does |
| 20 | |
| 21 | - Fetches the configured ASN and Geo source files over HTTPS, with gzip / zip transparently decoded. |
| 22 | - Parses the upstream format (MMDB or TSV/CSV), keeping the first source per family that covers a given range — first-source-wins on overlap. |
| 23 | - Re-emits the data as two Netdata-format MMDB files plus a metadata JSON manifest. |
| 24 | - Stamps Netdata classification metadata (`netdata.ip_class`, `netdata.track_individual`) over `localhost_cidrs`, `private_cidrs`, and any operator-defined `interesting_cidrs` so the plugin can identify private/loopback/operator-flagged ranges via a normal MMDB lookup. |
| 25 | - Publishes each output atomically via stage-then-`rename(2)` — the resolver never sees a torn file. |
| 26 | |
| 27 | The output is always the same fixed file set, regardless of which providers fed the run: |
| 28 | |
| 29 | ``` |
| 30 | /var/cache/netdata/topology-ip-intel/ |
| 31 | ├── topology-ip-asn.mmdb # ASN database |
| 32 | ├── topology-ip-geo.mmdb # Geographic database |
| 33 | └── topology-ip-intel.json # Manifest: when, from where, how many ranges |
| 34 | ``` |
| 35 | |
| 36 | The directory and filenames match the shipped defaults. |
| 37 | |
| 38 | ## Supported sources |
| 39 | |
| 40 | The tool only knows how to talk to a fixed set of providers — anything else is rejected at validation: |
| 41 | |
| 42 | | Provider:Artifact | Family | Format | Origin | |
| 43 | |---|---|---|---| |
| 44 | | `dbip:asn-lite` | ASN | `mmdb` (default) or `csv` | DB-IP free monthly download page | |
| 45 | | `dbip:country-lite` | Geo | `mmdb` (default) or `csv` | DB-IP free monthly download page | |
| 46 | | `dbip:city-lite` | Geo | `mmdb` (default) or `csv` | DB-IP free monthly download page | |
| 47 | | `iptoasn:combined` | ASN or Geo | `tsv` | `https://iptoasn.com/data/ip2asn-combined.tsv.gz` (direct URL) | |
| 48 | | `caida:prefix2as` | ASN | `tsv` | CAIDA RouteViews prefix-to-AS creation log | |
| 49 | | `maxmind:geolite2-asn` | ASN | `mmdb` | MaxMind authenticated GeoLite2 download | |
| 50 | | `maxmind:geolite2-country` | Geo | `csv` | MaxMind authenticated GeoLite2 Country CSV ZIP download | |
| 51 | | `ip2location:country-lite` | Geo | `csv` | IP2Location Lite country CSV ZIP download | |
| 52 | | `ipdeny:country-zones` | Geo | `cidr` | IPDeny country zone archive | |
| 53 | | `ipip:country` | Geo | `txt` | IPIP country text ZIP download | |
| 54 | |
| 55 | DB-IP artifacts are resolved from the current monthly URL on the DB-IP landing page (`https://db-ip.com/db/download/<artifact>`). The downloaded URL uses the DB-IP free database pattern `https://download.db-ip.com/free/dbip-<artifact>-YYYY-MM.<ext>.gz`. |
| 56 | |
| 57 | The IPtoASN TSV feed is converted into the same Netdata MMDB layout as the DB-IP feeds, so consumers don't care which source produced the file. |
| 58 | |
| 59 | CAIDA prefix2as is ASN-only and has no AS organization names. The downloader resolves the latest `.pfx2as.gz` entry from CAIDA's creation log before fetching it. |
| 60 | |
| 61 | MaxMind built-in sources require `MAXMIND_LICENSE_KEY` in the downloader environment. `maxmind:geolite2-asn@mmdb` downloads the official GeoLite2 ASN tarball and extracts the MMDB member. `maxmind:geolite2-country@csv` downloads the official GeoLite2 Country CSV **ZIP bundle** and needs the locations file plus the IPv4/IPv6 block CSVs inside that bundle; `csv` here does not mean a single raw CSV file. |
| 62 | |
| 63 | IP2Location `country-lite@csv` is also the provider's official CSV ZIP bundle. IPDeny `country-zones@cidr` is the `all-zones.tar.gz` archive, and IPIP `country@txt` is the country text ZIP. |
| 64 | |
| 65 | You can still pull *any* MMDB build (including a custom one) into the resolver by configuring `enrichment.geoip.asn_database` / `geo_database` directly — the downloader is one of several producers; the plugin doesn't care who wrote the MMDB. See the [Custom MMDB Database](/src/crates/netflow-plugin/integrations/custom_mmdb_database.md) card. If you prefer MaxMind's own updater, run [`geoipupdate`](/src/crates/netflow-plugin/integrations/maxmind_geoip_-_geolite2.md) and point `enrichment.geoip.asn_database` / `enrichment.geoip.geo_database` at the MMDB files it produces. |
| 66 | |
| 67 | ## Configuration file |
| 68 | |
| 69 | The downloader reads YAML config from the first existing file in this order: |
| 70 | |
| 71 | 1. `/etc/netdata/topology-ip-intel.yaml` (operator overrides) |
| 72 | 2. `/usr/lib/netdata/conf.d/topology-ip-intel.yaml` (stock, shipped by the package) |
| 73 | |
| 74 | If neither exists, the built-in defaults are used. Pass `--config /path/to/file.yaml` to force a specific path. |
| 75 | |
| 76 | The shipped stock file is: |
| 77 | |
| 78 | ```yaml |
| 79 | sources: |
| 80 | - name: dbip-asn |
| 81 | family: asn |
| 82 | provider: dbip |
| 83 | artifact: asn-lite |
| 84 | format: mmdb |
| 85 | |
| 86 | - name: dbip-geo |
| 87 | family: geo |
| 88 | provider: dbip |
| 89 | artifact: city-lite |
| 90 | format: mmdb |
| 91 | |
| 92 | output: |
| 93 | directory: /var/cache/netdata/topology-ip-intel |
| 94 | asn_file: topology-ip-asn.mmdb |
| 95 | geo_file: topology-ip-geo.mmdb |
| 96 | metadata_file: topology-ip-intel.json |
| 97 | |
| 98 | policy: |
| 99 | localhost_cidrs: |
| 100 | - 127.0.0.0/8 |
| 101 | - ::1/128 |
| 102 | private_cidrs: |
| 103 | - 10.0.0.0/8 |
| 104 | - 172.16.0.0/12 |
| 105 | - 192.168.0.0/16 |
| 106 | - 100.64.0.0/10 |
| 107 | - fc00::/7 |
| 108 | - fe80::/10 |
| 109 | interesting_cidrs: [] |
| 110 | |
| 111 | http: |
| 112 | timeout: 2m |
| 113 | user_agent: netdata-topology-ip-intel-downloader/1.0 |
| 114 | ``` |
| 115 | |
| 116 | | Key | Notes | |
| 117 | |---|---| |
| 118 | | `sources[]` | Ordered list per family. Each entry needs `family` (`asn` or `geo`), `provider`, `artifact`. `format` is inferred from the provider/artifact when omitted. Optional `url` overrides the built-in URL; optional `path` reads from a local file instead. Earlier entries win on overlap. | |
| 119 | | `output.directory` | Where the MMDB and metadata files land. Must match what the netflow plugin reads (see below). | |
| 120 | | `output.asn_file` / `output.geo_file` / `output.metadata_file` | File names only — paths are rejected by validation. | |
| 121 | | `policy.localhost_cidrs` / `private_cidrs` | Stamped into both MMDBs as `netdata.ip_class = "localhost"` / `"private"`. | |
| 122 | | `policy.interesting_cidrs` | Operator-defined public ranges to track individually. Stamped as `netdata.ip_class = "interesting"`. | |
| 123 | | `http.timeout` | Per-request timeout. Default `2m`. | |
| 124 | | `http.user_agent` | Sent to upstream providers. Default `netdata-topology-ip-intel-downloader/1.0`. | |
| 125 | |
| 126 | CLI flags can override the config without editing the file: |
| 127 | |
| 128 | | Flag | Purpose | |
| 129 | |---|---| |
| 130 | | `--config PATH` | Force a specific YAML config path. | |
| 131 | | `--output-dir DIR` | Override `output.directory`. | |
| 132 | | `--asn provider:artifact[@format]` | Replace the ASN source list. Repeatable; first wins. | |
| 133 | | `--geo provider:artifact[@format]` | Replace the Geo source list. Repeatable; first wins. | |
| 134 | | `--no-asn` | Disable ASN output and delete any stale `topology-ip-asn.mmdb`. | |
| 135 | | `--no-geo` | Disable Geo output and delete any stale `topology-ip-geo.mmdb`. | |
| 136 | |
| 137 | ## Scheduled execution |
| 138 | |
| 139 | **Netdata does not ship a systemd timer or cron entry for the downloader.** This is intentional — the appropriate refresh cadence depends on the provider's update cadence, your bandwidth, and your change-control policy, and a packaged timer would force one choice on every install. |
| 140 | |
| 141 | Set up your own. A simple systemd timer is the recommended pattern: |
| 142 | |
| 143 | ```ini |
| 144 | # /etc/systemd/system/netdata-topology-ip-intel.service |
| 145 | [Unit] |
| 146 | Description=Refresh Netdata IP intelligence databases |
| 147 | |
| 148 | [Service] |
| 149 | Type=oneshot |
| 150 | ExecStart=/usr/sbin/topology-ip-intel-downloader |
| 151 | User=netdata |
| 152 | Group=netdata |
| 153 | ``` |
| 154 | |
| 155 | ```ini |
| 156 | # /etc/systemd/system/netdata-topology-ip-intel.timer |
| 157 | [Unit] |
| 158 | Description=Weekly refresh of Netdata IP intelligence databases |
| 159 | |
| 160 | [Timer] |
| 161 | OnCalendar=weekly |
| 162 | RandomizedDelaySec=1h |
| 163 | Persistent=true |
| 164 | |
| 165 | [Install] |
| 166 | WantedBy=timers.target |
| 167 | ``` |
| 168 | |
| 169 | ```bash |
| 170 | sudo systemctl daemon-reload |
| 171 | sudo systemctl enable --now netdata-topology-ip-intel.timer |
| 172 | ``` |
| 173 | |
| 174 | Refresh cadence depends on the sources you enable. DB-IP refreshes its free Lite databases monthly; weekly is a safe over-poll that picks up every release within a few days while staying polite to the upstream. IPtoASN refreshes hourly, but downstream consumers rarely need that resolution — daily is plenty if you switch to it. CAIDA prefix2as, MaxMind, IP2Location, IPDeny, and IPIP have their own publication schedules and terms; choose a timer cadence that is polite to the upstream and fast enough for your environment. |
| 175 | |
| 176 | Run the packaged binary as the `netdata` user (or root) so it can write to `/var/cache/netdata/topology-ip-intel/`. |
| 177 | |
| 178 | ## Manual invocation |
| 179 | |
| 180 | Trigger an out-of-schedule refresh: |
| 181 | |
| 182 | ```bash |
| 183 | sudo systemctl start netdata-topology-ip-intel.service # if you set up the unit above |
| 184 | ``` |
| 185 | |
| 186 | Or invoke the binary directly — it loads the same config, prints the execution plan, and writes to the same destination: |
| 187 | |
| 188 | ```bash |
| 189 | sudo -u netdata /usr/sbin/topology-ip-intel-downloader |
| 190 | ``` |
| 191 | |
| 192 | A successful run finishes in well under a minute on a typical link and prints something like: |
| 193 | |
| 194 | ``` |
| 195 | effective source plan: |
| 196 | ASN sources (first wins): |
| 197 | - 1. dbip:asn-lite@mmdb |
| 198 | GEO sources (first wins): |
| 199 | - 1. dbip:city-lite@mmdb |
| 200 | output actions: |
| 201 | - write topology-ip-asn.mmdb |
| 202 | - write topology-ip-geo.mmdb |
| 203 | - write topology-ip-intel.json |
| 204 | updated IP intelligence databases using config /usr/lib/netdata/conf.d/topology-ip-intel.yaml |
| 205 | asn_mmdb=/var/cache/netdata/topology-ip-intel/topology-ip-asn.mmdb |
| 206 | geo_mmdb=/var/cache/netdata/topology-ip-intel/topology-ip-geo.mmdb |
| 207 | metadata=/var/cache/netdata/topology-ip-intel/topology-ip-intel.json |
| 208 | asn_ranges=1234567 geo_ranges=8901234 |
| 209 | ``` |
| 210 | |
| 211 | The plan is printed *before* any download, so you can verify the effective source list without committing to a fetch. |
| 212 | |
| 213 | ## Output and atomic replacement |
| 214 | |
| 215 | Atomic publication is the contract this tool provides to the netflow plugin's resolver: |
| 216 | |
| 217 | 1. A staging directory is created inside `output.directory` (`.tmp-topology-ip-intel-stage-*`) and removed on exit. |
| 218 | 2. The MMDB writer streams into a per-file temp inside that staging directory. |
| 219 | 3. Each finished MMDB is fsync-closed, chmodded `0644`, and renamed into its final name. |
| 220 | 4. The metadata JSON is renamed last, so a partially-completed run never updates the manifest. |
| 221 | |
| 222 | Because `rename(2)` is atomic on the same filesystem, a reader that opens the file at any moment sees either the old complete file or the new complete file — never a half-written one. The netflow plugin's resolver re-stats and re-opens the MMDBs every 30 seconds, so a fresh download is live within at most 30 seconds of completion. No plugin restart, no agent restart. |
| 223 | |
| 224 | ## Failure modes |
| 225 | |
| 226 | The tool exits non-zero with a diagnostic on `stderr` for any of these cases: |
| 227 | |
| 228 | | Failure | Behaviour | |
| 229 | |---|---| |
| 230 | | Config syntax / validation error | Exits before any network activity. Existing MMDBs are untouched. | |
| 231 | | Upstream unreachable / non-200 status | The run aborts before any output is staged. Existing MMDBs are untouched. | |
| 232 | | Decompression / parse error | Same as above — abort before publishing. | |
| 233 | | Disk full / rename failure during publish | The staging directory is cleaned up; the previously-published file remains in place. | |
| 234 | |
| 235 | Net result: **a failed run keeps the previously good databases**. The plugin keeps serving stale-but-correct enrichment until the next successful run replaces them. There is no built-in retry — schedule the timer often enough that a single missed run isn't critical. |
| 236 | |
| 237 | If you run the downloader from a systemd timer, the failure is visible via `systemctl status netdata-topology-ip-intel.service` and `journalctl -u netdata-topology-ip-intel`. There is no log file written by the tool itself; it only writes to stdout/stderr. |
| 238 | |
| 239 | ## Integration with the netflow plugin's auto-detect |
| 240 | |
| 241 | The netflow plugin auto-discovers MMDB files at startup when neither `enrichment.geoip.asn_database` nor `enrichment.geoip.geo_database` is set. The lookup order is: |
| 242 | |
| 243 | 1. `<cache_dir>/topology-ip-intel/topology-ip-asn.mmdb` and `topology-ip-geo.mmdb` — the directory the downloader writes to. |
| 244 | 2. `<stock_data_dir>/topology-ip-intel/...` — the package-shipped stock payload (typically `/usr/share/netdata/topology-ip-intel/`), used as fallback when no fresh copy exists yet. |
| 245 | |
| 246 | `<cache_dir>` defaults to `/var/cache/netdata`; `<stock_data_dir>` defaults to `/usr/share/netdata`. The downloader's default output directory matches the cache path the plugin checks first, so a fresh run automatically supersedes the stock payload. |
| 247 | |
| 248 | When the plugin auto-detects MMDBs this way it forces `optional: true` on the geoip stanza — a missing or transiently-unreadable file does not crash the plugin. If you instead set `asn_database` / `geo_database` explicitly in `netflow.yaml`, you control the `optional` flag yourself; see [Configuration](/docs/network-flows/configuration.md#enrichment). |
| 249 | |
| 250 | ## What's next |
| 251 | |
| 252 | - Per-provider details (refresh cadence, license, schema, attribution requirements): |
| 253 | - [DB-IP IP Intelligence](/src/crates/netflow-plugin/integrations/db-ip_ip_intelligence.md) — the default the downloader fetches. |
| 254 | - [IPtoASN](/src/crates/netflow-plugin/integrations/iptoasn.md) — public-domain TSV feed; converted to MMDB by this tool. |
| 255 | - [CAIDA RouteViews Prefix-to-AS](/src/crates/netflow-plugin/integrations/caida_routeviews_prefix-to-as.md) — prefix-to-AS TSV feed; ASN-only. |
| 256 | - [MaxMind GeoIP / GeoLite2](/src/crates/netflow-plugin/integrations/maxmind_geoip_-_geolite2.md) — authenticated MaxMind downloads or MMDB files managed by `geoipupdate`. |
| 257 | - [IP2Location LITE IP-Country](/src/crates/netflow-plugin/integrations/ip2location_lite_ip-country.md) — public country-only CSV ZIP feed. |
| 258 | - [IPDeny Country Zones](/src/crates/netflow-plugin/integrations/ipdeny_country_zones.md) — country CIDR archive. |
| 259 | - [IPIP Country Database](/src/crates/netflow-plugin/integrations/ipip_country_database.md) — country text ZIP feed. |
| 260 | - [Custom MMDB Database](/src/crates/netflow-plugin/integrations/custom_mmdb_database.md) — your own MMDB build. |
| 261 | - The enrichment mechanism that consumes these files: [Enrichment](/docs/network-flows/enrichment.md) (the MMDB shared mechanism section). |
| 262 | - The plugin knobs that point at the files: [Configuration › `enrichment.geoip`](/docs/network-flows/configuration.md#enrichment). |