| 1 | <!-- markdownlint-disable-file MD013 MD043 --> |
| 2 | |
| 3 | # topology-ip-intel-downloader |
| 4 | |
| 5 | `topology-ip-intel-downloader` builds the Netdata IP intelligence databases used by |
| 6 | topology and netflow enrichment. |
| 7 | |
| 8 | It always writes a fixed Netdata output set: |
| 9 | |
| 10 | - ASN MMDB: `topology-ip-asn.mmdb` |
| 11 | - GEO MMDB: `topology-ip-geo.mmdb` |
| 12 | - metadata JSON: `topology-ip-intel.json` |
| 13 | |
| 14 | The GEO MMDB is not country-only. It can contain: |
| 15 | |
| 16 | - `country.iso_code` |
| 17 | - `city.names.en` |
| 18 | - `region` |
| 19 | - `subdivisions` |
| 20 | - `location.latitude` |
| 21 | - `location.longitude` |
| 22 | |
| 23 | Both MMDB outputs also include Netdata classification metadata under `netdata.*` |
| 24 | for CIDRs that must be tracked individually. |
| 25 | |
| 26 | ## Stock databases and refreshed databases |
| 27 | |
| 28 | Netdata packages ship a stock payload under: |
| 29 | |
| 30 | - `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-asn.mmdb` |
| 31 | - `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-geo.mmdb` |
| 32 | - `${NETDATA_STOCK_DATA_DIR}/topology-ip-intel/topology-ip-intel.json` |
| 33 | |
| 34 | Refreshed databases generated by this downloader are written to: |
| 35 | |
| 36 | - `${NETDATA_CACHE_DIR}/topology-ip-intel` |
| 37 | |
| 38 | The netflow plugin auto-detects the cache copy first and then falls back to the |
| 39 | stock copy. This means: |
| 40 | |
| 41 | - packaged stock files provide out-of-box enrichment |
| 42 | - downloader runs update the cache copy |
| 43 | - the refreshed cache copy overrides the packaged stock copy automatically |
| 44 | |
| 45 | Important: |
| 46 | |
| 47 | - source installs from a Git checkout do not carry the packaged stock MMDBs |
| 48 | - packaged builds stage the stock payload from CI/release tooling |
| 49 | - local/source installs should run the downloader if they want a local cache copy |
| 50 | - 32-bit packaged builds still ship the staged stock payload, but do not build or install the downloader binary |
| 51 | |
| 52 | ## Config lookup |
| 53 | |
| 54 | If `--config` is not provided, the downloader loads the first existing file from: |
| 55 | |
| 56 | 1. `/etc/netdata/topology-ip-intel.yaml` |
| 57 | 2. `/usr/lib/netdata/conf.d/topology-ip-intel.yaml` |
| 58 | |
| 59 | These paths are resolved from the compiled Netdata install layout, so prefixed |
| 60 | installs use their own prefix automatically. |
| 61 | |
| 62 | ## Common CLI usage |
| 63 | |
| 64 | The CLI is optimized for the common case: |
| 65 | |
| 66 | - `--asn provider:artifact[@format]` |
| 67 | - `--geo provider:artifact[@format]` |
| 68 | - `--no-asn` |
| 69 | - `--no-geo` |
| 70 | |
| 71 | Rules: |
| 72 | |
| 73 | - `--asn` affects only ASN sources |
| 74 | - `--geo` affects only GEO sources |
| 75 | - repeated flags are ordered by precedence |
| 76 | - first source wins when address ranges overlap |
| 77 | - `--no-asn` or `--no-geo` removes that output file if it exists |
| 78 | |
| 79 | Examples: |
| 80 | |
| 81 | Use the built-in defaults: |
| 82 | |
| 83 | ```bash |
| 84 | topology-ip-intel-downloader |
| 85 | ``` |
| 86 | |
| 87 | Refresh both families explicitly from DB-IP: |
| 88 | |
| 89 | ```bash |
| 90 | topology-ip-intel-downloader \ |
| 91 | --asn dbip:asn-lite \ |
| 92 | --geo dbip:city-lite |
| 93 | ``` |
| 94 | |
| 95 | Use public ASN and country providers: |
| 96 | |
| 97 | ```bash |
| 98 | topology-ip-intel-downloader \ |
| 99 | --asn iptoasn:combined \ |
| 100 | --geo ip2location:country-lite |
| 101 | ``` |
| 102 | |
| 103 | Use MaxMind GeoLite2 sources when `MAXMIND_LICENSE_KEY` is available in the |
| 104 | environment: |
| 105 | |
| 106 | ```bash |
| 107 | MAXMIND_LICENSE_KEY="..." topology-ip-intel-downloader \ |
| 108 | --asn maxmind:geolite2-asn \ |
| 109 | --geo maxmind:geolite2-country |
| 110 | ``` |
| 111 | |
| 112 | Use CAIDA RouteViews prefix2as for ASN-only attribution: |
| 113 | |
| 114 | ```bash |
| 115 | topology-ip-intel-downloader \ |
| 116 | --asn caida:prefix2as \ |
| 117 | --geo dbip:country-lite |
| 118 | ``` |
| 119 | |
| 120 | Prefer a custom GEO source, then fall back to DB-IP: |
| 121 | |
| 122 | ```bash |
| 123 | topology-ip-intel-downloader \ |
| 124 | --geo iptoasn:combined \ |
| 125 | --geo dbip:city-lite |
| 126 | ``` |
| 127 | |
| 128 | Disable GEO output entirely for one run: |
| 129 | |
| 130 | ```bash |
| 131 | topology-ip-intel-downloader --no-geo |
| 132 | ``` |
| 133 | |
| 134 | The downloader prints the effective execution plan before downloading, including: |
| 135 | |
| 136 | - effective ASN source order |
| 137 | - effective GEO source order |
| 138 | - which output files will be written |
| 139 | - which output files will be removed |
| 140 | |
| 141 | ## Advanced config |
| 142 | |
| 143 | The config file uses ordered `sources[]` entries. Each entry is explicit about: |
| 144 | |
| 145 | - `family`: `asn` or `geo` |
| 146 | - `provider` |
| 147 | - `artifact` |
| 148 | - `format` |
| 149 | - optional `url` or `path` |
| 150 | |
| 151 | Built-in DB-IP sources can omit `url` and `path`. The downloader resolves the |
| 152 | current monthly download URL from the official DB-IP landing page. |
| 153 | |
| 154 | Built-in MaxMind sources require `MAXMIND_LICENSE_KEY` in the environment. |
| 155 | The generated metadata redacts URL query strings so license keys are not written |
| 156 | to `topology-ip-intel.json`. |
| 157 | |
| 158 | User-provided databases can point directly to: |
| 159 | |
| 160 | - a local file with `path` |
| 161 | - a direct URL with `url` |
| 162 | |
| 163 | Example: |
| 164 | |
| 165 | ```yaml |
| 166 | sources: |
| 167 | - name: custom-asn |
| 168 | family: asn |
| 169 | provider: dbip |
| 170 | artifact: asn-lite |
| 171 | path: /srv/ip-intel/custom-asn.mmdb |
| 172 | |
| 173 | - name: fallback-asn |
| 174 | family: asn |
| 175 | provider: dbip |
| 176 | artifact: asn-lite |
| 177 | format: mmdb |
| 178 | |
| 179 | - name: custom-geo |
| 180 | family: geo |
| 181 | provider: dbip |
| 182 | artifact: city-lite |
| 183 | url: https://example.internal/geo.mmdb.gz |
| 184 | format: mmdb |
| 185 | |
| 186 | output: |
| 187 | directory: /var/cache/netdata/topology-ip-intel |
| 188 | asn_file: topology-ip-asn.mmdb |
| 189 | geo_file: topology-ip-geo.mmdb |
| 190 | metadata_file: topology-ip-intel.json |
| 191 | ``` |
| 192 | |
| 193 | Important: |
| 194 | |
| 195 | - source order is per family |
| 196 | - earlier entries win on overlap |
| 197 | - removing every source of a family disables that family |
| 198 | - when a family is disabled, its stale output file is deleted on the next run |
| 199 | |
| 200 | ## Supported built-in sources |
| 201 | |
| 202 | ASN sources: |
| 203 | |
| 204 | - `dbip:asn-lite` (`mmdb`, `csv`) |
| 205 | - `iptoasn:combined` (`tsv`) |
| 206 | - `caida:prefix2as` (`tsv`) |
| 207 | - `maxmind:geolite2-asn` (`mmdb`, requires `MAXMIND_LICENSE_KEY`) |
| 208 | |
| 209 | GEO sources: |
| 210 | |
| 211 | - `dbip:country-lite` (`mmdb`, `csv`) |
| 212 | - `dbip:city-lite` (`mmdb`, `csv`) |
| 213 | - `iptoasn:combined` (`tsv`, country only) |
| 214 | - `maxmind:geolite2-country` (`csv`, requires `MAXMIND_LICENSE_KEY`, country only) |
| 215 | - `ip2location:country-lite` (`csv`, country only) |
| 216 | - `ipdeny:country-zones` (`cidr`, country only) |
| 217 | - `ipip:country` (`txt`, country only) |
| 218 | |
| 219 | Supported formats: |
| 220 | |
| 221 | - `mmdb` |
| 222 | - `csv` |
| 223 | - `tsv` for `iptoasn:combined` and `caida:prefix2as` |
| 224 | - `cidr` for `ipdeny:country-zones` |
| 225 | - `txt` for `ipip:country` |
| 226 | |
| 227 | The downloader accepts direct file/URL config for advanced cases, but the CLI |
| 228 | source tokens are intentionally focused on built-in source families. |
| 229 | |
| 230 | ## Output behavior |
| 231 | |
| 232 | Every successful run stages all files privately and then publishes each final |
| 233 | output atomically with `rename()`. |
| 234 | |
| 235 | This gives two guarantees: |
| 236 | |
| 237 | - readers never see a half-written MMDB file |
| 238 | - the final visible output set matches the currently enabled families |
| 239 | |
| 240 | If a family is disabled: |
| 241 | |
| 242 | - its final MMDB file is removed |
| 243 | - metadata is rewritten to describe the new output set |
| 244 | |
| 245 | ## Preparing a local stock payload staging directory |
| 246 | |
| 247 | From the repository root: |
| 248 | |
| 249 | ```bash |
| 250 | ./src/go/tools/topology-ip-intel-downloader/refresh-stock.sh |
| 251 | ``` |
| 252 | |
| 253 | This generates a local staging directory at: |
| 254 | |
| 255 | - `./artifacts/topology-ip-intel-stock/README.md` |
| 256 | - `./artifacts/topology-ip-intel-stock/topology-ip-asn.mmdb` |
| 257 | - `./artifacts/topology-ip-intel-stock/topology-ip-geo.mmdb` |
| 258 | - `./artifacts/topology-ip-intel-stock/topology-ip-intel.json` |
| 259 | |
| 260 | This is useful for: |
| 261 | |
| 262 | - checking the current packaged stock payload locally |
| 263 | - preparing a staged payload for package/release work |
| 264 | - verifying DB-IP downloads without storing generated binaries in Git |