| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/azure_ip_ranges.md" |
| 3 | meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml" |
| 4 | sidebar_label: "Azure IP Ranges" |
| 5 | learn_status: "Published" |
| 6 | learn_rel_path: "Network Flows/Enrichment Methods" |
| 7 | keywords: ['azure', 'microsoft', 'cloud', 'ip ranges', 'service tags'] |
| 8 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE" |
| 9 | endmeta--> |
| 10 | |
| 11 | <!-- markdownlint-disable-file --> |
| 12 | |
| 13 | # Azure IP Ranges |
| 14 | |
| 15 | |
| 16 | <img src="https://netdata.cloud/img/network-wired.svg" width="150"/> |
| 17 | |
| 18 | |
| 19 | Plugin: netflow-plugin |
| 20 | Module: azure-ip-ranges |
| 21 | |
| 22 | <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> |
| 23 | |
| 24 | ## Overview |
| 25 | |
| 26 | Annotate network flows with Azure service and region labels from Azure Service |
| 27 | Tags. Microsoft publishes Azure's public IP prefixes as **Service Tags JSON |
| 28 | files**. A service tag is Microsoft's name for a group of IP prefixes belonging |
| 29 | to one Azure service (e.g., `Storage`, `Sql`, `AzureFrontDoor.Backend`); the |
| 30 | JSON enumerates every tag together with the prefixes, the platform, and -- for |
| 31 | tags that support regional scope -- the Azure region. |
| 32 | |
| 33 | Microsoft publishes four separate JSON files, one per Azure cloud: |
| 34 | |
| 35 | - **Azure Public** -- commercial Azure, the file most users want. |
| 36 | - **Azure US Government** -- separate cloud for US public-sector workloads. |
| 37 | - **Microsoft Azure operated by 21Vianet** -- the China cloud. |
| 38 | - **Azure Germany** -- legacy German cloud. |
| 39 | |
| 40 | This integration tags flow records to/from Azure-hosted services with |
| 41 | `*_NET_TENANT="azure"` plus per-region and per-service labels. Use it to |
| 42 | identify traffic to Azure SQL, Azure Storage, Azure Front Door, App Service, |
| 43 | and the rest -- without maintaining the prefix list yourself. |
| 44 | |
| 45 | For the full network-identity concept (merge order, jq output shape, TLS |
| 46 | verification, failure modes), see |
| 47 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 48 | |
| 49 | |
| 50 | Periodic HTTPS GET against the Service Tags JSON, jq transform via the |
| 51 | [jaq](https://github.com/01mf02/jaq) library, merge into the network-attributes |
| 52 | trie. Same mechanism as AWS / GCP IP Ranges -- different URL and JSON shape, |
| 53 | plus one important operational caveat documented below. |
| 54 | |
| 55 | |
| 56 | This integration is only supported on the following platforms: |
| 57 | |
| 58 | - Linux |
| 59 | |
| 60 | This integration supports multiple instances configured side-by-side. |
| 61 | |
| 62 | |
| 63 | ### Default Behavior |
| 64 | |
| 65 | #### Auto-Detection |
| 66 | |
| 67 | Disabled by default. Add an entry under enrichment.network_sources to enable. |
| 68 | |
| 69 | #### Limits |
| 70 | |
| 71 | One full Azure Service Tags document is fetched per refresh. Resource use scales with the number of Azure prefixes selected by your transform and the refresh interval. |
| 72 | |
| 73 | #### Performance Impact |
| 74 | |
| 75 | One HTTPS request per refresh interval plus a jq transform over the Azure Service Tags document. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records. |
| 76 | |
| 77 | ## Setup |
| 78 | |
| 79 | |
| 80 | ### Prerequisites |
| 81 | |
| 82 | #### Azure publishes weekly, but the URL rotates |
| 83 | |
| 84 | Microsoft updates the Service Tags JSON **weekly** (per |
| 85 | <https://learn.microsoft.com/en-us/azure/virtual-network/service-tags-overview#discover-service-tags-by-using-downloadable-json-files>: |
| 86 | "These lists are updated and published weekly"), and Microsoft asks you |
| 87 | to delay rolling out new prefixes for at least one week after publication |
| 88 | ("When new IP addresses are added to service tags, they aren't used in |
| 89 | Azure for at least one week"). |
| 90 | |
| 91 | The catch: the actual download URL is **not stable**. The download portal |
| 92 | at <https://www.microsoft.com/en-us/download/details.aspx?id=56519> serves |
| 93 | a `ServiceTags_Public_<YYYYMMDD>.json` filename whose date stamp changes |
| 94 | every week. A naive `interval`-based fetch against last week's URL will |
| 95 | start 404'ing as soon as a new file is published. |
| 96 | |
| 97 | You have three reasonable options, in increasing order of effort: |
| 98 | |
| 99 | 1. **Skip Azure IP Ranges entirely** and rely on GeoIP / ASN data |
| 100 | (Microsoft's main ASN is 8075). Coarse but zero-maintenance. |
| 101 | 2. **Use a stable redirect** if your environment provides one (some |
| 102 | enterprise mirrors do). |
| 103 | 3. **Run a small mirror script** -- a cron job that fetches the latest |
| 104 | JSON via the [Service Tag Discovery REST API](https://learn.microsoft.com/en-us/rest/api/virtualnetwork/servicetags/list) |
| 105 | (or `Get-AzNetworkServiceTag` in PowerShell) and writes the result to |
| 106 | a stable path on an internal HTTP server. Point the plugin at that |
| 107 | stable URL. |
| 108 | |
| 109 | Option 3 is what most operators end up with. The Service Tag Discovery |
| 110 | API is authoritative and authenticated (Azure subscription with read |
| 111 | role); the downloadable JSON is unauthenticated but URL-rotated. Pick |
| 112 | whichever fits your operational model. |
| 113 | |
| 114 | |
| 115 | #### Outbound HTTPS to your mirror (or to Microsoft, if you script the URL discovery) |
| 116 | |
| 117 | The plugin only needs to reach the configured `url`. Azure credentials |
| 118 | are needed only on the side that resolves "what's this week's URL?" -- |
| 119 | not by the plugin itself. |
| 120 | |
| 121 | |
| 122 | #### A jq transform that matches the Service Tags schema |
| 123 | |
| 124 | The Service Tags JSON has a nested shape: |
| 125 | |
| 126 | ```text |
| 127 | { |
| 128 | "changeNumber": 123, |
| 129 | "cloud": "Public", |
| 130 | "values": [ |
| 131 | { |
| 132 | "name": "Storage.WestUS", |
| 133 | "id": "Storage.WestUS", |
| 134 | "properties": { |
| 135 | "changeNumber": 456, |
| 136 | "region": "westus", |
| 137 | "regionId": 30, |
| 138 | "platform": "Azure", |
| 139 | "systemService": "AzureStorage", |
| 140 | "addressPrefixes": ["13.105.16.4/30", "20.150.0.0/17", ...], |
| 141 | "networkFeatures": ["API", "NSG"] |
| 142 | } |
| 143 | }, |
| 144 | ... |
| 145 | ] |
| 146 | } |
| 147 | ``` |
| 148 | |
| 149 | The transform must unwrap `values[]` -> `properties.addressPrefixes[]` |
| 150 | and emit one object per prefix. Schema reference: `values[].name`, |
| 151 | `values[].id`, |
| 152 | `values[].properties.region` (region tag, can be empty for |
| 153 | cloud-wide tags like the bare `Storage`), `values[].properties.platform`, |
| 154 | `values[].properties.systemService` (e.g. `AzureStorage`, `AzureSql`), |
| 155 | `values[].properties.addressPrefixes[]` (CIDR strings, IPv4 and IPv6 |
| 156 | mixed). Top-level `changeNumber` increments on every publication. |
| 157 | |
| 158 | |
| 159 | |
| 160 | ### Configuration |
| 161 | |
| 162 | #### Options |
| 163 | |
| 164 | Add a named entry under `enrichment.network_sources` pointing at your |
| 165 | stable mirror URL (or, with the limitations above, at this week's |
| 166 | `ServiceTags_Public_<date>.json` URL). |
| 167 | |
| 168 | |
| 169 | <details open><summary>Config options</summary> |
| 170 | |
| 171 | |
| 172 | |
| 173 | | Option | Description | Default | Required | |
| 174 | |:-----|:------------|:--------|:---------:| |
| 175 | | url | Stable URL to your locally-mirrored Azure Service Tags JSON, or this week's date-stamped Microsoft URL. | | yes | |
| 176 | | interval | How often to fetch. Microsoft publishes weekly; daily is plenty. | 60s (loop floor) | no | |
| 177 | | timeout | Per-request timeout. Service Tags JSON files are large (tens of MB) -- give it room. | 60s | no | |
| 178 | | transform | jq expression that maps `values[].properties.addressPrefixes[]` into per-prefix objects. | . | yes | |
| 179 | |
| 180 | |
| 181 | </details> |
| 182 | |
| 183 | |
| 184 | |
| 185 | #### via File |
| 186 | |
| 187 | The configuration file name for this integration is `netflow.yaml`. |
| 188 | |
| 189 | |
| 190 | You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the |
| 191 | Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory). |
| 192 | |
| 193 | ```bash |
| 194 | cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata |
| 195 | sudo ./edit-config netflow.yaml |
| 196 | ``` |
| 197 | |
| 198 | ##### Examples |
| 199 | |
| 200 | ###### Azure Public via internal mirror -- tag by region and service |
| 201 | |
| 202 | Internal mirror serving the latest Public-cloud Service Tags JSON. |
| 203 | Tags every Azure prefix with `tenant=azure`, the region, and the |
| 204 | lowercase systemService name as the role. |
| 205 | |
| 206 | |
| 207 | ```yaml |
| 208 | enrichment: |
| 209 | network_sources: |
| 210 | azure: |
| 211 | url: "https://internal.example/azure-service-tags.json" |
| 212 | interval: 24h |
| 213 | timeout: 60s |
| 214 | transform: | |
| 215 | .values[] |
| 216 | | .properties as $p |
| 217 | | $p.addressPrefixes[] |
| 218 | | { |
| 219 | prefix: ., |
| 220 | tenant: "azure", |
| 221 | region: ($p.region // ""), |
| 222 | role: (($p.systemService // "") | ascii_downcase) |
| 223 | } |
| 224 | |
| 225 | ``` |
| 226 | ###### Azure Storage only |
| 227 | |
| 228 | Filter to a single Azure systemService for narrower tagging. |
| 229 | |
| 230 | <details open><summary>Config</summary> |
| 231 | |
| 232 | ```yaml |
| 233 | enrichment: |
| 234 | network_sources: |
| 235 | azure-storage: |
| 236 | url: "https://internal.example/azure-service-tags.json" |
| 237 | interval: 24h |
| 238 | transform: | |
| 239 | .values[] |
| 240 | | select(.properties.systemService == "AzureStorage") |
| 241 | | .properties as $p |
| 242 | | $p.addressPrefixes[] |
| 243 | | { |
| 244 | prefix: ., |
| 245 | tenant: "azure", |
| 246 | role: "storage", |
| 247 | region: ($p.region // "") |
| 248 | } |
| 249 | |
| 250 | ``` |
| 251 | </details> |
| 252 | |
| 253 | ###### Azure US Government cloud |
| 254 | |
| 255 | The US Government cloud is published as a separate JSON file with its |
| 256 | own download ID (57063). Same schema as the Public cloud; tag tenant |
| 257 | differently so you can tell them apart in dashboards. |
| 258 | |
| 259 | |
| 260 | <details open><summary>Config</summary> |
| 261 | |
| 262 | ```yaml |
| 263 | enrichment: |
| 264 | network_sources: |
| 265 | azure-usgov: |
| 266 | url: "https://internal.example/azure-service-tags-usgov.json" |
| 267 | interval: 24h |
| 268 | transform: | |
| 269 | .values[] |
| 270 | | .properties as $p |
| 271 | | $p.addressPrefixes[] |
| 272 | | { |
| 273 | prefix: ., |
| 274 | tenant: "azure-usgov", |
| 275 | region: ($p.region // ""), |
| 276 | role: (($p.systemService // "") | ascii_downcase) |
| 277 | } |
| 278 | |
| 279 | ``` |
| 280 | </details> |
| 281 | |
| 282 | |
| 283 | |
| 284 | ### 404 a week after deploy |
| 285 | |
| 286 | You configured `url` against `ServiceTags_Public_<date>.json` directly. |
| 287 | Microsoft rotates the date stamp weekly, so the URL stops resolving |
| 288 | within a week. Move to an internal mirror that resolves the latest URL |
| 289 | each cycle, or to the Service Tag Discovery REST API |
| 290 | (<https://learn.microsoft.com/en-us/rest/api/virtualnetwork/servicetags/list>) |
| 291 | fed into a static file. |
| 292 | |
| 293 | |
| 294 | ### Empty results from the transform |
| 295 | |
| 296 | The Service Tags JSON is nested two levels deep |
| 297 | (`values[].properties.addressPrefixes[]`). If the jq doesn't unwrap |
| 298 | both levels, every fetch yields zero rows and the source backs off as if |
| 299 | it errored. Test the jq locally with `jq < azure-service-tags.json` and |
| 300 | confirm at least one `{prefix, tenant, ...}` object comes out. |
| 301 | |
| 302 | |
| 303 | ### Service Tag API data lags the JSON file |
| 304 | |
| 305 | Per Microsoft's note on the Service Tags overview page, "It takes up to |
| 306 | four weeks for new Service Tag data to propagate in the API results |
| 307 | across all Azure regions". If you mirror via the REST API, expect newly |
| 308 | added tags to appear in the JSON download a few weeks before the API, |
| 309 | not the other way around. For prefix tagging this rarely matters; for |
| 310 | firewall rules it can. |
| 311 | |
| 312 | |
| 313 | ### TLS verification cannot be disabled |
| 314 | |
| 315 | `tls.skip_verify: true` is rejected by validation. Use `tls.ca_file` |
| 316 | for custom-CA paths (e.g., on an internal mirror with a private CA). |
| 317 | |
| 318 | |
| 319 |