| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/aws_ip_ranges.md" |
| 3 | meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml" |
| 4 | sidebar_label: "AWS IP Ranges" |
| 5 | learn_status: "Published" |
| 6 | learn_rel_path: "Network Flows/Enrichment Methods" |
| 7 | keywords: ['aws', 'amazon', 'cloud', 'ip ranges', 'vpc', 'ec2', 'prefix list', 'service tags'] |
| 8 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE" |
| 9 | endmeta--> |
| 10 | |
| 11 | <!-- markdownlint-disable-file --> |
| 12 | |
| 13 | # AWS IP Ranges |
| 14 | |
| 15 | |
| 16 | <img src="https://netdata.cloud/img/network-wired.svg" width="150"/> |
| 17 | |
| 18 | |
| 19 | Plugin: netflow-plugin |
| 20 | Module: aws-ip-ranges |
| 21 | |
| 22 | <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> |
| 23 | |
| 24 | ## Overview |
| 25 | |
| 26 | Annotate network flows with AWS service and region labels from AWS public IP |
| 27 | ranges. AWS publishes its current public IP allocations as a single JSON document at |
| 28 | [`https://ip-ranges.amazonaws.com/ip-ranges.json`](https://ip-ranges.amazonaws.com/ip-ranges.json). |
| 29 | The file lists every prefix AWS uses for a published service, tagged with the AWS |
| 30 | region and the AWS service that owns the prefix (`AMAZON`, `EC2`, `S3`, |
| 31 | `CLOUDFRONT`, `API_GATEWAY`, ...). This integration fetches the file periodically, |
| 32 | transforms it via a jq expression, and labels matching flow records with provider, |
| 33 | region, and service tags. |
| 34 | |
| 35 | Once configured, traffic to / from AWS shows up in dashboards as `tenant=amazon` |
| 36 | with per-region (`region=us-east-1`, `region=eu-central-1`, ...) and per-service |
| 37 | (`role=s3`, `role=cloudfront`, ...) breakdown. This is **complementary to GeoIP |
| 38 | and ASN enrichment**: instead of a flow being labeled simply `AS16509 Amazon`, |
| 39 | it becomes labeled with the specific AWS service and region carrying the |
| 40 | traffic -- without parsing AS-name strings. |
| 41 | |
| 42 | The file is public; no AWS credentials are needed. |
| 43 | |
| 44 | Schema of `ip-ranges.json`: |
| 45 | |
| 46 | - top-level: `syncToken`, `createDate`, `prefixes`, `ipv6_prefixes` |
| 47 | - each entry in `prefixes[]`: `ip_prefix`, `region`, `service`, `network_border_group` |
| 48 | - each entry in `ipv6_prefixes[]`: `ipv6_prefix`, `region`, `service`, `network_border_group` |
| 49 | |
| 50 | For the cross-cutting concepts (how multiple network sources merge, the lookup |
| 51 | priority vs static `networks` config, TLS verification rules, the jq engine and |
| 52 | its expected output schema, fetch loop and retry/backoff), see |
| 53 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 54 | |
| 55 | |
| 56 | The plugin issues a periodic GET to `https://ip-ranges.amazonaws.com/ip-ranges.json`, |
| 57 | parses the JSON body, runs the configured jq transform via the |
| 58 | [jaq](https://github.com/01mf02/jaq) library, and merges the resulting |
| 59 | prefix-labeled rows into the network-attributes trie shared by all enrichment |
| 60 | sources. |
| 61 | |
| 62 | AWS does not publish a fixed refresh cadence; the file is updated whenever the |
| 63 | AWS IP space changes (typically several times per day) and the `syncToken` / |
| 64 | `createDate` fields advance on every change. AWS recommends polling no faster |
| 65 | than the file actually changes; daily is enough for most flow-attribution use |
| 66 | cases. The plugin floors the configured `interval` at 60s. |
| 67 | |
| 68 | |
| 69 | This integration is only supported on the following platforms: |
| 70 | |
| 71 | - Linux |
| 72 | |
| 73 | This integration supports multiple instances configured side-by-side. |
| 74 | |
| 75 | |
| 76 | ### Default Behavior |
| 77 | |
| 78 | #### Auto-Detection |
| 79 | |
| 80 | Disabled by default. Add an entry under `enrichment.network_sources` to enable. |
| 81 | |
| 82 | #### Limits |
| 83 | |
| 84 | One full AWS prefix document is fetched per refresh. Resource use scales with the number of AWS prefixes selected by your transform and the refresh interval. |
| 85 | |
| 86 | #### Performance Impact |
| 87 | |
| 88 | One HTTPS request per refresh interval plus a jq transform over the AWS prefix document. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records. |
| 89 | |
| 90 | ## Setup |
| 91 | |
| 92 | |
| 93 | ### Prerequisites |
| 94 | |
| 95 | #### Outbound HTTPS to AWS |
| 96 | |
| 97 | The agent host must be able to reach `https://ip-ranges.amazonaws.com/ip-ranges.json`. |
| 98 | No AWS credentials are needed -- the file is public. |
| 99 | |
| 100 | |
| 101 | |
| 102 | ### Configuration |
| 103 | |
| 104 | #### Options |
| 105 | |
| 106 | Add a named entry under `enrichment.network_sources`. The map key (e.g. `aws`) |
| 107 | is the source name; it is used in plugin logs but is **not** automatically |
| 108 | attached to flow records -- the record labels come entirely from your `transform` |
| 109 | output (`tenant`, `region`, `role`, `name`, `site`, `city`, `country`, `state`, |
| 110 | `asn`, `asn_name`). |
| 111 | |
| 112 | The expected `transform` output is a stream of objects, each carrying a `prefix` |
| 113 | (CIDR string) plus any subset of the optional attribute fields. See the |
| 114 | cross-cutting [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment) |
| 115 | page for the full output schema. |
| 116 | |
| 117 | |
| 118 | <details open><summary>Config options</summary> |
| 119 | |
| 120 | |
| 121 | |
| 122 | | Option | Description | Default | Required | |
| 123 | |:-----|:------------|:--------|:---------:| |
| 124 | | url | AWS publishes the master file at this URL. Use it as-is unless you mirror it locally for air-gapped environments. | https://ip-ranges.amazonaws.com/ip-ranges.json | yes | |
| 125 | | interval | How often to fetch the file. AWS updates the document whenever its IP space changes (often several times per day, sometimes more); daily is enough for most uses. The configured value is floored at 60s. | 60s | no | |
| 126 | | timeout | Per-request timeout for the HTTPS GET. | 60s | no | |
| 127 | | method | HTTP method. AWS serves the file via GET; leave at the default. | GET | no | |
| 128 | | headers | Extra HTTP headers added to the request. Not required for the public AWS URL; only needed if you front the file behind your own authenticated mirror. | {} | no | |
| 129 | | transform | jq expression (compiled by jaq) that converts the AWS JSON into a stream of `{prefix, ...}` objects. The default `.` does **not** match the AWS schema -- you must supply a real transform (see examples below) or fetches fail because output rows cannot be mapped to the required `{prefix, ...}` schema. | . | yes | |
| 130 | |
| 131 | |
| 132 | </details> |
| 133 | |
| 134 | |
| 135 | |
| 136 | #### via File |
| 137 | |
| 138 | The configuration file name for this integration is `netflow.yaml`. |
| 139 | |
| 140 | |
| 141 | You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the |
| 142 | Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory). |
| 143 | |
| 144 | ```bash |
| 145 | cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata |
| 146 | sudo ./edit-config netflow.yaml |
| 147 | ``` |
| 148 | |
| 149 | ##### Examples |
| 150 | |
| 151 | ###### Tag all AWS prefixes by region and service (IPv4 + IPv6) |
| 152 | |
| 153 | Recommended starting point. Sets `tenant=amazon`, `region=<aws-region>`, |
| 154 | `role=<service-name-lowercased>`. Covers both `prefixes[]` and |
| 155 | `ipv6_prefixes[]` in one stream so v4 and v6 traffic are tagged |
| 156 | consistently. |
| 157 | |
| 158 | |
| 159 | ```yaml |
| 160 | enrichment: |
| 161 | network_sources: |
| 162 | aws: |
| 163 | url: "https://ip-ranges.amazonaws.com/ip-ranges.json" |
| 164 | interval: 24h |
| 165 | timeout: 60s |
| 166 | transform: | |
| 167 | (.prefixes + .ipv6_prefixes)[] | { |
| 168 | prefix: (.ip_prefix // .ipv6_prefix), |
| 169 | tenant: "amazon", |
| 170 | region: .region, |
| 171 | role: (.service | ascii_downcase) |
| 172 | } |
| 173 | |
| 174 | ``` |
| 175 | ###### AWS S3 only |
| 176 | |
| 177 | Filter to a single AWS service for narrower tagging. Useful when you only |
| 178 | care about identifying S3 traffic (e.g. egress-cost attribution). |
| 179 | |
| 180 | |
| 181 | <details open><summary>Config</summary> |
| 182 | |
| 183 | ```yaml |
| 184 | enrichment: |
| 185 | network_sources: |
| 186 | aws-s3: |
| 187 | url: "https://ip-ranges.amazonaws.com/ip-ranges.json" |
| 188 | interval: 24h |
| 189 | transform: | |
| 190 | (.prefixes + .ipv6_prefixes)[] |
| 191 | | select(.service == "S3") |
| 192 | | { |
| 193 | prefix: (.ip_prefix // .ipv6_prefix), |
| 194 | tenant: "amazon", |
| 195 | role: "s3", |
| 196 | region: .region |
| 197 | } |
| 198 | |
| 199 | ``` |
| 200 | </details> |
| 201 | |
| 202 | ###### Use network_border_group as the site label |
| 203 | |
| 204 | AWS exposes a `network_border_group` field that distinguishes Local Zones |
| 205 | / Wavelength Zones from the parent Region. Map it to the `site` attribute |
| 206 | if you want that distinction visible in flow dashboards. |
| 207 | |
| 208 | |
| 209 | <details open><summary>Config</summary> |
| 210 | |
| 211 | ```yaml |
| 212 | enrichment: |
| 213 | network_sources: |
| 214 | aws: |
| 215 | url: "https://ip-ranges.amazonaws.com/ip-ranges.json" |
| 216 | interval: 24h |
| 217 | transform: | |
| 218 | (.prefixes + .ipv6_prefixes)[] | { |
| 219 | prefix: (.ip_prefix // .ipv6_prefix), |
| 220 | tenant: "amazon", |
| 221 | region: .region, |
| 222 | site: .network_border_group, |
| 223 | role: (.service | ascii_downcase) |
| 224 | } |
| 225 | |
| 226 | ``` |
| 227 | </details> |
| 228 | |
| 229 | |
| 230 | |
| 231 | ### Default `transform: "."` fails because output rows are missing `prefix` |
| 232 | |
| 233 | The default `.` returns the raw JSON object, not the per-prefix stream the |
| 234 | plugin expects. You **must** supply a transform that yields one object per |
| 235 | prefix (with at least a `prefix` field). Use the first example above as your |
| 236 | starting template. |
| 237 | |
| 238 | |
| 239 | ### AWS service name precedence inside one prefix |
| 240 | |
| 241 | AWS sometimes lists the same `ip_prefix` under multiple `service` entries |
| 242 | (e.g. once under `AMAZON` and again under `EC2`). The plugin merges all |
| 243 | records into a single prefix trie, and the **last write wins** for a given |
| 244 | prefix length. If you want a specific service to take precedence, filter the |
| 245 | transform with `select(.service != "AMAZON")` so the broader catch-all |
| 246 | entries are dropped. |
| 247 | |
| 248 | |
| 249 | ### Static `networks:` config silently overrides AWS labels |
| 250 | |
| 251 | When a prefix is defined in both this source and `enrichment.networks`, the |
| 252 | static config wins on every non-empty field. This is by design (operator |
| 253 | intent overrides imported data) but can surprise operators who expect the |
| 254 | remote feed to be authoritative. See the cross-cutting Network Identity page |
| 255 | for the full lookup-priority rules. |
| 256 | |
| 257 | |
| 258 | ### Empty result from the transform is treated as failure |
| 259 | |
| 260 | If the jq filter happens to produce nothing (bad selector, wrong field name, |
| 261 | upstream JSON shape change), the source backs off as if it errored. Check |
| 262 | the journal for `network-sources` warnings: |
| 263 | `journalctl --namespace netdata | grep network-sources`. |
| 264 | |
| 265 | |
| 266 | ### TLS verification cannot be disabled |
| 267 | |
| 268 | `tls.skip_verify: true` is rejected by validation. Use `tls.ca_file` for |
| 269 | custom-CA paths if you front AWS behind an internal TLS-terminating proxy |
| 270 | with a private CA. |
| 271 | |
| 272 | |
| 273 |