| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/gcp_ip_ranges.md" |
| 3 | meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml" |
| 4 | sidebar_label: "GCP IP Ranges" |
| 5 | learn_status: "Published" |
| 6 | learn_rel_path: "Network Flows/Enrichment Methods" |
| 7 | keywords: ['gcp', 'google cloud', 'google', 'cloud', 'ip ranges', 'cloud.json', 'goog.json', 'prefix list', 'network identity'] |
| 8 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE" |
| 9 | endmeta--> |
| 10 | |
| 11 | <!-- markdownlint-disable-file --> |
| 12 | |
| 13 | # GCP IP Ranges |
| 14 | |
| 15 | |
| 16 | <img src="https://netdata.cloud/img/network-wired.svg" width="150"/> |
| 17 | |
| 18 | |
| 19 | Plugin: netflow-plugin |
| 20 | Module: gcp-ip-ranges |
| 21 | |
| 22 | <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> |
| 23 | |
| 24 | ## Overview |
| 25 | |
| 26 | Annotate network flows with Google ownership, service, and scope labels from |
| 27 | Google public IP ranges. Google publishes its public IP ranges as two static JSON |
| 28 | files served from `gstatic.com`. Wire one of them as a |
| 29 | `network_source` and the netflow plugin fetches it periodically, transforms each |
| 30 | prefix entry through your jq filter, and merges the resulting CIDR-keyed records |
| 31 | into the network-attributes trie. Flows whose source or destination IP falls |
| 32 | inside a Google-owned prefix are then labeled with `*_NET_TENANT`, |
| 33 | `*_NET_REGION`, `*_NET_ROLE`, etc. according to the labels your transform emits. |
| 34 | |
| 35 | Two upstream files exist and they are NOT interchangeable: |
| 36 | |
| 37 | - **`https://www.gstatic.com/ipranges/cloud.json`** -- the prefixes used by |
| 38 | Google Cloud (Compute Engine, GKE, Cloud Run, Cloud Functions, Cloud SQL, |
| 39 | and the rest of Google Cloud Platform). This is what you almost always want |
| 40 | for cloud-traffic attribution. |
| 41 | - **`https://www.gstatic.com/ipranges/goog.json`** -- the broader Google IP |
| 42 | space, including consumer-facing properties (Search, Gmail, YouTube, Google |
| 43 | Workspace, Maps, Ads, etc.). It is a strict superset of `cloud.json` but |
| 44 | **carries no service or region metadata** -- only the bare CIDRs. |
| 45 | |
| 46 | For the cross-cutting network-identity behavior (merge order with GeoIP and |
| 47 | static config, jq output contract, TLS rules, single-page fetch, failure / |
| 48 | backoff handling), see |
| 49 | [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment). |
| 50 | |
| 51 | |
| 52 | Periodic HTTPS GET against the chosen `gstatic.com` URL. The body is parsed |
| 53 | as JSON and the configured `transform` (a [jaq](https://github.com/01mf02/jaq) |
| 54 | jq-equivalent expression) runs over the parsed value. Each output object is |
| 55 | merged into the network-attributes trie keyed on its `prefix` field; remaining |
| 56 | fields (`name`, `role`, `site`, `region`, `country`, `state`, `city`, `tenant`, |
| 57 | `asn`, `asn_name`) populate the corresponding `*_NET_*` flow-record columns |
| 58 | when a flow's IP matches the prefix. |
| 59 | |
| 60 | The upstream files are static JSON snapshots, not pageable APIs -- one fetch |
| 61 | per cycle is the whole dataset. Each contains a `syncToken` and a |
| 62 | `creationTime` you can inspect to see when Google last regenerated them. |
| 63 | The plugin fetches the full document on each refresh cycle. |
| 64 | |
| 65 | |
| 66 | This integration is only supported on the following platforms: |
| 67 | |
| 68 | - Linux |
| 69 | |
| 70 | This integration supports multiple instances configured side-by-side. |
| 71 | |
| 72 | |
| 73 | ### Default Behavior |
| 74 | |
| 75 | #### Auto-Detection |
| 76 | |
| 77 | Disabled by default. Add an entry under enrichment.network_sources to enable. |
| 78 | |
| 79 | #### Limits |
| 80 | |
| 81 | One full GCP prefix document is fetched per refresh. Resource use scales with the number of GCP prefixes selected by your transform and the refresh interval. |
| 82 | |
| 83 | #### Performance Impact |
| 84 | |
| 85 | One short HTTPS GET per `interval`, plus a jq pass over the response. |
| 86 | `cloud.json` is on the order of ~1000 prefix entries and the trie merge |
| 87 | is bounded by that count; runtime cost is negligible against typical flow |
| 88 | ingest. `goog.json` is similar in size. |
| 89 | |
| 90 | |
| 91 | ## Setup |
| 92 | |
| 93 | |
| 94 | ### Prerequisites |
| 95 | |
| 96 | #### Outbound HTTPS to gstatic.com |
| 97 | |
| 98 | The agent host must be able to reach `https://www.gstatic.com/ipranges/cloud.json` |
| 99 | (or `goog.json`). The files are public CDN-served static JSON; no Google |
| 100 | Cloud project, no service account, no API key, and no IAM permission is |
| 101 | required. |
| 102 | |
| 103 | |
| 104 | #### Pick the right file for your use case |
| 105 | |
| 106 | - Use `cloud.json` to attribute traffic to/from **Google Cloud Platform** |
| 107 | services (Compute Engine VMs, GKE, Cloud Run, Cloud Functions, Cloud |
| 108 | SQL, BigQuery API, Pub/Sub, GCS endpoints, etc.). Carries `service` and |
| 109 | `scope` (region) per prefix. |
| 110 | - Use `goog.json` to also catch **non-cloud Google traffic** (Gmail, |
| 111 | YouTube, Google Workspace, Search, Ads, Maps, etc.). It carries no |
| 112 | `service` or `scope` -- you only get "this is Google" attribution. |
| 113 | - Configure both as separate sources if you want both attributions and |
| 114 | a deterministic merge order between them. Within network-identity, more- |
| 115 | specific prefixes win at lookup time, so a `cloud.json` match for a |
| 116 | Compute Engine prefix will out-rank the broader `goog.json` umbrella. |
| 117 | |
| 118 | |
| 119 | |
| 120 | ### Configuration |
| 121 | |
| 122 | #### Options |
| 123 | |
| 124 | Add a named entry under `enrichment.network_sources`. The entry name is |
| 125 | cosmetic (used in logs and metrics); the labels that end up in flow records |
| 126 | come from the fields your `transform` emits. |
| 127 | |
| 128 | |
| 129 | <details open><summary>Config options</summary> |
| 130 | |
| 131 | |
| 132 | |
| 133 | | Option | Description | Default | Required | |
| 134 | |:-----|:------------|:--------|:---------:| |
| 135 | | url | The Google IP-ranges JSON URL. Use `cloud.json` for GCP services or `goog.json` for the broader Google IP space. | https://www.gstatic.com/ipranges/cloud.json | yes | |
| 136 | | interval | How often to refetch. Google does not publish a fixed cadence; their documentation states the lists are "published and updated frequently". Daily is plenty for most deployments and avoids hammering the CDN. | 60s (loop floor) | no | |
| 137 | | timeout | Per-request timeout. | 60s | no | |
| 138 | | transform | jq expression that converts each entry in `prefixes[]` into an object with a `prefix` field (CIDR string) plus any of the optional label fields (`name`, `role`, `site`, `region`, `country`, `state`, `city`, `tenant`, `asn`, `asn_name`). Required. | . | yes | |
| 139 | |
| 140 | |
| 141 | </details> |
| 142 | |
| 143 | |
| 144 | |
| 145 | #### via File |
| 146 | |
| 147 | The configuration file name for this integration is `netflow.yaml`. |
| 148 | |
| 149 | |
| 150 | You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the |
| 151 | Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory). |
| 152 | |
| 153 | ```bash |
| 154 | cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata |
| 155 | sudo ./edit-config netflow.yaml |
| 156 | ``` |
| 157 | |
| 158 | ##### Examples |
| 159 | |
| 160 | ###### GCP cloud.json -- tag Google Cloud prefixes by region |
| 161 | |
| 162 | Tags every flow whose IP matches a Google Cloud prefix with |
| 163 | `*_NET_TENANT="gcp"`, `*_NET_ROLE="google-cloud"`, and |
| 164 | `*_NET_REGION` set to the GCP scope (`us-central1`, `europe-west1`, |
| 165 | `global`, etc.). Both `ipv4Prefix` and `ipv6Prefix` entries are |
| 166 | covered by the `// (slash-or)` fallback. |
| 167 | |
| 168 | |
| 169 | ```yaml |
| 170 | enrichment: |
| 171 | network_sources: |
| 172 | gcp: |
| 173 | url: "https://www.gstatic.com/ipranges/cloud.json" |
| 174 | interval: 24h |
| 175 | timeout: 60s |
| 176 | transform: | |
| 177 | .prefixes[] | { |
| 178 | prefix: (.ipv4Prefix // .ipv6Prefix), |
| 179 | tenant: "gcp", |
| 180 | role: "google-cloud", |
| 181 | region: .scope |
| 182 | } |
| 183 | |
| 184 | ``` |
| 185 | ###### GCP cloud.json -- IPv4 only |
| 186 | |
| 187 | Same as above but skips the IPv6 prefixes (some operators only need |
| 188 | IPv4 attribution and prefer to keep the trie smaller). |
| 189 | |
| 190 | |
| 191 | <details open><summary>Config</summary> |
| 192 | |
| 193 | ```yaml |
| 194 | enrichment: |
| 195 | network_sources: |
| 196 | gcp-v4: |
| 197 | url: "https://www.gstatic.com/ipranges/cloud.json" |
| 198 | interval: 24h |
| 199 | transform: | |
| 200 | .prefixes[] |
| 201 | | select(.ipv4Prefix) |
| 202 | | { |
| 203 | prefix: .ipv4Prefix, |
| 204 | tenant: "gcp", |
| 205 | role: "google-cloud", |
| 206 | region: .scope |
| 207 | } |
| 208 | |
| 209 | ``` |
| 210 | </details> |
| 211 | |
| 212 | ###### Tag a single GCP region |
| 213 | |
| 214 | Narrow the source to one region (here `us-central1`) so you can build |
| 215 | dashboards that distinguish that region from the rest of GCP. |
| 216 | |
| 217 | |
| 218 | <details open><summary>Config</summary> |
| 219 | |
| 220 | ```yaml |
| 221 | enrichment: |
| 222 | network_sources: |
| 223 | gcp-us-central1: |
| 224 | url: "https://www.gstatic.com/ipranges/cloud.json" |
| 225 | interval: 24h |
| 226 | transform: | |
| 227 | .prefixes[] |
| 228 | | select(.scope == "us-central1") |
| 229 | | { |
| 230 | prefix: (.ipv4Prefix // .ipv6Prefix), |
| 231 | tenant: "gcp", |
| 232 | role: "google-cloud", |
| 233 | region: "us-central1", |
| 234 | site: "us-central1" |
| 235 | } |
| 236 | |
| 237 | ``` |
| 238 | </details> |
| 239 | |
| 240 | ###### Broader Google IP space (goog.json) |
| 241 | |
| 242 | Use `goog.json` to also attribute traffic to non-Cloud Google services |
| 243 | (Gmail, YouTube, Workspace, Search, Ads, Maps). The file has no |
| 244 | `service` or `scope` fields, so only `tenant` / `role` are set. |
| 245 | Configure this **alongside** `cloud.json` if you want both -- the more |
| 246 | specific GCP prefixes from `cloud.json` will still win at lookup time. |
| 247 | |
| 248 | |
| 249 | <details open><summary>Config</summary> |
| 250 | |
| 251 | ```yaml |
| 252 | enrichment: |
| 253 | network_sources: |
| 254 | google-all: |
| 255 | url: "https://www.gstatic.com/ipranges/goog.json" |
| 256 | interval: 24h |
| 257 | transform: | |
| 258 | .prefixes[] | { |
| 259 | prefix: (.ipv4Prefix // .ipv6Prefix), |
| 260 | tenant: "google", |
| 261 | role: "google" |
| 262 | } |
| 263 | |
| 264 | ``` |
| 265 | </details> |
| 266 | |
| 267 | |
| 268 | |
| 269 | ### cloud.json vs goog.json -- pick the right one |
| 270 | |
| 271 | `cloud.json` is the Google Cloud Platform list (Compute Engine, GKE, |
| 272 | Cloud Run, Cloud SQL, BigQuery, GCS, etc.) and carries `service` plus |
| 273 | `scope` (region) per entry. `goog.json` is the broader Google list |
| 274 | (consumer products: Gmail, YouTube, Workspace, Search, Ads, Maps) and |
| 275 | carries **only the bare CIDRs** -- no service, no scope, no region. |
| 276 | Using `goog.json` as if it were `cloud.json` will give you "this is |
| 277 | Google" attribution but no per-region or per-service breakdown. Most |
| 278 | operators want `cloud.json`; some configure both as separate sources. |
| 279 | |
| 280 | |
| 281 | ### No per-service breakdown from cloud.json |
| 282 | |
| 283 | Today every entry in `cloud.json` reports `service: "Google Cloud"` -- |
| 284 | the file does not split prefixes by individual GCP service (Compute |
| 285 | Engine vs GKE vs Cloud Run, etc.). Region (`scope`) is the dimension |
| 286 | you can actually pivot on. Per-service attribution requires a different |
| 287 | data source. |
| 288 | |
| 289 | |
| 290 | ### Empty result from the transform is treated as failure |
| 291 | |
| 292 | If the jq expression yields zero objects (for example, an over-narrow |
| 293 | `select()` that no entry passes), the source backs off as if the fetch |
| 294 | had errored. Check the journal for `network-sources` warnings and verify |
| 295 | your filter against a saved copy of `cloud.json`. |
| 296 | |
| 297 | |
| 298 | ### TLS verification cannot be disabled |
| 299 | |
| 300 | `tls.skip_verify: true` (and `tls.verify: false`) are rejected by |
| 301 | validation. `gstatic.com` is publicly trusted so this is rarely an issue |
| 302 | for GCP; if you front the URL through an internal proxy with a private |
| 303 | CA, supply it via `tls.ca_file`. |
| 304 | |
| 305 | |
| 306 | ### Update cadence is not contractual |
| 307 | |
| 308 | Google states the lists are "published and updated frequently" but does |
| 309 | not guarantee a fixed cadence. The `syncToken` and `creationTime` keys |
| 310 | at the top of each file let you check when Google last regenerated it. |
| 311 | Daily polling is a safe default; sub-hourly is unnecessary. |
| 312 | |
| 313 | |
| 314 |