@cryptotaxi247 / netdata
Code Commits Issues Pulls Sessions CI/CD
master
master
md 364 lines 13.5 KB
Rendered Raw
1 <!--startmeta
2 custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/integrations/generic_json-over-http_ipam.md"
3 meta_yaml: "https://github.com/netdata/netdata/edit/master/src/crates/netflow-plugin/metadata.yaml"
4 sidebar_label: "Generic JSON-over-HTTP IPAM"
5 learn_status: "Published"
6 learn_rel_path: "Network Flows/Enrichment Methods"
7 keywords: ['ipam', 'cmdb', 'infoblox', 'bluecat', 'phpipam', 'servicenow', 'custom', 'prefix list', 'network identity']
8 message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE FLOWS' metadata.yaml FILE"
9 endmeta-->
10
11 <!-- markdownlint-disable-file -->
12
13 # Generic JSON-over-HTTP IPAM
14
15
16 <img src="https://netdata.cloud/img/network-wired.svg" width="150"/>
17
18
19 Plugin: netflow-plugin
20 Module: generic-ipam
21
22 <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" />
23
24 ## Overview
25
26 Annotate network flows with network-identity labels from any JSON-over-HTTP IPAM
27 or CMDB endpoint. The plugin does not care WHO produces the JSON -- it cares
28 about the JSON shape after your `transform` (a jq expression) runs over it.
29 Examples: Infoblox WAPI, BlueCat REST API, phpIPAM, ServiceNow CMDB queries, an
30 internal aggregator, a Lambda function, a static file served from S3, a
31 hand-rolled CMDB.
32
33 This card is also the **reference description** of how all the other Network
34 Identity cards work under the hood. AWS IP Ranges, GCP IP Ranges, Azure IP Ranges
35 and NetBox are all special-cased instances of this same mechanism: each is just
36 a known URL plus a recommended jq `transform`. Read this card to understand the
37 generic shape, then read the per-source cards for the URL and transform that fit
38 that specific provider.
39
40 For the full network-identity concept (merge order vs GeoIP and static, what
41 fields you can populate, why TLS verification cannot be disabled), see
42 [Network Identity](https://learn.netdata.cloud/docs/network-flows/enrichment).
43
44
45 The plugin issues a periodic HTTP request (GET by default, POST optionally) to
46 your configured URL with the headers you specify, parses the response as JSON,
47 runs your `transform` jq expression (compiled by the
48 [jaq](https://github.com/01mf02/jaq) library) over the parsed body, and merges
49 the resulting per-prefix rows into the network-attributes trie.
50
51 The same flow applies to every JSON-over-HTTP source; the integration-specific
52 part is the URL and the `transform` expression.
53
54
55 This integration is only supported on the following platforms:
56
57 - Linux
58
59 This integration supports multiple instances configured side-by-side.
60
61
62 ### Default Behavior
63
64 #### Auto-Detection
65
66 Disabled by default. Add a named entry under `enrichment.network_sources` for each IPAM/CMDB/endpoint you want to import.
67
68 #### Limits
69
70 Resource use scales with response size, transform complexity, refresh interval, and the number of emitted prefixes. Empty transform output is treated as a fetch failure.
71
72 #### Performance Impact
73
74 One HTTP request per refresh interval plus a jq transform over the response. Runtime enrichment does prefix matching for source and destination IPs, and cost scales with the number of loaded network-source records.
75
76 ## Setup
77
78
79 ### Prerequisites
80
81 #### An HTTP/HTTPS endpoint returning JSON
82
83 The endpoint must respond with a parseable JSON document (the plugin sets
84 `Accept: application/json`). Only GET and POST are accepted as request
85 methods. There is no pagination, no
86 cursor following, no `Link: rel=next` handling -- the fetch is one-shot
87 per cycle. If your IPAM paginates, expose a bulk endpoint or wrap it in a
88 server-side aggregator that returns the full list at one URL.
89
90
91 #### Authentication via headers
92
93 The plugin has no built-in OAuth flow, basic-auth helper, or token
94 refresh. Whatever the API needs (bearer tokens, API keys, custom header
95 names, basic-auth realms) goes into the `headers:` map. The header map is
96 an arbitrary string-to-string mapping, so any single-shot scheme works.
97 For short-lived tokens, refresh them outside Netdata and reload the
98 plugin config.
99
100
101 #### A POST endpoint must accept an empty body
102
103 When `method: POST` is configured, the plugin sends the request with the
104 configured headers but **no request body**. If your CMDB requires a JSON query body to return prefixes,
105 wrap it server-side with an endpoint that accepts GET (or POST with no
106 body) and returns the full prefix set.
107
108
109
110 ### Configuration
111
112 #### Options
113
114 Add a named entry under `enrichment.network_sources`. Unknown keys cause a
115 config error.
116
117
118 <details open><summary>Config options</summary>
119
120
121
122 | Option | Description | Default | Required |
123 |:-----|:------------|:--------|:---------:|
124 | url | HTTP/HTTPS endpoint. Required (a non-empty URL is enforced by validation). | | yes |
125 | method | HTTP method. Only `GET` and `POST` are accepted; anything else fails validation. Note that `POST` is sent with no request body. | GET | no |
126 | headers | Map of additional HTTP request headers. Use this for any authentication scheme (`Authorization: Bearer ...`, `token: ...`, custom API-key headers, basic-auth realms encoded explicitly, etc.). Values are passed through verbatim. | {} | no |
127 | interval | Refresh cadence. Values below 60 seconds are accepted, but refreshes still run no faster than once per minute. Pick the value that matches how often your IPAM actually changes (5-15 minutes for a curated CMDB; daily for slow-moving prefix lists). | 60s | no |
128 | timeout | Per-request timeout. Must be greater than 0. | 60s | no |
129 | proxy | Whether to honor the system HTTP/HTTPS proxy environment variables. Set to `false` to bypass the proxy for this source (useful when the IPAM is on the internal network and the proxy is for outbound traffic only). | true | no |
130 | tls.enable | Enables custom TLS settings (custom CA bundle, mTLS client certificate). Must be `true` whenever any of `tls.ca_file`, `tls.cert_file`, `tls.key_file` is set. | false | no |
131 | tls.verify | Certificate verification toggle. **Setting this to `false` is rejected by validation** -- network-identity data flows directly into security-relevant enrichment, so MITM-able responses are not allowed. Use `tls.ca_file` to trust internal CAs instead. | true | no |
132 | tls.skip_verify | Legacy alias for the same intent as `tls.verify: false`. **Setting this to `true` is rejected by validation.** Use `tls.ca_file`. | false | no |
133 | tls.ca_file | PEM file with the CA bundle to trust for this endpoint (instead of, or in addition to, the system roots). The recommended way to talk to an internal IPAM behind your own PKI. | | no |
134 | tls.cert_file | PEM file with the client certificate, for mTLS-protected endpoints. | | no |
135 | tls.key_file | PEM file with the client private key. Required only when the key is in a separate file from the certificate; if omitted, the cert file is reused. `tls.cert_file` must also be set whenever `tls.key_file` is set. | | no |
136 | transform | jq expression compiled by [jaq](https://github.com/01mf02/jaq). Receives the entire parsed JSON body once and must produce a **stream of objects**, where each object has at minimum a `prefix` field (a CIDR string -- `IPv4/len` or `IPv6/len`) plus any of the optional attribute fields: `name`, `role`, `site`, `region`, `country`, `state`, `city`, `tenant`, `asn`, `asn_name`. Any field not produced is treated as empty. The transform compiles at startup -- a syntax error fails the config load. An empty stream at runtime is treated as a fetch failure and triggers backoff. | . | yes |
137
138
139 </details>
140
141
142
143 #### via File
144
145 The configuration file name for this integration is `netflow.yaml`.
146
147
148 You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the
149 Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory).
150
151 ```bash
152 cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
153 sudo ./edit-config netflow.yaml
154 ```
155
156 ##### Examples
157
158 ###### Flat list with `{prefix, name, env}`
159
160 The simplest IPAM shape: a top-level array of subnets, each with a CIDR
161 string and a few labels. Maps `env` -> tenant and the `name` -> NET_NAME.
162
163
164 ```yaml
165 enrichment:
166 network_sources:
167 custom_ipam:
168 url: "https://ipam.corp.example/api/v1/prefixes"
169 headers:
170 Authorization: "Bearer abcdef0123456789"
171 interval: 5m
172 transform: |
173 .[] | {
174 prefix: .prefix,
175 name: .name,
176 tenant: .env
177 }
178
179 ```
180 ###### phpIPAM with API token
181
182 phpIPAM exposes `/api/<APP>/subnets/`. Replace `<APP>` with your phpIPAM
183 application name. phpIPAM splits the address into a base and a mask --
184 the transform reassembles them into a CIDR string.
185
186
187 <details open><summary>Config</summary>
188
189 ```yaml
190 enrichment:
191 network_sources:
192 phpipam:
193 url: "https://ipam.example/api/netdata/subnets/"
194 headers:
195 token: "abcdef..."
196 interval: 10m
197 transform: |
198 .data[] | {
199 prefix: (.subnet + "/" + (.mask|tostring)),
200 name: .description,
201 tenant: (.custom_tenant // ""),
202 site: (.location.name // "")
203 }
204
205 ```
206 </details>
207
208 ###### Hierarchical CMDB export
209
210 A CMDB that returns nested objects -- top-level sites, each with a list
211 of vlans, each with a list of prefixes. Flattens to one row per prefix,
212 inheriting site and role labels from the parent.
213
214
215 <details open><summary>Config</summary>
216
217 ```yaml
218 enrichment:
219 network_sources:
220 cmdb:
221 url: "https://cmdb.example/export/networks.json"
222 headers:
223 Authorization: "Bearer <CMDB_TOKEN>"
224 interval: 15m
225 transform: |
226 .sites[] as $site
227 | $site.vlans[] as $vlan
228 | $vlan.prefixes[]
229 | {
230 prefix: .cidr,
231 name: .label,
232 site: $site.name,
233 region: $site.region,
234 role: $vlan.role,
235 tenant: $vlan.tenant
236 }
237
238 ```
239 </details>
240
241 ###### Internal IPAM behind mTLS
242
243 When the IPAM is fronted by your internal PKI and requires a client
244 certificate. `tls.enable: true` activates the custom TLS path; `tls.verify`
245 must remain true (cannot be disabled).
246
247
248 <details open><summary>Config</summary>
249
250 ```yaml
251 enrichment:
252 network_sources:
253 corp_ipam:
254 url: "https://ipam.corp/api/networks"
255 tls:
256 enable: true
257 ca_file: /etc/netdata/ssl/corp-ca.pem
258 cert_file: /etc/netdata/ssl/netdata.crt
259 key_file: /etc/netdata/ssl/netdata.key
260 interval: 10m
261 transform: |
262 .[] | {
263 prefix: .cidr,
264 name: .label,
265 tenant: .tenant
266 }
267
268 ```
269 </details>
270
271 ###### Source with the system proxy bypassed
272
273 When the agent host has an outbound HTTP proxy configured but the IPAM
274 lives on the internal network and should be reached directly.
275
276
277 <details open><summary>Config</summary>
278
279 ```yaml
280 enrichment:
281 network_sources:
282 internal_ipam:
283 url: "https://ipam.internal/api/networks"
284 proxy: false
285 headers:
286 Authorization: "Bearer ..."
287 interval: 5m
288 transform: |
289 .[] | { prefix: .cidr, name: .label }
290
291 ```
292 </details>
293
294
295
296 ### Endpoint requires pagination
297
298 The plugin does not paginate. Either raise the page size in the URL to
299 cover your full inventory, or wrap the endpoint with a server-side
300 aggregator that returns all results at one URL. There is no built-in
301 `Link: rel=next` follower.
302
303
304 ### POST endpoint requires a request body
305
306 The plugin's POST request is sent with no body. If your CMDB requires a
307 JSON query body to return prefixes, wrap it server-side with an endpoint
308 that accepts GET (or accepts POST with no body) and returns the full prefix
309 set.
310
311
312 ### TLS verification cannot be disabled
313
314 `tls.verify: false` and `tls.skip_verify: true` are both rejected during
315 configuration validation. Use `tls.ca_file` to trust internal CAs. This is
316 deliberate -- network-identity rows feed enrichment used in security
317 investigations and capacity decisions, where silently accepting MITM-able
318 responses would corrupt every downstream analysis.
319
320
321 ### Empty result back-off
322
323 An empty stream from the jq transform is treated as a fetch failure.
324 The source then backs
325 off exponentially -- starting at `interval / 10` (floor 1s), doubling on
326 each consecutive failure, and capped at the regular `interval`. On the
327 next successful non-empty fetch the cadence resets to `interval`. If your
328 IPAM legitimately has no prefixes (a quiet state), have the upstream
329 return at least one synthetic prefix so the source does not back off.
330
331
332 ### Refresh appears slower than configured
333
334 The fetch loop floors the configured `interval` at 60 seconds. Configuring
335 `interval: 5s` does not produce a 5-second loop -- it produces a 60-second
336 loop. Pick a value at or above 60s that matches how often your IPAM
337 actually changes (5-15 minutes is typical for a curated CMDB).
338
339
340 ### Unknown config keys cause errors
341
342 The config struct uses `deny_unknown_fields`. Typos like `headres:` or a
343 non-existent option fail config load with a parse error rather than being
344 silently ignored. Check the `enrichment.network_sources.<name>` schema
345 listed under "Config options".
346
347
348 ### JSON parse errors are silent in the dashboard
349
350 Decode failures (HTTP error, JSON parse error, jq runtime error, schema
351 mismatch on `prefix`) are logged but do not surface in the dashboard.
352 Watch the Netdata journal for warnings:
353 `journalctl --namespace netdata | grep network_sources`.
354
355
356 ### Prefer explicit authorization headers over URL credentials
357
358 URLs with embedded credentials (`https://user:pass@host`) are converted to
359 HTTP Basic authentication by the HTTP client. Prefer `headers:` for clarity
360 and to avoid storing credentials in URLs -- e.g.
361 `headers: { Authorization: "Basic dXNlcjpwYXNz" }` for HTTP basic-auth.
362
363
364