| 1 | # yamllint disable rule:line-length |
| 2 | --- |
| 3 | id: 'service-discovery-http' |
| 4 | meta: |
| 5 | kind: 'http' |
| 6 | name: 'HTTP endpoint' |
| 7 | tagline: 'Items returned by an HTTP/HTTPS endpoint (JSON or YAML).' |
| 8 | link: 'https://datatracker.ietf.org/doc/html/rfc9110' |
| 9 | icon_filename: 'http.svg' |
| 10 | keywords: |
| 11 | - 'service discovery' |
| 12 | - 'sd' |
| 13 | - 'http' |
| 14 | - 'rest' |
| 15 | - 'discovery' |
| 16 | - 'cmdb' |
| 17 | overview: |
| 18 | description: | |
| 19 | Netdata can pull a list of monitorable targets from any HTTP endpoint you control — a CMDB API, an internal asset registry, a static file served by nginx, or a Prometheus-style file_sd export. The discoverer fetches the endpoint, decodes JSON or YAML, and feeds each item into the `services:` rule engine. This is the "bring your own source-of-truth" discoverer. |
| 20 | |
| 21 | This page covers HTTP-specific setup. For the broader Service Discovery model and the shared template-helper reference, see [Service Discovery](/src/collectors/SERVICE-DISCOVERY.md). |
| 22 | how_it_works: | |
| 23 | Each discovery cycle, the discoverer: |
| 24 | |
| 25 | 1. **Fetches** the configured `url` over HTTP/HTTPS, honouring all standard go.d collector HTTP options (auth, headers, TLS, proxy, timeout). |
| 26 | 2. **Decodes** the response as either JSON or YAML according to `format` (auto / json / yaml). With `format: auto`, the decoder uses `Content-Type` if it is unambiguous, otherwise tries JSON first then YAML. |
| 27 | 3. **Accepts two shapes** at the top level: a bare array (`[ item, item, … ]`) **or** an envelope (`{ "items": [ … ] }`). Anything else is rejected. |
| 28 | 4. **Builds one target per array element**, exposing `.Item` (the decoded element — could be a string, a map, a number, …), `.TUID`, and `.Hash`. |
| 29 | 5. **Runs the `services:` rules** against each target. The default stock rule passes the item through unchanged via the `toYaml` helper, so an endpoint that already serves go.d job configurations works with zero rule authoring. |
| 30 | 6. **Reconciles** disappeared items — when a target is no longer in the response, the corresponding job stops on the next reconcile. |
| 31 | limitations: | |
| 32 | - Only **one URL per pipeline**. To pull from multiple sources, configure multiple HTTP discovery pipelines (each as its own UI entry, or split the file into one job per source). |
| 33 | - **Response size** is capped at 10 MiB. |
| 34 | - **One-shot mode** (`interval: 0`) fetches a single time when the pipeline starts. It does **not** refetch on SD reload — recreate the pipeline to refresh. |
| 35 | - **`bearer_token_file`** under `/var/run/secrets/` is treated as optional when Netdata is **not** running in Kubernetes (so the same config can be used in a Helm deployment without erroring out on dev hosts). |
| 36 | - The discoverer does not introspect the items it received — anything beyond what the upstream endpoint provides must be inferred via service rules. |
| 37 | setup: |
| 38 | prerequisites: |
| 39 | list: |
| 40 | - title: 'Endpoint that returns JSON or YAML' |
| 41 | description: | |
| 42 | Stand up an HTTP endpoint that returns either a top-level array (`[ "https://a/health", "https://b/health" ]`) or an envelope (`{ "items": [...] }`). Items can be primitives (strings, numbers), maps, or any nestable value the rule engine knows how to consume. |
| 43 | - title: 'Choose a pass-through vs. curated approach' |
| 44 | description: | |
| 45 | - **Pass-through**: have your endpoint emit ready-made go.d job configurations and use the stock rule, which renders each item directly via `toYaml`. Zero rule authoring on the Netdata side. |
| 46 | - **Curated**: have your endpoint emit raw data (URLs, hostnames, tags) and write `services:` rules that map the data to the right collector module. More work, more flexibility. |
| 47 | configuration: |
| 48 | file: |
| 49 | name: 'go.d/sd/http.conf' |
| 50 | options: |
| 51 | description: | |
| 52 | The configuration file has two top-level blocks: `discoverer:` (the options below) and `services:` (rules that turn fetched items into collector jobs — see [Service Rules](#service-rules)). |
| 53 | |
| 54 | After editing the file, restart the Netdata Agent to load the updated discovery pipeline. |
| 55 | folding: |
| 56 | title: 'Discoverer options' |
| 57 | enabled: false |
| 58 | list: |
| 59 | - name: 'url' |
| 60 | description: 'HTTP/HTTPS endpoint that returns the items.' |
| 61 | default_value: '' |
| 62 | required: true |
| 63 | detailed_description: | |
| 64 | Must be a fully-qualified `http://` or `https://` URL. The endpoint is expected to return either a bare array or an `{"items": [...]}` envelope (see [Service Rules](#service-rules) for the input model). |
| 65 | - name: 'interval' |
| 66 | description: 'How often to refetch the endpoint.' |
| 67 | default_value: '1m' |
| 68 | required: false |
| 69 | detailed_description: | |
| 70 | Set to `0` for one-shot mode — the endpoint is fetched once when the pipeline starts and never again. SD reload does not retrigger; recreate the pipeline to refresh. |
| 71 | - name: 'format' |
| 72 | description: 'Response format. One of `auto`, `json`, `yaml`.' |
| 73 | default_value: 'auto' |
| 74 | required: false |
| 75 | detailed_description: | |
| 76 | With `auto`, the decoder uses `Content-Type` when it is unambiguous (`application/json`, `application/yaml`, `*+json`, `*+yaml`), otherwise tries JSON first then YAML. |
| 77 | - name: 'timeout' |
| 78 | description: 'Per-request HTTP timeout.' |
| 79 | default_value: '2s' |
| 80 | required: false |
| 81 | - name: 'headers / username / password / bearer_token_file / proxy_url / tls_skip_verify / etc.' |
| 82 | description: 'All standard go.d HTTP options are accepted (basic auth, bearer tokens, custom headers, HTTP proxy, TLS options).' |
| 83 | default_value: '' |
| 84 | required: false |
| 85 | detailed_description: | |
| 86 | See any go.d HTTP-based collector (`httpcheck`, `prometheus`, `nginx`, …) for the full set. Notable: when `bearer_token_file` points under `/var/run/secrets/` and Netdata is **not** running inside Kubernetes, missing token files are silently ignored. |
| 87 | examples: |
| 88 | folding: |
| 89 | title: 'Configuration examples' |
| 90 | enabled: true |
| 91 | list: |
| 92 | - name: 'Pass-through go.d jobs (stock rule)' |
| 93 | description: | |
| 94 | The endpoint serves go.d job configurations directly. Each item must include a `module` field. The stock rule pipes the item through `toYaml` unchanged. |
| 95 | config: | |
| 96 | disabled: no |
| 97 | discoverer: |
| 98 | http: |
| 99 | url: https://cmdb.example.com/netdata/jobs.yaml |
| 100 | interval: 5m |
| 101 | format: auto |
| 102 | services: |
| 103 | - id: passthrough |
| 104 | match: '{{ true }}' |
| 105 | config_template: | |
| 106 | {{ .Item | toYaml }} |
| 107 | - name: 'Array of bare URLs → httpcheck' |
| 108 | description: | |
| 109 | The endpoint returns `[ "https://a/health", "https://b/health" ]`. Map each URL to an `httpcheck` job. |
| 110 | config: | |
| 111 | disabled: no |
| 112 | discoverer: |
| 113 | http: |
| 114 | url: https://cmdb.example.com/netdata/health-urls.json |
| 115 | interval: 1m |
| 116 | services: |
| 117 | - id: httpcheck |
| 118 | match: '{{ kindIs "string" .Item }}' |
| 119 | config_template: | |
| 120 | name: {{ .TUID }} |
| 121 | url: {{ .Item }} |
| 122 | - name: 'Array of objects with custom shape' |
| 123 | description: | |
| 124 | The endpoint returns `[ { "name": "api", "url": "https://api.example.com/health" }, … ]`. |
| 125 | config: | |
| 126 | disabled: no |
| 127 | discoverer: |
| 128 | http: |
| 129 | url: https://cmdb.example.com/netdata/services.json |
| 130 | services: |
| 131 | - id: httpcheck |
| 132 | match: '{{ and (kindIs "map" .Item) (hasKey .Item "url") }}' |
| 133 | config_template: | |
| 134 | name: {{ .Item.name }} |
| 135 | url: {{ .Item.url }} |
| 136 | - name: 'Bearer-token authentication' |
| 137 | description: 'Authenticate against the source-of-truth endpoint using a bearer token from a file.' |
| 138 | config: | |
| 139 | disabled: no |
| 140 | discoverer: |
| 141 | http: |
| 142 | url: https://cmdb.example.com/api/v1/netdata/jobs |
| 143 | bearer_token_file: /etc/netdata/secrets/cmdb-token |
| 144 | headers: |
| 145 | Accept: application/yaml |
| 146 | services: |
| 147 | - id: passthrough |
| 148 | match: '{{ true }}' |
| 149 | config_template: | |
| 150 | {{ .Item | toYaml }} |
| 151 | services: |
| 152 | description: | |
| 153 | A `services:` rule turns each fetched item into one or more collector jobs. The HTTP discoverer is unique among SD discoverers in that the target's data shape is **defined by the upstream endpoint**, not by this discoverer — `.Item` is whatever JSON/YAML element the endpoint returned. |
| 154 | |
| 155 | The shared rule model — function reference (`match`, sprig including `kindIs`/`hasKey`/`toYaml`), `config_template` rendering rules, and the `missingkey=error` failure semantics — lives on the [Service Discovery](/src/collectors/SERVICE-DISCOVERY.md) hub page. The notes below are HTTP-specific. |
| 156 | evaluation: |
| 157 | description: | |
| 158 | Quick reference — see [Rule evaluation semantics](/src/collectors/SERVICE-DISCOVERY.md#rule-evaluation-semantics) on the hub page for the full model. |
| 159 | list: |
| 160 | - name: 'Type-check `.Item` first' |
| 161 | description: | |
| 162 | Because `.Item` is whatever the endpoint serves, write defensive rules that check the type before reading sub-fields. `kindIs "string" .Item`, `kindIs "map" .Item`, and `hasKey .Item "<key>"` are the workhorses. A rule that does `{{ .Item.url }}` on a non-map item will fail at template-render time (`missingkey=error`) and the rule will be skipped. |
| 163 | - name: 'Pass-through requires `module:` in the upstream payload' |
| 164 | description: | |
| 165 | The pass-through rule (`{{ .Item | toYaml }}`) forwards the item unchanged to the collector subsystem. The collector subsystem requires every job to have `name:` and `module:`. If your endpoint omits `module:`, the resulting job has no module and the agent rejects it. Either include `module:` upstream or wrap with a curated rule that adds it. |
| 166 | - name: 'Module inference from rule id' |
| 167 | description: | |
| 168 | When you write a curated rule and the rendered job omits `module:`, the rule `id` is used as the module name. So `id: httpcheck` is enough to produce httpcheck jobs without writing `module: httpcheck` in every template. |
| 169 | template_variables: |
| 170 | description: 'Available inside both `match` expressions and `config_template` bodies for HTTP targets.' |
| 171 | list: |
| 172 | - name: '.Item' |
| 173 | type: 'any' |
| 174 | description: 'The decoded array element. Type depends on the upstream endpoint — could be a string, a number, a bool, a map, or a nested structure. Always type-check before reading sub-fields.' |
| 175 | - name: '.TUID' |
| 176 | type: 'string' |
| 177 | description: | |
| 178 | Stable per-target ID (`http_<endpoint-label>_<hash>`). Useful as a job `name:` when the upstream payload does not provide one. |
| 179 | - name: '.Hash' |
| 180 | type: 'uint64' |
| 181 | description: 'Hash of the item content. Used internally for change detection.' |
| 182 | examples: |
| 183 | description: 'Each example shows one or more entries from the `services:` array. Order matters — see [How rules are evaluated](#how-rules-are-evaluated).' |
| 184 | list: |
| 185 | - name: 'Pass-through (default stock rule)' |
| 186 | description: | |
| 187 | The endpoint already returns valid go.d job configurations. Forward each item unchanged via `toYaml`. Each item **must** include a `module` field (and `name`). |
| 188 | config: | |
| 189 | - id: passthrough |
| 190 | match: '{{ true }}' |
| 191 | config_template: | |
| 192 | {{ .Item | toYaml }} |
| 193 | - name: 'Array of strings → httpcheck (curated)' |
| 194 | description: | |
| 195 | Endpoint serves `[ "https://a/health", "https://b/health" ]`. Use `kindIs "string"` to gate the rule, then map each string to an `httpcheck` job. `id: httpcheck` makes the module infer automatically. |
| 196 | config: | |
| 197 | - id: httpcheck |
| 198 | match: '{{ kindIs "string" .Item }}' |
| 199 | config_template: | |
| 200 | name: {{ .TUID }} |
| 201 | url: {{ .Item }} |
| 202 | - name: 'Array of objects → httpcheck (curated)' |
| 203 | description: | |
| 204 | Endpoint serves `[ { "name": "api", "url": "https://api/health" }, … ]`. Type-check that the item is a map and has a `url` key, then map fields into the rendered job. |
| 205 | config: | |
| 206 | - id: httpcheck |
| 207 | match: '{{ and (kindIs "map" .Item) (hasKey .Item "url") }}' |
| 208 | config_template: | |
| 209 | name: {{ .Item.name }} |
| 210 | url: {{ .Item.url }} |
| 211 | - name: 'Multiple modules from one endpoint' |
| 212 | description: | |
| 213 | Your endpoint mixes shapes — some items target `httpcheck`, some target `prometheus`. Use `hasKey` to discriminate, with each rule producing its own module's jobs. |
| 214 | config: | |
| 215 | - id: prometheus |
| 216 | match: '{{ and (kindIs "map" .Item) (hasKey .Item "metrics_url") }}' |
| 217 | config_template: | |
| 218 | name: {{ .Item.name }} |
| 219 | url: {{ .Item.metrics_url }} |
| 220 | |
| 221 | - id: httpcheck |
| 222 | match: '{{ and (kindIs "map" .Item) (hasKey .Item "health_url") }}' |
| 223 | config_template: | |
| 224 | name: {{ .Item.name }} |
| 225 | url: {{ .Item.health_url }} |
| 226 | verify: |
| 227 | description: 'After enabling the discoverer, confirm the endpoint is reachable and items are being parsed.' |
| 228 | checks: |
| 229 | list: |
| 230 | - name: 'Confirm the endpoint is being fetched' |
| 231 | description: | |
| 232 | Watch the agent log for `discoverer=http` messages. With systemd: |
| 233 | |
| 234 | ```bash |
| 235 | journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep "discoverer=http" |
| 236 | ``` |
| 237 | |
| 238 | A successful fetch logs the number of items decoded. Failures (DNS, TLS, auth, parse) appear at warn level. |
| 239 | - name: 'Reproduce the fetch with curl' |
| 240 | description: | |
| 241 | When the discoverer log shows a parse error, hit the endpoint with `curl` to inspect what it returned: |
| 242 | |
| 243 | ```bash |
| 244 | curl -sS -H "Accept: application/yaml" https://cmdb.example.com/netdata/jobs.yaml | head -40 |
| 245 | ``` |
| 246 | |
| 247 | The response **must** be a top-level array or `{"items": [...]}` envelope. |
| 248 | - name: 'Confirm jobs are being created' |
| 249 | description: | |
| 250 | In the Netdata UI go to `Collectors -> go.d -> <module>`. Pass-through jobs use the `name` your endpoint provided; curated rules use whatever you set in the `config_template`. |
| 251 | troubleshooting: |
| 252 | problems: |
| 253 | list: |
| 254 | - name: 'parse response as json: ...; parse response as yaml: ...' |
| 255 | description: | |
| 256 | The response is neither valid JSON nor valid YAML. Common causes: the endpoint returned an HTML error page (check status code and `Content-Type`), the JSON has trailing garbage, or YAML indentation is wrong. Reproduce with `curl -i` to see the headers + body. |
| 257 | - name: 'Items decoded but no jobs created' |
| 258 | description: | |
| 259 | Your `services:` rules are not matching, or they match but the rendered template is empty. With pass-through (`{{ .Item | toYaml }}`), make sure each upstream item includes `module:` and `name:`. With curated rules, double-check the type checks (`kindIs`, `hasKey`). |
| 260 | - name: 'TLS/certificate errors against an internal endpoint' |
| 261 | description: | |
| 262 | Use `tls_skip_verify: yes` to bypass for testing, then mount the issuing CA and set `tls_ca: /path/to/ca.crt` for production. |
| 263 | - name: 'Bearer token file not found' |
| 264 | description: | |
| 265 | When Netdata runs **outside** Kubernetes and the configured `bearer_token_file` points under `/var/run/secrets/`, missing tokens are silently ignored — this is intentional so the same config works in dev and in Helm. If you are inside k8s, the file must exist. |