| 1 | # Service Discovery |
| 2 | |
| 3 | Stop hand-writing collector jobs for every host, container, or device. Netdata's Service Discovery (SD) finds monitorable targets in your environment and turns them into collector jobs automatically. |
| 4 | |
| 5 | Each SD pipeline is a `discoverer:` (where to look) plus a list of `services:` rules (how to turn what was found into collector jobs). The discoverer emits **targets**; the rules render **collector job YAML** from those targets using Go templates. |
| 6 | |
| 7 | ### Jump To |
| 8 | |
| 9 | [How it works](#how-it-works) • [Configuration file structure](#configuration-file-structure) • [Rule evaluation semantics](#rule-evaluation-semantics) • [Template helper reference](#template-helper-reference) • [config_template rendering](#config_template-rendering) • [Supported discoverers](#supported-discoverers) • [Mixing discoverers](#mixing-discoverers) • [Troubleshooting](#troubleshooting) |
| 10 | |
| 11 | |
| 12 | ## How it works |
| 13 | |
| 14 | Each discovery pipeline runs in five stages: |
| 15 | |
| 16 | 1. **Discover** — The discoverer probes its environment for monitorable things — running containers, listening sockets, kubernetes resources, SNMP devices on a subnet, items returned by an HTTP endpoint. What gets probed and how often is controlled by `discoverer:` options. |
| 17 | 2. **Build a target** — Each discovered thing becomes a target. The target carries discoverer-specific fields (variables) — for example, net_listeners targets have `.Port` / `.Comm` / `.Cmdline`; docker targets have `.Image` / `.Name` / `.Labels`; snmp targets have `.IPAddress` / `.SysInfo.*` / `.Credential.*`. Per-discoverer pages list the full variable set. |
| 18 | 3. **Match against services rules** — The rule engine evaluates each `services:` rule against each target, top to bottom. A rule's `match` expression is a Go template that must render to the literal string `"true"` for the rule to apply. |
| 19 | 4. **Render the collector job** — When a rule matches, its `config_template` is executed with the target as context, producing collector job YAML. The rendered YAML can be a single job map or a sequence of jobs. |
| 20 | 5. **Hand off to the collector** — The agent registers the rendered jobs with the matching collector module. From this point on, the collector runs the job — the discoverer is no longer involved. |
| 21 | |
| 22 | |
| 23 | ## Configuration file structure |
| 24 | |
| 25 | Each discoverer has its own file under `/etc/netdata/go.d/sd/`. The filename determines the discoverer kind (`net_listeners.conf`, `docker.conf`, `http.conf`, `snmp.conf`, `k8s.conf`). |
| 26 | |
| 27 | Every SD file has the same shape: |
| 28 | |
| 29 | ```yaml |
| 30 | disabled: no # set to yes to disable this pipeline |
| 31 | |
| 32 | discoverer: |
| 33 | <kind>: # discoverer kind (must match filename) |
| 34 | # discoverer-specific options — see the per-discoverer page |
| 35 | |
| 36 | services: |
| 37 | - id: <rule-id> |
| 38 | match: <go-template> # must render to the literal string "true" |
| 39 | config_template: | # optional — omit to make this a "skip rule" |
| 40 | # collector job YAML, rendered with target as context |
| 41 | ``` |
| 42 | |
| 43 | - `disabled: yes` keeps the file on disk but turns the pipeline off. |
| 44 | - Editing a stock file requires restarting the agent. UI-managed pipelines apply live. |
| 45 | - Where each discoverer's stock conf ships (with the Netdata package, with the Helm chart, or not at all) is documented on its per-discoverer page. |
| 46 | |
| 47 | |
| 48 | ## Rule evaluation semantics |
| 49 | |
| 50 | For each discovered target, the engine walks the `services:` array top-to-bottom. The two rule shapes behave differently: |
| 51 | |
| 52 | - **Skip rule (no `config_template`)** — When a skip rule matches, the target is dropped immediately — no jobs are produced and **no further rules run for that target**. Use skip rules to exclude targets the catch-all would otherwise pick up. Place them **before** any template rule. |
| 53 | - **Template rule (with `config_template`)** — When a template rule matches, its `config_template` is rendered into one or more collector jobs and rule evaluation **continues** with the next rule. A single target can therefore produce jobs from multiple matching template rules. |
| 54 | |
| 55 | ### Order matters |
| 56 | |
| 57 | Recommended ordering for a multi-rule pipeline: |
| 58 | |
| 59 | 1. Skip rules — drop targets you don't want monitored. |
| 60 | 2. Specific template rules — vendor-specific, label-specific, port-specific. |
| 61 | 3. A catch-all template rule (`match: '{{ true }}'`) — the default fallback. |
| 62 | |
| 63 | If a specific template rule already produced the right job, follow it with a skip rule keyed on the same condition to suppress the catch-all for those targets. |
| 64 | |
| 65 | |
| 66 | ## Template helper reference |
| 67 | |
| 68 | Match expressions and config templates are [Go `text/template`](https://pkg.go.dev/text/template) bodies with three additional helper sets: the standard Go template builtins, the [sprig](https://masterminds.github.io/sprig/) function library, and a small set of Netdata-specific helpers. |
| 69 | |
| 70 | All templates run with `missingkey=error`. Referencing a field that does not exist on the target type (e.g. typo `.SysInfoo.Name` instead of `.SysInfo.Name`) makes the template execution fail; the agent logs the error and skips that rule for that target. |
| 71 | |
| 72 | |
| 73 | ### Go template builtins |
| 74 | |
| 75 | Standard Go `text/template` syntax: pipelines, conditionals, loops, variable assignment, whitespace control. |
| 76 | |
| 77 | | Construct | Description | |
| 78 | |:----------|:------------| |
| 79 | | `if`/`else if`/`else`/`end` | Conditional rendering. | |
| 80 | | `range`/`end` | Iterate over a slice or map. | |
| 81 | | `with`/`end` | Set the dot to a value if it is non-empty. | |
| 82 | | `{{- ... -}}` | Whitespace-trim left/right around the action. | |
| 83 | | `{{ $var := ... }}` | Variable assignment, scoped to the enclosing block. | |
| 84 | | `eq`, `ne`, `lt`, `le`, `gt`, `ge` | Comparison. `eq A B C ...` is true if `A` equals **any** of the following arguments. | |
| 85 | | `and`, `or`, `not` | Boolean composition (variadic). | |
| 86 | | `index` | Index a slice or map: `index .Labels "app"`. | |
| 87 | | `printf` | Formatted string. | |
| 88 | |
| 89 | |
| 90 | ### Sprig functions |
| 91 | |
| 92 | The full [sprig library](https://masterminds.github.io/sprig/) is included. The functions most commonly used in stock SD configs: |
| 93 | |
| 94 | | Function | Description | |
| 95 | |:---------|:------------| |
| 96 | | `default DEFAULT VALUE` | Return `VALUE` if non-empty; otherwise `DEFAULT`. | |
| 97 | | `empty VALUE` | True if the value is the zero value for its type. | |
| 98 | | `hasKey MAP KEY` | True if `MAP` contains `KEY`. Used heavily by the `http` discoverer. | |
| 99 | | `kindIs KIND VALUE` | True if `VALUE`'s reflect kind matches: `string`, `map`, `slice`, `bool`, … | |
| 100 | | `lower S` / `upper S` | Lowercase / uppercase a string. | |
| 101 | | `trim S` / `trimPrefix PREFIX S` / `trimSuffix SUFFIX S` | Whitespace and prefix/suffix trimming. | |
| 102 | | `replace OLD NEW S` | String replace. | |
| 103 | | `regexFind RE S` | Return the first regex match, or empty. | |
| 104 | | `regexMatch RE S` | True if `S` matches `RE` (substring match unless anchored). | |
| 105 | | `printf FMT V...` | Same as Go's `fmt.Sprintf`. | |
| 106 | |
| 107 | See the [sprig docs](https://masterminds.github.io/sprig/) for the full list (string, math, encoding, list, dict, date helpers). |
| 108 | |
| 109 | |
| 110 | ### Netdata helpers |
| 111 | |
| 112 | Custom helpers added to the SD template engine: |
| 113 | |
| 114 | - `match TYPE VALUE PATTERN [PATTERN...]` — Returns the string `"true"` if `VALUE` matches **any** of the patterns under the named matcher type. `TYPE` is one of: |
| 115 | - `"glob"` — shell glob (`*`, `?`, `[abc]`). |
| 116 | - `"sp"` — Netdata simple patterns (space-separated, `*` wildcard, `!` for negation). |
| 117 | - `"re"` — RE2 regular expression. Matches if the regex matches anywhere in `VALUE` unless explicitly anchored with `^` / `$`. |
| 118 | - `"dstar"` — [doublestar](https://github.com/bmatcuk/doublestar) glob (supports `**` for path-style matching). |
| 119 | - `glob VALUE PATTERN [PATTERN...]` — Shortcut for `match "glob" VALUE PATTERN...`. |
| 120 | - `promPort PORT` — Returns the well-known Prometheus exporter module name registered for `PORT`, or the empty string. `net_listeners`-specific. |
| 121 | - `toYaml VALUE` — Serialize `VALUE` as a YAML string. Used by `http` discoverer rules that pass through items as collector job configs. |
| 122 | |
| 123 | **Notes:** |
| 124 | |
| 125 | - `match` and `glob` are case-sensitive. |
| 126 | - `match "re" ...` is unanchored. Use `^...$` to require a full-string match. |
| 127 | - There is **no** standalone `regexp` or `regex` helper. Use `match "re"` (or sprig's `regexFind` / `regexMatch`) for regex matching. |
| 128 | - `match` and `glob` return the **string** `"true"` or `"false"`, not a Go bool. The rule engine compares the trimmed template output against the literal `"true"`, so a top-level `match: '{{ glob .X "foo*" }}'` works directly. |
| 129 | - **Composing with `if` / `and` / `or` is a footgun**: in Go templates a non-empty string is truthy, so both `"true"` *and* `"false"` evaluate truthy under `if`. `{{ if glob .X "foo*" }}` is therefore **wrong**. Either wrap each result with `eq ... "true"` to get a real bool, or use an explicit `if-then-true` block: |
| 130 | ``` |
| 131 | match: '{{ if and (eq (glob .Vendor "Cisco*") "true") (eq .Category "router") }}true{{ end }}' |
| 132 | ``` |
| 133 | |
| 134 | |
| 135 | ## config_template rendering |
| 136 | |
| 137 | When a template rule matches, its `config_template` is executed with the target as the dot context. The rendered output is parsed as YAML to produce one or more collector jobs. |
| 138 | |
| 139 | - **Single map → one job** — If the rendered YAML is a map, a single collector job is created. |
| 140 | - **Sequence of maps → one job per element** — If the rendered YAML is a sequence (top-level YAML `-` items), one job is created per element. Use this for multi-job rules. Example: a `net_listeners` rule that emits both a TCP and a Unix-socket MySQL job from the same target — |
| 141 | |
| 142 | ```yaml |
| 143 | - id: mysql |
| 144 | match: '{{ or (eq .Port "3306") (eq .Comm "mysqld") }}' |
| 145 | config_template: | |
| 146 | - name: local |
| 147 | dsn: netdata@unix(/var/run/mysqld/mysqld.sock)/ |
| 148 | - name: local |
| 149 | dsn: netdata@tcp({{.Address}})/ |
| 150 | ``` |
| 151 | - **Module inference from rule `id`** — If the rendered job map has no `module:` key, the rule's `id` is used as the module name. Set `id: snmp` (or `docker`, `http`, …) to omit `module:` from your template; otherwise include `module:` explicitly. |
| 152 | - **`id` uniqueness** — Rule IDs are not required to be unique across rules — multiple rules can share an `id`. The `id` is used for module inference and shows up in agent logs to help you identify which rule produced a job. Pick descriptive IDs (`cisco`, `hp-printers`, `skip-vips`). |
| 153 | - **Failure handling** — Render errors, YAML parse errors, and `missingkey=error` failures are logged at warn level. The agent skips the rule for that target and continues evaluation. |
| 154 | |
| 155 | |
| 156 | ## Supported discoverers |
| 157 | |
| 158 | Each discoverer has its own page covering its options, target variables, evaluation specifics, and worked examples. |
| 159 | |
| 160 | | Discoverer | Kind | Stock conf | Discovers | |
| 161 | |:-----------|:-----|:-----------|:----------| |
| 162 | | [Docker](/src/go/plugin/go.d/discovery/sdext/discoverer/dockersd/README.md) | `docker` | `/etc/netdata/go.d/sd/docker.conf` | Running containers on the local Docker daemon. | |
| 163 | | [HTTP endpoint](/src/go/plugin/go.d/discovery/sdext/discoverer/httpsd/README.md) | `http` | `/etc/netdata/go.d/sd/http.conf` | Items returned by an HTTP/HTTPS endpoint (JSON or YAML). | |
| 164 | | [Kubernetes](/src/go/plugin/go.d/discovery/sdext/discoverer/k8ssd/README.md) | `k8s` | `/etc/netdata/go.d/sd/k8s.conf` | Pods and services in a Kubernetes cluster. | |
| 165 | | [Local listening processes](/src/go/plugin/go.d/discovery/sdext/discoverer/netlistensd/README.md) | `net_listeners` | `/etc/netdata/go.d/sd/net_listeners.conf` | Local processes that listen on TCP/UDP ports. | |
| 166 | | [SNMP](/src/go/plugin/go.d/discovery/sdext/discoverer/snmpsd/README.md) | `snmp` | `/etc/netdata/go.d/sd/snmp.conf` | SNMP-capable devices on configured network subnets. | |
| 167 | |
| 168 | |
| 169 | ## Mixing discoverers |
| 170 | |
| 171 | All discoverers can run simultaneously. Each `/etc/netdata/go.d/sd/<kind>.conf` is independent. The same target can theoretically be discovered by more than one discoverer (for example, a containerised application appears in both `docker` and `net_listeners`); each discoverer's pipeline is independent and may produce its own job. |
| 172 | |
| 173 | Use `disabled: yes` at the top of a stock file to keep it on disk but turn the pipeline off. |
| 174 | |
| 175 | UI-managed pipelines and file-based pipelines coexist. UI-managed pipelines apply live; file-based pipelines require an agent restart to reload. |
| 176 | |
| 177 | |
| 178 | ## Troubleshooting |
| 179 | |
| 180 | Common cross-discoverer problems. For discoverer-specific issues, see the per-discoverer page. |
| 181 | |
| 182 | ### No targets discovered |
| 183 | |
| 184 | Check the agent log for `discoverer=<kind>` lines. Confirm `disabled: no` and that the discoverer's prerequisites are met (network reachability, credentials, API access). |
| 185 | |
| 186 | ### Targets discovered but no jobs created |
| 187 | |
| 188 | Check that at least one `services:` rule has both a matching `match` expression and a `config_template`. A rule with no `config_template` is a skip rule — it drops the target instead of producing a job. |
| 189 | |
| 190 | ### `match` always evaluates false |
| 191 | |
| 192 | Match expressions must render to the literal string `"true"`. A bare `{{ if ... }}` block that does not output anything renders to the empty string, which is treated as false. Use `{{ true }}` for catch-all, `{{ glob .X "..." }}` for pattern matches, or `{{ if ... }}true{{ end }}` for ad-hoc conditions. |
| 193 | |
| 194 | ### Template render error |
| 195 | |
| 196 | Look for `failed to execute services[N]->config_template on target` in the log. The most common cause is a typo in a variable reference (e.g. `.SysInfoo.Name`) — `missingkey=error` rejects unknown fields. Use the per-discoverer page's variable table to verify spelling. |
| 197 | |
| 198 | ### YAML parse error after rendering |
| 199 | |
| 200 | `failed to parse services[N] template data` means the rendered output is not valid YAML. Common cause: a discovered string field contains a colon, hash, or other YAML special character. YAML-quote dynamic values (`name: "{{ .X }}"`) when they may be irregular. |