| 1 | # yamllint disable rule:line-length |
| 2 | --- |
| 3 | id: 'service-discovery-k8s' |
| 4 | meta: |
| 5 | kind: 'k8s' |
| 6 | name: 'Kubernetes' |
| 7 | tagline: 'Pods and services in a Kubernetes cluster.' |
| 8 | link: 'https://kubernetes.io/' |
| 9 | icon_filename: 'kubernetes.svg' |
| 10 | keywords: |
| 11 | - 'service discovery' |
| 12 | - 'sd' |
| 13 | - 'k8s' |
| 14 | - 'kubernetes' |
| 15 | - 'pods' |
| 16 | - 'services' |
| 17 | - 'discovery' |
| 18 | overview: |
| 19 | description: | |
| 20 | Netdata can automatically discover monitorable workloads inside a Kubernetes cluster — pods (with their containers and ports) or Services. The discoverer watches the Kubernetes API in real time, exposes per-pod-container or per-service-port targets to the rule engine, and lets you generate collector jobs from labels, annotations, container images, and ports. |
| 21 | |
| 22 | This page covers Kubernetes-specific setup. For the broader Service Discovery model and the shared template-helper reference, see [Service Discovery](/src/collectors/SERVICE-DISCOVERY.md). |
| 23 | how_it_works: | |
| 24 | Each Kubernetes discovery pipeline runs as either a **pod** discoverer or a **service** discoverer (selected by the `role` option). It then: |
| 25 | |
| 26 | 1. **Connects** to the Kubernetes API using the in-cluster service-account credentials (no `api_server` config — the discoverer uses the standard k8s client config-loader chain). |
| 27 | 2. **Watches** Pods (or Services) in the configured `namespaces[]`, optionally narrowed by label/field selectors. |
| 28 | 3. **Builds targets**: |
| 29 | - `role: pod` → one target per `(pod, container, container-port)` triple. Container env, image, labels, annotations, and node name are all exposed. |
| 30 | - `role: service` → one target per `(service, service-port)` pair, with the cluster-internal DNS name (`name.ns.svc:port`) as `.Address`. |
| 31 | 4. **Runs the `services:` rules** against each target, producing collector jobs. |
| 32 | 5. **Reconciles** in real time — pod/service add/update/delete events update the target set without polling. |
| 33 | limitations: | |
| 34 | - **Stock conf ships in the Helm chart, not this repo**: a stock `/etc/netdata/go.d/sd/k8s.conf` is not packaged with the agent. On Kubernetes deployments you should install Netdata via the [Helm chart](https://github.com/netdata/helmchart) — the chart renders both the discoverer config and a curated rule set tailored to your cluster's Netdata setup. |
| 35 | - **Outside Kubernetes**: this discoverer requires kube-API access (in-cluster service-account or kubeconfig). Running it on a workstation requires a kubeconfig and is not a typical use case. |
| 36 | - **Two roles per pipeline, never both**: `role` is a single-valued option. If you want both pod and service discovery, configure two pipelines. |
| 37 | - **`local_mode` for pods is opt-in**: by default the pod discoverer watches **all** pods in the configured namespaces. Set `pod.local_mode: true` to restrict to pods on the **same node** as the Netdata Agent (intended for the parent-on-every-node Helm topology). When `local_mode` is enabled, the env var `MY_NODE_NAME` must be set on the Netdata pod (the Helm chart sets this via the downward API). |
| 38 | - **TLS to the API server is mTLS via the in-cluster CA bundle** — there is no per-pipeline TLS configuration to override. |
| 39 | setup: |
| 40 | prerequisites: |
| 41 | list: |
| 42 | - title: 'Run on Kubernetes via the Netdata Helm chart' |
| 43 | description: | |
| 44 | The supported way to run the k8s discoverer is via the [Netdata Helm chart](https://github.com/netdata/helmchart). The chart provisions the right RBAC (`get/list/watch` on `pods`, `services`, `configmaps`, `secrets`), wires `MY_NODE_NAME` for `local_mode`, and ships a stock `services:` rule set tuned to its parent/child topology. |
| 45 | - title: 'RBAC permissions' |
| 46 | description: | |
| 47 | The discoverer needs the following verbs from its service account: |
| 48 | |
| 49 | - `pods`: `get`, `list`, `watch` (cluster-wide or per-namespace, matching `namespaces[]`) |
| 50 | - `services`: `get`, `list`, `watch` (only when `role: service`) |
| 51 | - `configmaps`, `secrets`: `get`, `list`, `watch` (only when `role: pod` — used to enrich pod targets with referenced env values) |
| 52 | |
| 53 | The Helm chart's default RBAC role covers all of these. |
| 54 | - title: 'For `pod.local_mode: true`, set MY_NODE_NAME' |
| 55 | description: | |
| 56 | When `local_mode` is enabled, the Netdata Agent reads its node name from `MY_NODE_NAME`. The Helm chart sets this via the downward API: |
| 57 | |
| 58 | ```yaml |
| 59 | env: |
| 60 | - name: MY_NODE_NAME |
| 61 | valueFrom: |
| 62 | fieldRef: |
| 63 | fieldPath: spec.nodeName |
| 64 | ``` |
| 65 | configuration: |
| 66 | file: |
| 67 | name: 'go.d/sd/k8s.conf' |
| 68 | options: |
| 69 | description: | |
| 70 | The configuration file has two top-level blocks: `discoverer:` (the options below) and `services:` (rules that turn discovered pods/services into collector jobs — see [Service Rules](#service-rules)). |
| 71 | |
| 72 | After editing the file, restart the Netdata Agent to load the updated discovery pipeline. The default and recommended deployment path on Kubernetes is the [Netdata Helm chart](https://github.com/netdata/helmchart) — the chart renders this file and the rules for you. |
| 73 | folding: |
| 74 | title: 'Discoverer options' |
| 75 | enabled: false |
| 76 | list: |
| 77 | - name: 'role' |
| 78 | description: 'What to discover. One of `pod` or `service`.' |
| 79 | default_value: '' |
| 80 | required: true |
| 81 | detailed_description: | |
| 82 | - `pod` — produces one target per `(pod, container, port)` triple. Use this for the bulk of in-cluster monitoring (databases, exporters, applications). |
| 83 | - `service` — produces one target per `(service, port)` pair. Use this for cluster-internal endpoints monitored at the service-name DNS level. |
| 84 | |
| 85 | To watch both, configure two pipelines. |
| 86 | - name: 'namespaces' |
| 87 | description: 'Namespaces to watch. Empty means all namespaces.' |
| 88 | default_value: '[] (all namespaces)' |
| 89 | required: false |
| 90 | - name: 'selector.label' |
| 91 | description: 'Label selector applied at watch time (server-side filtering).' |
| 92 | default_value: '' |
| 93 | required: false |
| 94 | detailed_description: | |
| 95 | Standard Kubernetes label-selector syntax: `app=foo`, `environment in (prod, staging)`, etc. Reduces watch traffic when only a subset of pods/services is interesting. |
| 96 | - name: 'selector.field' |
| 97 | description: 'Field selector applied at watch time.' |
| 98 | default_value: '' |
| 99 | required: false |
| 100 | detailed_description: | |
| 101 | Useful field selectors: `status.phase=Running`, `spec.nodeName=node-1`. When `pod.local_mode: true`, the discoverer automatically appends `spec.nodeName=$MY_NODE_NAME`. |
| 102 | - name: 'pod.local_mode' |
| 103 | description: 'Restrict pod discovery to pods on the same node as the Netdata Agent.' |
| 104 | default_value: 'false' |
| 105 | required: false |
| 106 | detailed_description: | |
| 107 | Only applies when `role: pod`. Requires `MY_NODE_NAME` to be set on the Netdata container. Used by the Helm chart's parent-on-every-node topology to keep watch traffic local. |
| 108 | examples: |
| 109 | folding: |
| 110 | title: 'Configuration examples' |
| 111 | enabled: true |
| 112 | list: |
| 113 | - name: 'Pod discovery, local mode (Helm-style)' |
| 114 | description: 'The configuration the Helm chart renders by default for the parent-on-every-node topology.' |
| 115 | config: | |
| 116 | disabled: no |
| 117 | discoverer: |
| 118 | k8s: |
| 119 | role: pod |
| 120 | pod: |
| 121 | local_mode: true |
| 122 | services: [ ] |
| 123 | - name: 'Service discovery in a specific namespace' |
| 124 | description: 'Watch only Services in the `monitoring` namespace, scoped by a label selector.' |
| 125 | config: | |
| 126 | disabled: no |
| 127 | discoverer: |
| 128 | k8s: |
| 129 | role: service |
| 130 | namespaces: |
| 131 | - monitoring |
| 132 | selector: |
| 133 | label: app.kubernetes.io/component=metrics-endpoint |
| 134 | services: [ ] |
| 135 | services: |
| 136 | description: | |
| 137 | A `services:` rule turns each discovered pod-container target (`role: pod`) or service-port target (`role: service`) into one or more collector jobs. The two target shapes have different fields — annotations and labels are common to both, but pod targets additionally expose container-level info (image, env, controller). |
| 138 | |
| 139 | The shared rule model — function reference (`match`, `glob`, `hasKey`, `index`, sprig), `config_template` rendering rules, and the `missingkey=error` failure semantics — lives on the [Service Discovery](/src/collectors/SERVICE-DISCOVERY.md) hub page. The notes below are k8s-specific. |
| 140 | evaluation: |
| 141 | description: | |
| 142 | Quick reference — see [Rule evaluation semantics](/src/collectors/SERVICE-DISCOVERY.md#rule-evaluation-semantics) on the hub page for the full model. |
| 143 | list: |
| 144 | - name: 'Different target shape per role' |
| 145 | description: | |
| 146 | `role: pod` and `role: service` produce different target structs. Rules in a pipeline must assume one shape — design your pipeline to match the discoverer's `role`. To handle both, run two pipelines. |
| 147 | - name: 'Annotation-driven matching is idiomatic' |
| 148 | description: | |
| 149 | Standard Kubernetes practice is to opt pods/services into monitoring via annotations (e.g. `prometheus.io/scrape: "true"`, `netdata.cloud/scrape: "true"`). Use `hasKey .Annotations "key"` and `index .Annotations "key"` to read them. |
| 150 | - name: 'Container ports vs. service ports' |
| 151 | description: | |
| 152 | Pod targets expose `.Port` / `.PortName` / `.PortProtocol` from the container's `ports[]`. Service targets expose them from the service's `ports[]`. Container ports may not be advertised through a Service — when you want both granularities, run two pipelines. |
| 153 | - name: 'Module inference from rule id' |
| 154 | description: | |
| 155 | For Kubernetes, set `id: <module-name>` so the rendered job inherits the module name automatically — same as the other discoverers. |
| 156 | template_variables: |
| 157 | description: 'Two distinct target shapes — `PodTarget` for `role: pod` and `ServiceTarget` for `role: service`.' |
| 158 | list: |
| 159 | - name: '.Address' |
| 160 | type: 'string' |
| 161 | description: 'For pods: `<pod-IP>:<port>` (or just `<pod-IP>` when no container port is exposed). For services: `<svc-name>.<namespace>.svc:<port>`.' |
| 162 | - name: '.Namespace' |
| 163 | type: 'string' |
| 164 | description: 'Pod/Service namespace.' |
| 165 | - name: '.Name' |
| 166 | type: 'string' |
| 167 | description: 'Pod or Service name.' |
| 168 | - name: '.Annotations' |
| 169 | type: 'map' |
| 170 | description: 'Pod/Service annotations. Read with `index .Annotations "key"`.' |
| 171 | - name: '.Labels' |
| 172 | type: 'map' |
| 173 | description: 'Pod/Service labels. Read with `index .Labels "key"`.' |
| 174 | - name: '.Port' |
| 175 | type: 'string' |
| 176 | description: 'Container port (pod target) or service port (service target).' |
| 177 | - name: '.PortName' |
| 178 | type: 'string' |
| 179 | description: 'Port name as declared in the spec (`http`, `metrics`, …).' |
| 180 | - name: '.PortProtocol' |
| 181 | type: 'string' |
| 182 | description: 'Port protocol (`TCP`, `UDP`).' |
| 183 | - name: '.PodIP' |
| 184 | type: 'string' |
| 185 | description: '**Pod targets only.** IP address of the pod.' |
| 186 | - name: '.NodeName' |
| 187 | type: 'string' |
| 188 | description: '**Pod targets only.** Name of the node hosting the pod.' |
| 189 | - name: '.ContName' |
| 190 | type: 'string' |
| 191 | description: '**Pod targets only.** Container name (within the pod).' |
| 192 | - name: '.Image' |
| 193 | type: 'string' |
| 194 | description: '**Pod targets only.** Container image.' |
| 195 | - name: '.Env' |
| 196 | type: 'map' |
| 197 | description: '**Pod targets only.** Container environment, with values from referenced ConfigMaps and Secrets resolved.' |
| 198 | - name: '.ControllerName' |
| 199 | type: 'string' |
| 200 | description: '**Pod targets only.** Owning controller name (e.g. ReplicaSet name).' |
| 201 | - name: '.ControllerKind' |
| 202 | type: 'string' |
| 203 | description: '**Pod targets only.** Owning controller kind (`ReplicaSet`, `StatefulSet`, `DaemonSet`, `Job`, …).' |
| 204 | - name: '.ClusterIP' |
| 205 | type: 'string' |
| 206 | description: '**Service targets only.** Cluster IP.' |
| 207 | - name: '.ExternalName' |
| 208 | type: 'string' |
| 209 | description: '**Service targets only.** External name (for `type: ExternalName` services).' |
| 210 | - name: '.Type' |
| 211 | type: 'string' |
| 212 | description: '**Service targets only.** Service type (`ClusterIP`, `NodePort`, `LoadBalancer`, `ExternalName`).' |
| 213 | examples: |
| 214 | description: 'Each example shows one entry from the `services:` array. Order matters — see [How rules are evaluated](#how-rules-are-evaluated).' |
| 215 | list: |
| 216 | - name: 'Pod with prometheus.io/scrape annotation' |
| 217 | description: | |
| 218 | The de-facto standard "scrape me" annotation. Match pods that opt in, route to the `prometheus` module. |
| 219 | config: | |
| 220 | - id: prometheus |
| 221 | match: '{{ and (hasKey .Annotations "prometheus.io/scrape") (eq (index .Annotations "prometheus.io/scrape") "true") }}' |
| 222 | config_template: | |
| 223 | name: {{ .Namespace }}_{{ .Name }}_{{ .ContName }} |
| 224 | url: http://{{ .Address }}{{ index .Annotations "prometheus.io/path" | default "/metrics" }} |
| 225 | - name: 'Service-role: monitor each metrics-endpoint Service' |
| 226 | description: | |
| 227 | Run with `role: service`. Match Services that carry a `metrics-endpoint` component label. |
| 228 | config: | |
| 229 | - id: prometheus |
| 230 | match: '{{ and (hasKey .Labels "app.kubernetes.io/component") (eq (index .Labels "app.kubernetes.io/component") "metrics-endpoint") }}' |
| 231 | config_template: | |
| 232 | name: {{ .Namespace }}_{{ .Name }} |
| 233 | url: http://{{ .Address }}/metrics |
| 234 | - name: 'Image-driven: nginx pods' |
| 235 | description: | |
| 236 | Match nginx-image pods on a known port. Use `match "sp"` for the four-form image family. |
| 237 | config: | |
| 238 | - id: nginx |
| 239 | match: '{{ and (eq .Port "80") (match "sp" .Image "nginx nginx:* */nginx */nginx:*") }}' |
| 240 | config_template: | |
| 241 | name: {{ .Namespace }}_{{ .Name }} |
| 242 | url: http://{{ .Address }}/stub_status |
| 243 | verify: |
| 244 | description: 'After enabling the discoverer, confirm it is watching the API and producing targets.' |
| 245 | checks: |
| 246 | list: |
| 247 | - name: 'Confirm the discoverer registered' |
| 248 | description: | |
| 249 | Watch the Netdata Agent log inside the pod for `discoverer=kubernetes` messages: |
| 250 | |
| 251 | ```bash |
| 252 | kubectl logs -n netdata <netdata-pod> | grep "discoverer=kubernetes" |
| 253 | ``` |
| 254 | |
| 255 | On startup you should see "instance is started", role information, and which namespaces are being watched. RBAC failures appear as `forbidden` errors from the watch. |
| 256 | - name: 'Confirm the API is reachable' |
| 257 | description: | |
| 258 | From the pod: |
| 259 | |
| 260 | ```bash |
| 261 | kubectl exec -n netdata <netdata-pod> -- curl -sSk \ |
| 262 | -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ |
| 263 | https://kubernetes.default.svc/api/v1/namespaces |
| 264 | ``` |
| 265 | |
| 266 | A 401 / 403 indicates the service account lacks the right RBAC. The Helm chart provisions the correct role. |
| 267 | - name: 'Confirm jobs are being created' |
| 268 | description: | |
| 269 | In the Netdata UI go to `Collectors -> go.d -> <module>`. Job names follow your `config_template` — the examples above use `<namespace>_<name>` patterns. |
| 270 | troubleshooting: |
| 271 | problems: |
| 272 | list: |
| 273 | - name: 'Permission denied (RBAC)' |
| 274 | description: | |
| 275 | The service account needs `get`, `list`, `watch` on `pods` (or `services`), and on `configmaps` + `secrets` for pod-role env enrichment. The Helm chart provisions this; out-of-Helm deployments must bind the equivalent role. |
| 276 | - name: '`local_mode` enabled but env "MY_NODE_NAME" not set' |
| 277 | description: | |
| 278 | When `pod.local_mode: true` is set but `MY_NODE_NAME` is missing, the discoverer fails at startup with `local_mode is enabled, but env 'MY_NODE_NAME' not set`. Set the env via the downward API on the Netdata pod (the Helm chart does this). |
| 279 | - name: 'No targets discovered' |
| 280 | description: | |
| 281 | - Confirm pods/services exist in the configured `namespaces[]`. |
| 282 | - If `selector.label` or `selector.field` is set, verify the targets actually carry the matching labels/fields. |
| 283 | - With `local_mode`, only pods on the same node as the Netdata pod are visible. |
| 284 | - name: 'Generated jobs fail to start' |
| 285 | description: | |
| 286 | The Address resolves to the pod's CNI IP — the Netdata Agent must be able to reach pod IPs. Most CNIs allow this from a pod running in the same cluster, but flat-network requirements differ. For service-role targets, the cluster-internal DNS name (`<svc>.<ns>.svc`) is used and should always resolve from inside the cluster. |