| 1 | --- |
| 2 | name: project-writing-collectors |
| 3 | description: Best practices and orientation for AI assistants authoring or modifying Netdata data-collection plugins or modules in any language. Read before adding a new collector, modifying an existing one, working on logs, topology, NetFlow/sFlow/IPFIX, OTEL ingestion, SNMP profiles, statsd, Prometheus scraping, or interactive Functions. Covers the mental model, framework-agnostic best practices, dashboard-shaping mechanisms (NIDL, SNMP profiles, statsd synthetic_charts, OTEL mappings, Prometheus exposition), production quality criteria, the plugin landscape, per-data-type patterns (metrics, logs, snapshots, topology, enrichment), per-domain common practices, and a pre-PR self-check. |
| 4 | type: project |
| 5 | --- |
| 6 | |
| 7 | # Writing Netdata data collection plugins and modules |
| 8 | |
| 9 | ## What this skill is |
| 10 | |
| 11 | You are about to add or modify data collection in the Netdata Agent. This skill is a manifesto and a routing map. It tells you the mindset to apply, the principles you cannot violate, the ways the dashboard gets shaped from upstream data, the quality bar that separates a draft from a shippable collector, and where to look for depth. It is not a tutorial — the deep references already exist in the repo. Your job is to know they exist, pick the right one, and produce work that blends with the patterns the maintainers already accept. |
| 12 | |
| 13 | The skill is organized as: AI fast path → mental model → best practices → dashboard shaping → quality bar → environment reference → applied per data type → applied per domain. For go.d work, follow the fast path first; for other collector families, read top to bottom on your first pass and come back to specific sections as the task narrows. |
| 14 | |
| 15 | ## AI Fast Path |
| 16 | |
| 17 | For implementation agents, route to the concrete workflow first and use the |
| 18 | rest of this skill as background: |
| 19 | |
| 20 | - New go.d collector: read `src/go/AGENTS.md`, then |
| 21 | `src/go/plugin/go.d/docs/how-to-write-a-collector.md`, |
| 22 | `.agents/skills/project-writing-go-modules-framework-v2/SKILL.md`, and |
| 23 | `.agents/skills/integrations-lifecycle/recipes/add-go-collector.md`. |
| 24 | - Existing go.d collector update: read `src/go/AGENTS.md`, the collector's |
| 25 | local files, `.agents/skills/integrations-lifecycle/consistency.md`, and |
| 26 | `.agents/skills/integrations-lifecycle/recipes/update-collector.md`. |
| 27 | - V1-to-V2 migration: read `src/go/AGENTS.md`, |
| 28 | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`, and the V2 skill before |
| 29 | changing code. |
| 30 | - Framework or shared-helper change: stop and satisfy |
| 31 | `src/go/plugin/framework/docs/changing-framework-code.md` before writing |
| 32 | code. |
| 33 | |
| 34 | Do not use this broad skill as the only implementation guide for go.d work. |
| 35 | |
| 36 | ## 1. Mental model |
| 37 | |
| 38 | How to think about Netdata data collection. Internalize this before designing anything. |
| 39 | |
| 40 | ### 1.1 Frequent collection at scale |
| 41 | |
| 42 | The Agent ships on >1.5M new daily installs across physical servers, VMs, containers, IoT devices, embedded systems, and exotic Unixes. Default collection is 1-second; many collectors raise it (`ping` 5s, SNMP 10s) when the source warrants it. Anything you do inside the collection cycle — allocate, log, reconnect, retry, parse, format — is multiplied by that population. Hot-path discipline is the entry ticket, not an optimization. |
| 43 | |
| 44 | ### 1.2 Metric structure is dashboard UX |
| 45 | |
| 46 | How dimensions group into charts and how labels attach to instances *is* the dashboard the user sees. Mirroring upstream data structures one-to-one produces a chart per metric, which is unusable. **NIDL** — Nodes, Instances, Dimensions, Labels — is the model. Every dashboard-shaping mechanism (§3) feeds into it. |
| 47 | |
| 48 | ### 1.3 IDs are public contracts |
| 49 | |
| 50 | Chart `context`, chart IDs, dimension IDs, instance labels — once shipped, they bind health alerts, dashboards, exports, anomaly detection, ML jobs, streaming consumers, and Netdata Cloud. Renaming silently breaks all of them. Treat them as permanent. |
| 51 | |
| 52 | ### 1.4 Gaps are data |
| 53 | |
| 54 | When you cannot measure a value this iteration, emit nothing for that dimension. The dashboard renders the gap; the user knows collection is broken. Defaulting to `0` fabricates a working state and hides the bug. Past pain in `src/collectors/proc.plugin/proc_net_dev.c` (search `shouldn't use 0 value, but NULL`). |
| 55 | |
| 56 | ### 1.5 Obsolete what's gone |
| 57 | |
| 58 | When the collector knows an entity has gone away — a process exited, a container was removed, a profile target was dropped, a network interface disappeared, a managed device went offline — mark its chart obsolete. The dashboard then renders it as historical, not as actively collected; alerts stop binding to it; streaming and ML stop costing for it. |
| 59 | |
| 60 | This is a truthfulness principle, not a cardinality one. It applies at any cardinality, including a single instance. Without obsoletion, the chart looks alive on the dashboard, alerts may continue evaluating against frozen data, and the user is misled about what is and isn't being collected. |
| 61 | |
| 62 | Mechanics: |
| 63 | - C: `rrdset_is_obsolete___safe_from_collector_thread()` in `src/database/rrdset.c:116` flags `RRDSET_FLAG_OBSOLETE`. Reverse with `rrdset_isnot_obsolete()` (line 140) when the entity reappears. |
| 64 | - go.d V1: `c.Obsolete = true` or `MarkRemove()` on the chart marks it obsolete. go.d V2: chart lifetime is controlled by `charts.yaml` lifecycle policy and `chartengine`; start from `src/go/plugin/go.d/docs/how-to-write-a-collector.md` for new collectors and `src/go/plugin/go.d/docs/migrate-v1-to-v2.md` for migrations. |
| 65 | - Anti-flip-flop: if an entity may disappear and reappear quickly, wait roughly 1 minute of absence before obsoleting. Thrashing charts hurt streaming and ML. |
| 66 | |
| 67 | ### 1.6 Your knowledge is stale — research the current spec |
| 68 | |
| 69 | Specs, vendor protocols, RFCs, and SDK behavior move. Before you design a collector or interpret a payload: |
| 70 | |
| 71 | - Read the **current** spec from the official source (RFC, vendor portal, SDK docs). |
| 72 | - For application/database/protocol collectors, read the **current** application's release notes — fields, defaults, and semantics shift between versions. |
| 73 | - Do not trust your prior-knowledge interpretation of a binary format, OID semantics, or HTTP/JSON shape. Verify against an authoritative document or live behavior. |
| 74 | |
| 75 | Prior-knowledge mistakes that recur: confused field names in NetFlow v5 vs v9 vs IPFIX, wrong endianness on a vendor MIB, outdated PostgreSQL `pg_stat_*` columns, deprecated Kubernetes API resources. |
| 76 | |
| 77 | ### 1.7 When the spec is ambiguous, look at how others solved it |
| 78 | |
| 79 | Specs leave many decisions implementation-defined. Vendor implementations bend specs in well-known ways. When you face an interpretation dilemma: |
| 80 | |
| 81 | - Read 2–3 popular open-source monitoring tools that already collect this data — Prometheus exporters, Zabbix templates, Datadog Agent integrations, ntopng (network protocols), librenms / OpenNMS / Akvorado (SNMP and flow), collectd (system data), pmacct / nfdump (flow protocols). |
| 82 | - Compare their parsers, field interpretation, and edge-case handling. |
| 83 | - Their code encodes real-world device quirks the spec doesn't document. |
| 84 | - Cross-check against the upstream protocol's reference implementation when one exists. |
| 85 | |
| 86 | This is how you avoid shipping a parser that fails on the first real device. If you have a local mirror of monitoring projects, use it; otherwise clone the relevant upstreams to `/tmp/` and read their source. |
| 87 | |
| 88 | ### 1.8 Mirror an existing Netdata collector |
| 89 | |
| 90 | The repo holds many go.d modules and internal C plugins. Maintainer patterns |
| 91 | live there, not in any prose doc. After you've reality-checked the upstream |
| 92 | protocol, pick the closest existing Netdata collector by domain and mirror its |
| 93 | structure. New go.d modules MUST use framework V2 and start from the current |
| 94 | V2 authoring guide — see §5.3. |
| 95 | |
| 96 | ### 1.9 Remote-monitored systems are vnodes |
| 97 | |
| 98 | When one collector talks to N targets (SNMP devices, remote DBs, cloud APIs, IPMI hosts, vCenter clusters), each target is a **vnode** so its metrics, alerts, and RBAC behave as if it were a separate node in Netdata Cloud. Every remote-target collector wires vnodes from the start. |
| 99 | |
| 100 | For Go v2 collectors that route one job's samples to multiple virtual nodes, use first-class `metrix.HostScope` rather than adding vnode identity as normal metric labels. Write per-resource metrics through scoped meters or vecs such as `meter.WithHostScope(scope)`, and leave metrics unscoped when they should follow the default job vnode or global host path. Scope keys must be stable for the virtual node identity; unbounded scope cardinality has the same operational cost profile as unbounded chart/cardinality growth. |
| 101 | |
| 102 | ### 1.10 Cardinality discipline |
| 103 | |
| 104 | - A chart with thousands of dimensions, or an instance list with thousands of entries, is unusable on the dashboard. The user cannot read it. |
| 105 | - A collector that emits potentially thousands of instances per monitored application is operationally wasteful — the data carries no insight. It pollutes streaming, ML, alerts, and queries for no benefit. |
| 106 | - A series is paid for across multiple subsystems: dbengine storage, agent memory, streaming bandwidth (per hop, including Netdata Cloud), ML training (one model per series), alert evaluation, dashboard render. None of these costs is large in isolation; together they justify ending up with what the user actually wants to see. |
| 107 | |
| 108 | Design for usefulness, not raw count. Bound cardinality (§2.5), and never ship "one chart per request / per PID / per ephemeral connection" without bounds. |
| 109 | |
| 110 | ### 1.11 Layered configuration |
| 111 | |
| 112 | Per-job source priority: `stock < discovered < user < dyncfg`, matched by job identity. A higher-priority source replaces a lower-priority job with the same identity; non-colliding jobs continue to load. IaC users configure via files in `/etc/netdata`; dashboard users configure via DYNCFG; both paths must work for the same collector. |
| 113 | |
| 114 | ## 2. Best practices |
| 115 | |
| 116 | Framework-agnostic, ordered by impact. |
| 117 | |
| 118 | ### 2.0 Mandatory clean end state and scope discipline |
| 119 | |
| 120 | You MUST aim for the clean end state, not the smallest diff. While |
| 121 | implementing, keep checking whether the design still looks like the structure |
| 122 | maintainers should want after the work is complete. |
| 123 | |
| 124 | At each coherent batch, you MUST check for scope drift. If the work exposes an |
| 125 | independent collector cleanup, framework change, docs correction, or migration, |
| 126 | either defer it explicitly or submit it as its own step before continuing. |
| 127 | |
| 128 | ### 2.1 Test against reality |
| 129 | |
| 130 | Source test data based on what you're collecting: |
| 131 | |
| 132 | - **Open-source / freely available applications** (MySQL, PostgreSQL, NGINX, Redis, MongoDB, RabbitMQ): run the actual application locally (Docker, native install). Validate against real output. Cover multiple versions when defaults diverge. |
| 133 | - **Closed-source / vendor / SaaS** (vendor switches, IBM workloads, cloud APIs, hypervisors): harvest fixtures from other open-source monitoring projects — Prometheus exporters, Zabbix templates, Datadog Agent integrations, vendor SDK samples, anonymized traces in vendor PRs/issues. Their fixtures are the most complete "real-world" dataset publicly available. |
| 134 | - **Hardware-dependent** (network gear, IPMI, PCIe sensors): capture pcaps from real devices when accessible; otherwise vendor SDK samples, public packet captures, fixtures from pmacct / nfdump / ntopng (for flow protocols). |
| 135 | - **Protocol parsing** (NetFlow / sFlow / IPFIX / OTEL / SNMP): vendor SDK samples, public dumps, fuzz-test corpora. NetFlow keeps fixtures under `src/crates/netflow-plugin/testdata/flows/` with sourcing recorded in `testdata/ATTRIBUTION.md` — do the same for any new fixtures with redistribution-sensitive provenance. |
| 136 | |
| 137 | Don't fabricate test data the parser passes by accident. Don't skip tests "because this protocol can't be tested locally" — that's exactly when fixtures matter most. Standard go.d test-function names: `Test_testDataIsValid`, `TestCollector_ConfigurationSerialize`, `TestCollector_Init`, `TestCollector_Check`, `TestCollector_Collect` — match the convention in adjacent collectors. Functions get a dedicated validator at `src/go/tools/functions-validation/` (E2E plus schema checks). |
| 138 | |
| 139 | For Go tests, prefer table-driven tests using `map[string]struct{}` keyed by |
| 140 | test-case name when cases share setup and assertion shape. Use separate test |
| 141 | functions only when setup or assertions are materially different. Prefer map |
| 142 | keys over a `name` field in `[]struct{}` so case names stay prominent and |
| 143 | order-independent. |
| 144 | |
| 145 | ### 2.2 Hot-path discipline |
| 146 | |
| 147 | `Collect()` runs every `update_every` seconds. It MUST: |
| 148 | |
| 149 | - Allocate buffers, maps, slices, parsed regexes, matchers, and metric |
| 150 | instruments once at `Init()` / `New()` and reuse them. Reset at the top of |
| 151 | `Collect()` only when needed; see `cato_networks/metrix.go` for the typed |
| 152 | V2 metric-instrument pattern. |
| 153 | - Hold persistent connections; reconnect only on failure with backoff. |
| 154 | - Cache anything stable between iterations: schema, capabilities, profile selections. |
| 155 | - Finish well under one cycle even on a slow target. |
| 156 | |
| 157 | Anti-pattern (search and avoid): `mx := make(map[string]int64)` per `Collect()` (e.g., `src/go/plugin/go.d/collector/ap/collect.go`). Don't allocate fresh structures per cycle. Don't reconnect every cycle. |
| 158 | |
| 159 | ### 2.3 Error handling |
| 160 | |
| 161 | Every error log answers three questions: **what operation, what target, what was expected vs observed**. Wrap errors with context (Go: `fmt.Errorf("...: %w", err)`); preserve the cause; check return codes from system calls and library functions. |
| 162 | |
| 163 | Don't return a bare `err` with no context. Don't log `"failed"`. Don't ignore syscall returns or library NULLs. |
| 164 | |
| 165 | ### 2.4 Logging discipline |
| 166 | |
| 167 | - `debug` inside the collection loop. |
| 168 | - `warn` or `error` once per known-recoverable condition, gated by an internal flag — never per cycle. |
| 169 | - `info` / `notice` for once-at-startup events. |
| 170 | - Reserve `error` severity for operator-actionable issues; transient conditions are `warn`. |
| 171 | |
| 172 | Past pain: an `ebpf.plugin` regression flooded logs because the collection loop logged every PID allocation. Per-cycle logs are forbidden. |
| 173 | |
| 174 | ### 2.5 Cardinality bounding |
| 175 | |
| 176 | When a collector emits one chart per discovered entity (process, connection, profile target, container, schema, queue, route), bound the count and let the operator scope it. (Obsoletion of entities the collector knows have gone is a separate concern — see §1.5.) |
| 177 | |
| 178 | **`max_*` is REQUIRED for entities that may grow without bounds.** Without a cap, a single misbehaving target (a runaway log rotator, a container churn loop, a vendor-specific deep table) can produce thousands of charts. |
| 179 | |
| 180 | **`max_*` MUST be coupled with selectors.** A cap alone silently truncates whatever happens to land in the first N entries — the operator has no say in *which* entities survive. A selector lets the operator pick what's actually important. Cap and selector together: cap protects the system, selector lets the operator drive. |
| 181 | |
| 182 | **Where to filter — depends on what the monitored application exposes:** |
| 183 | |
| 184 | - **Application exposes all instances with no upstream filter.** The collector caps at `max_*` and adds an aggregated **"Other"** chart that sums whatever was capped. Don't silently drop — totals must remain truthful even when individual instances are hidden. |
| 185 | - **Application supports upstream cherry-picking** (e.g. specifying which schemas / databases / queues to monitor at connection time). Push the operator's selector into the application call. Less wire data, less collector work, narrower blast radius if the operator narrows the scope. |
| 186 | - **Application provides aggregations or grouping keys** (totals, group-by-kind, group-by-type, group-by-class). Expose those aggregations as additional charts; let the operator choose which grouping keys to surface. Aggregations are bounded-cardinality views that survive any selector cut and are usually what dashboards actually want — per-instance detail is a drill-down case, not the default. |
| 187 | |
| 188 | **Anti-patterns:** |
| 189 | |
| 190 | - One chart per HTTP route × method × status code → N×M×K series per service. |
| 191 | - Histogram / percentile splits with high-cardinality labels (per-IP, per-tenant, per-trace) → multiplicative blow-up. |
| 192 | - Per-PID charts with no obsolete handler → growth at process churn rate (the bound is here in §2.5; the obsolete handler is the §1.5 concern). |
| 193 | |
| 194 | For go.d V2 collectors, keep selector/cap behavior in the collector design and |
| 195 | document the public config only when the operator has a real decision to make. |
| 196 | Start from `src/go/plugin/go.d/docs/how-to-write-a-collector.md`. |
| 197 | |
| 198 | ### 2.6 Configuration discipline |
| 199 | |
| 200 | Public tunables are part of the collector consistency contract. When a config |
| 201 | option is added, removed, renamed, or given a new default, you MUST follow |
| 202 | `.agents/skills/integrations-lifecycle/consistency.md`; you MUST NOT update |
| 203 | only the Go struct or only the docs. The stock `.conf` shows safe, |
| 204 | representative examples -- not necessarily every tunable. |
| 205 | |
| 206 | Collectors MUST NOT hardcode timeouts, paths, ports, or credentials. Stock |
| 207 | config and schema MUST NOT contradict each other. |
| 208 | |
| 209 | Credentials use the `${env:}/${file:}/${cmd:}/${store:}` indirection — see `src/collectors/SECRETS.md`. Privileged operations route through `src/collectors/utils/ndsudo.c`. |
| 210 | |
| 211 | ### 2.7 Generated artifacts are not source |
| 212 | |
| 213 | Several artifacts are produced from upstream definitions and MUST NOT be hand-edited: |
| 214 | |
| 215 | - `integrations/<name>.md` — generated from `metadata.yaml` (banner: `DO NOT EDIT THIS FILE DIRECTLY`). |
| 216 | - `ibm.d` modules — generated `README.md`, `metadata.yaml`, `config.go`, `zz_generated_*.go` from `contexts.yaml` via `go generate`. |
| 217 | - Rust plugin charts — derived at compile time via the `charts-derive` proc-macro. |
| 218 | |
| 219 | When a generated file looks wrong, fix the source of truth (`metadata.yaml`, `contexts.yaml`, derive macro input) and regenerate. Note: go.d uses `//go:embed` for static assets — there is no `go generate` step. |
| 220 | |
| 221 | ### 2.8 Documentation/configuration consistency |
| 222 | |
| 223 | Collector consistency has one detailed checklist: |
| 224 | `.agents/skills/integrations-lifecycle/consistency.md`. Treat code, |
| 225 | integration metadata, taxonomy, config, stock examples, alerts, and generated |
| 226 | documentation as one unit, but do not maintain a second artifact matrix here. |
| 227 | |
| 228 | If a collector exposes a Function, its response shape MUST also conform to the |
| 229 | relevant Function schema, such as `src/plugins.d/FUNCTION_UI_SCHEMA.json` or |
| 230 | `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`. |
| 231 | |
| 232 | ### 2.9 Cross-plugin enrichment via netipc |
| 233 | |
| 234 | When one collector needs data from another, use **netipc** — never shell out, open private sockets, poll log files, or reinvent IPC. In-tree libraries: |
| 235 | |
| 236 | - C: `src/libnetdata/netipc/` |
| 237 | - Go: `src/go/pkg/netipc/` |
| 238 | - Rust: `src/crates/netipc/` |
| 239 | |
| 240 | Both clients (consume) and servers (offer) exist in all three languages. Real example: `src/collectors/cgroups.plugin/cgroup-netipc.c` is a netipc server offering cgroup metadata to other plugins. Upstream spec, tests, fuzz suite: <https://github.com/netdata/plugin-ipc>. |
| 241 | |
| 242 | ### 2.10 Vnodes for remote targets |
| 243 | |
| 244 | Set `Vnode` in job config when the collector has one remote target. For Go V2 |
| 245 | collectors that emit multiple remote nodes from one job, use |
| 246 | `metrix.HostScope`; see `.agents/sow/specs/go-v2-host-scope.md` and |
| 247 | `src/go/plugin/go.d/docs/how-to-write-a-collector.md`. Past pain: an older |
| 248 | refactor had to retroactively split job-name validation per vnode/domain because |
| 249 | earlier collectors had not accounted for it. |
| 250 | |
| 251 | ## 3. Structuring dashboards |
| 252 | |
| 253 | The dashboard is built from charts. The way upstream data turns into charts depends on the ingestion path. Six mechanisms exist; pick the one that matches your collector and *learn how it shapes the result*. |
| 254 | |
| 255 | ### 3.1 NIDL framework — the model |
| 256 | |
| 257 | **N**odes, **I**nstances, **D**imensions, **L**abels. This is the conceptual model every other mechanism feeds into. Read `docs/NIDL-Framework.md` before designing metrics. Group dimensions into charts that answer *one operational question*. Use labels for instance and context annotations. Pick the right chart type (`line`, `area`, `stacked`, `heatmap` — see `src/database/rrdset-type.h`) and dimension algorithm (`absolute`, `incremental`, `percentage-of-incremental-row`, `percentage-of-absolute-row` — see `src/database/rrd-algorithm.h`, documented in `src/plugins.d/README.md`). |
| 258 | |
| 259 | Common bugs: `absolute` on a counter (counters are `incremental`); `line` when `stacked` is the right shape (CPU states, disk-time breakdown). Reuse shared metric definitions from `src/collectors/common-contexts/` for C plugins. |
| 260 | |
| 261 | ### 3.2 SNMP profiles — declarative spec → NIDL |
| 262 | |
| 263 | SNMP collection is profile-driven. A profile is a YAML document declaring OIDs, metric definitions, table indexing, units, chart families, and selectors. Stock profiles ship from `src/go/plugin/go.d/config/go.d/snmp.profiles/default/`; spec at `src/go/plugin/go.d/collector/snmp/profile-format.md` (~2000 lines). |
| 264 | |
| 265 | Adding or extending SNMP coverage means writing or extending a profile, not adding code. The SNMP topology collector (`snmp_topology`) builds on top of profiles — extending profiles is usually the right starting point for topology work too. |
| 266 | |
| 267 | Past pain: pre-profile SNMP code required per-vendor branches that became unmaintainable. Don't hardcode OID-to-metric mappings inside a custom collector or vendor branch. |
| 268 | |
| 269 | ### 3.3 statsd `synthetic_charts` — operator-curated dashboards |
| 270 | |
| 271 | The statsd plugin lets the operator group raw statsd metrics into curated charts via INI configs at `/etc/netdata/statsd.d/*.conf`. Each config defines: |
| 272 | |
| 273 | - `[app]` — match raw metrics by pattern, group them under an application name |
| 274 | - `[dictionary]` — rename raw metric names to display names |
| 275 | - chart sections — declare a chart with `title`, `family`, `context`, `units`, `type`, and explicit `dimension =` lines mapping source metrics to display dimensions |
| 276 | |
| 277 | Wildcard patterns extract dimension names from the matched portion: `dimension = pattern 'myapp.api.*.200' '' last 1 1` creates dimensions named after the wildcard match. Three-layer dimension lookup (dimension name in dictionary → metric name in dictionary → fallback to original). Stock examples: `src/collectors/statsd.plugin/k6.conf`, `src/collectors/statsd.plugin/asterisk.conf`. Full spec: `src/collectors/statsd.plugin/README.md` lines 397-639. |
| 278 | |
| 279 | This is the most operator-controllable shaping mechanism — the dashboard is whatever the operator declares. |
| 280 | |
| 281 | ### 3.4 OTEL mappings — per-metric YAML routing |
| 282 | |
| 283 | Netdata's OTEL plugin (`src/crates/netdata-otel/otel-plugin/`) accepts any OTLP gRPC metric. Mapping is **generic by default** — all resource attributes, scope attributes, and data point attributes become chart labels — but the operator controls routing via per-metric YAML files at `/etc/netdata/otel.d/v1/metrics/*.yaml`. Key knobs: |
| 284 | |
| 285 | - `instrumentation_scope.name` / `version` — regex match to scope an entry to a specific OTel instrumentation |
| 286 | - `dimension_attribute_key` — which data point attribute becomes the dimension name (default: `"value"`); other attributes become chart labels |
| 287 | - `interval_secs`, `grace_period_secs` — per-metric timing overrides |
| 288 | |
| 289 | Aggregation temporality drives the chart algorithm: Gauge → absolute, Sum delta → DeltaSum, Sum cumulative monotonic → CumulativeSum, Sum cumulative non-monotonic → treated as Gauge (`src/crates/netdata-otel/otel-plugin/src/chart.rs:84`). |
| 290 | |
| 291 | The plugin does **not** recognize OTel semantic conventions specifically (`host.name`, `service.name`, `deployment.environment`) — they pass through as labels. Cardinality control is `metrics.max_new_charts_per_request` in `otel.yaml`. Stock examples: `src/crates/netdata-otel/otel-plugin/configs/otel.d/v1/metrics/`. |
| 292 | |
| 293 | ### 3.5 Prometheus — deterministic; shape upstream to shape dashboard |
| 294 | |
| 295 | The generic Prometheus scraper (`src/go/plugin/go.d/collector/prometheus/`) auto-maps from the exposition format with no per-metric synthetic shaping: |
| 296 | |
| 297 | - metric name → chart ID + dimension ID |
| 298 | - Prometheus labels → Netdata chart labels (with optional `label_prefix`) |
| 299 | - type (`counter`, `gauge`, `histogram`, `summary`) → chart type and dimension algorithm |
| 300 | - histograms and summaries explode into 3 charts each (buckets/quantiles, `_sum`, `_count`) |
| 301 | - recognized suffixes: `_total` (counter), `_bucket` + `le` label (histogram), `_sum`, `_count`, `quantile` label (summary), `_info` (skipped) |
| 302 | - unit suffixes drive the units string: `_seconds`, `_bytes`, `_hertz` |
| 303 | |
| 304 | Operator controls are **scoping, not shaping**: time-series **selectors** (allow/deny on metric name and label values, `src/go/plugin/go.d/collector/prometheus/README.md:110-127`) and `fallback_type` glob patterns for untyped metrics. There is **no** equivalent of statsd `synthetic_charts` — you cannot group disparate Prometheus metrics into a composite chart Netdata-side. To shape the dashboard, shape the upstream exporter: rename metrics, add labels, fix types upstream. |
| 305 | |
| 306 | ### 3.6 Chart priorities |
| 307 | |
| 308 | Chart priorities (`priority` field in C, `Priority` in Go) drive UI ordering. C plugins follow conventions in `src/collectors/all.h`. Don't pick priorities arbitrarily; mirror an adjacent collector's range. |
| 309 | |
| 310 | ## 4. Production-quality criteria & pre-PR checklist |
| 311 | |
| 312 | A collector is *production-quality* when it satisfies all of: |
| 313 | |
| 314 | - **Survives target unavailability for hours** without log floods, fd leaks, memory growth, or runaway retries. |
| 315 | - **Bounded memory under failure** — buffers do not grow on parse errors or stuck connections. |
| 316 | - **No fd / goroutine / thread leaks** across `Cleanup()` cycles or job reloads. |
| 317 | - **Cycle-latency budget respected** — `Collect()` finishes well under one cycle even on a slow target. |
| 318 | - **Graceful with partial / malformed upstream responses** — parser does not crash, log-flood, or skip downstream collection. |
| 319 | - **High-cardinality entities bounded** via `max_*` and selectors so the operator can scope them. |
| 320 | - **Disappeared entities obsoleted** so the dashboard reflects what is actually being collected (this applies even at low cardinality). |
| 321 | - **IDs (chart context, chart ID, dimension ID, instance labels) are stable** — never renamed without a migration plan. |
| 322 | |
| 323 | ### Pre-PR checklist |
| 324 | |
| 325 | 1. Did I research the **current** spec/protocol/application from authoritative sources, not just from prior knowledge? |
| 326 | 2. For ambiguous specs: did I cross-check against 2–3 popular open-source monitoring projects? |
| 327 | 3. Do all metrics have units, chart families, and meaningful names? Did NIDL inform the grouping? Are chart types and dimension algorithms correct (`incremental` for counters, etc.)? |
| 328 | 4. Are gaps preserved (no zero defaults for missing values)? |
| 329 | 5. Does the collection cycle allocate, log per iteration, or reconnect every cycle? |
| 330 | 6. Do error logs answer *what operation, what target, what was expected vs observed*? |
| 331 | 7. Did I run the collector consistency checklist in `.agents/skills/integrations-lifecycle/consistency.md`, including the rule that generated integration pages are not hand-authored sources? |
| 332 | 8. For remote targets: is vnode wiring done? |
| 333 | 9. For SNMP: did I extend a profile rather than hardcode OIDs? |
| 334 | 10. For statsd / OTEL: did I document and ship the operator-side config (synthetic_charts file or OTEL mapping YAML)? |
| 335 | 11. For Prometheus scraping: are selectors correct? Are untyped metrics handled? |
| 336 | 12. For cross-plugin enrichment: am I using netipc? |
| 337 | 13. For Functions: does the response conform to one of the six shapes? Non-blocking with respect to the collection loop? Schema-validated? |
| 338 | 14. For ibm.d only: did I run `go generate` after touching `contexts.yaml`? |
| 339 | 15. For new go.d modules: are all four runtime-load wiring steps done (`collector/init.go` import, `go.d.conf`, stock conf, README)? |
| 340 | 16. Tests: real fixtures or real instances? Would they catch the bug I just fixed? |
| 341 | 17. High-cardinality labels / instances: bounded by `max_*` + selectors? Aggregated "Other" bucket or upstream-supplied aggregation present where applicable? |
| 342 | 18. Entities that can go away: obsoleted when the collector knows they're gone? Anti-flip-flop window applied where churn is expected? |
| 343 | 19. Production-quality criteria above — would this collector survive hours of target outage without leaks or log floods? |
| 344 | |
| 345 | ## 5. Plugins and frameworks — what's available and where |
| 346 | |
| 347 | Reference section. Use it after the mental model and best practices have framed your task. |
| 348 | |
| 349 | ### 5.1 The plugin landscape |
| 350 | |
| 351 | | Family | Lang | Platforms | Where in repo | Scope | |
| 352 | |---|---|---|---|---| |
| 353 | | `proc.plugin` | C | Linux | `src/collectors/proc.plugin/` | Kernel `/proc` and `/sys` | |
| 354 | | `apps.plugin` | C | Linux/FreeBSD/macOS/Windows | `src/collectors/apps.plugin/` | Per-process and per-user/group; `processes` Function | |
| 355 | | `cgroups.plugin` | C | Linux | `src/collectors/cgroups.plugin/` | Containers and control groups | |
| 356 | | `ebpf.plugin` | C + eBPF | Linux | `src/collectors/ebpf.plugin/` | Kernel function tracing | |
| 357 | | `network-viewer.plugin` | C | Linux | `src/collectors/network-viewer.plugin/` | L3/L4 sockets; `topology:` Functions | |
| 358 | | `systemd-journal.plugin` / `windows-events.plugin` | C | Linux/Windows | `src/collectors/{systemd-journal,windows-events}.plugin/` | Log/event explorers via Functions | |
| 359 | | `systemd-units.plugin` | C | Linux | `src/collectors/systemd-units.plugin/` | systemd unit state | |
| 360 | | `windows.plugin` | C | Windows | `src/collectors/windows.plugin/` | Windows performance counters | |
| 361 | | `freebsd.plugin` / `macos.plugin` | C | platform-specific | `src/collectors/{freebsd,macos}.plugin/` | OS analogs of `proc.plugin` | |
| 362 | | `statsd.plugin` | C | All | `src/collectors/statsd.plugin/` | StatsD ingestion + synthetic_charts | |
| 363 | | `log2journal` | C | Linux | `src/collectors/log2journal/` | Parse application logs into the systemd journal | |
| 364 | | Niche C plugins | C | various | `src/collectors/<name>.plugin/` | freeipmi, nfacct, tc, xenstat, debugfs, diskspace, slabinfo, idlejitter, timex, cups, ioping, perf | |
| 365 | | `go.d.plugin` | Go (no CGO) | All | `src/go/plugin/go.d/` | Application integrations | |
| 366 | | `ibm.d.plugin` | Go + CGO | Linux, IBM i | `src/go/plugin/ibm.d/modules/` | IBM workloads (DB2, IBM i / AS-400, IBM MQ, WebSphere) | |
| 367 | | `netflow-plugin` | Rust | Linux | `src/crates/netflow-plugin/` | NetFlow v5/v9, IPFIX, sFlow | |
| 368 | | `netdata-otel` | Rust | Linux | `src/crates/netdata-otel/otel-plugin/` | OpenTelemetry ingestion | |
| 369 | | `netdata-log-viewer` | Rust | Linux | `src/crates/netdata-log-viewer/` | OTEL signal viewer + journal Function backend | |
| 370 | | `charts.d.plugin` / `python.d.plugin` | Bash / Python | All | `src/collectors/{charts,python}.d.plugin/` | **Legacy** — do not add new modules | |
| 371 | |
| 372 | Path conventions: internal C plugins → `src/collectors/<name>.plugin/`; Go orchestrators → `src/go/plugin/{go.d,ibm.d}/`; Rust plugins → `src/crates/<name>/`. |
| 373 | |
| 374 | ### 5.2 Routing by task |
| 375 | |
| 376 | | If you are doing… | Start with | |
| 377 | |---|---| |
| 378 | | New off-the-shelf application integration (no CGO) | `src/go/plugin/go.d/docs/how-to-write-a-collector.md`; primary V2 reference: `src/go/plugin/go.d/collector/cato_networks/` | |
| 379 | | Migrating existing go.d collector to V2 | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`; V2 mechanics: `.agents/skills/project-writing-go-modules-framework-v2/SKILL.md` | |
| 380 | | New IBM workload integration (CGO) | `src/go/plugin/ibm.d/AGENTS.md`, `src/go/plugin/ibm.d/framework/README.md` | |
| 381 | | New Rust plugin | SDK at `src/crates/netdata-plugin/`; reference: `src/crates/netflow-plugin/` | |
| 382 | | New SNMP profile (no code change) | `src/go/plugin/go.d/collector/snmp/profile-format.md` | |
| 383 | | New interactive Function | `src/go/plugin/framework/functions/README.md`, `src/plugins.d/FUNCTION_UI_SCHEMA.json`, `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` | |
| 384 | | Topology work | `.agents/skills/project-create-topology/SKILL.md`, `src/go/pkg/topology/v1`, `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` | |
| 385 | | Auto-discovery for a new go.d module | rules under `src/go/plugin/go.d/config/go.d/sd/`; engine: `src/go/plugin/agent/discovery/` | |
| 386 | | OTEL ingestion | `src/crates/netdata-otel/otel-plugin/` | |
| 387 | | Log ingestion (parse → journal) | `src/collectors/log2journal/` and `log2journal.d/` rules | |
| 388 | | New external plugin in any language | `src/plugins.d/README.md` (PLUGINSD protocol) | |
| 389 | | New internal C plugin | `src/collectors/README.md`; mirror an adjacent collector | |
| 390 | | Cross-plugin data enrichment | netipc libraries (§5.4) | |
| 391 | | Privileged operations | `src/collectors/utils/ndsudo.c` | |
| 392 | | Credentials in config | `src/collectors/SECRETS.md` | |
| 393 | |
| 394 | ### 5.3 go.d V1 / V2 reality check |
| 395 | |
| 396 | Most go.d collectors are still V1, but the broad V1 authoring docs have been |
| 397 | retired because they taught stale patterns from general Go paths. Do not use |
| 398 | existing V1 collectors as the shape for new work. |
| 399 | |
| 400 | **New go.d modules MUST use V2.** Start with |
| 401 | `src/go/plugin/go.d/docs/how-to-write-a-collector.md`. Use |
| 402 | `src/go/plugin/go.d/collector/cato_networks/` as the primary modern reference, |
| 403 | but copy focused responsibilities rather than the entire collector. Copying a V1 |
| 404 | module mirrors legacy patterns and the maintainers will ask you to migrate. |
| 405 | |
| 406 | For migrating an existing V1 collector, start with |
| 407 | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`. Migration is compatibility work; |
| 408 | do not use the new-collector guide to justify chart, config, or lifecycle |
| 409 | contract changes. Temporary V1 parity bridges can help during development, but |
| 410 | the finished collector MUST NOT run through a V1-to-V2 bridge. |
| 411 | |
| 412 | V2 imports: `github.com/netdata/netdata/go/plugins/plugin/framework/collectorapi` and `.../pkg/metrix`. The `CollectorV2` interface lives at `src/go/plugin/framework/collectorapi/collector.go`. |
| 413 | |
| 414 | Lifecycle semantics: `Init()` is one-time setup (failure disables permanently); `Check()` is auto-detection probe (failure disables, retried later); `Collect()` is the hot path (every `update_every` seconds); `Cleanup()` is guaranteed on shutdown. |
| 415 | |
| 416 | **Silent-failure trap (go.d).** A new go.d module compiles and tests pass even when it is *not loaded* by the plugin at runtime. Runtime loading requires four wiring steps: import in `src/go/plugin/go.d/collector/init.go`, `modules:` toggle in `src/go/plugin/go.d/config/go.d.conf`, stock job config at `src/go/plugin/go.d/config/go.d/<name>.conf`, and entry in `src/go/plugin/go.d/README.md`. Same trap applies to `ibm.d`. |
| 417 | |
| 418 | ### 5.4 ibm.d, Rust SDK, internal C, PLUGINSD |
| 419 | |
| 420 | - **ibm.d** (CGO, IBM-vendor workloads) — use the ibm.d framework with `go generate` after touching `contexts.yaml`. See `src/go/plugin/ibm.d/AGENTS.md`. Don't reach for ibm.d for non-IBM CGO needs — the framework is shaped around vendor drivers; CGO outside the IBM ecosystem is a design discussion. |
| 421 | - **Rust SDK** at `src/crates/netdata-plugin/` — modules `bridge/`, `protocol/`, `rt/`, `charts-derive/`, `schema/`, `types/`, `error/`. Documentation lives in `lib.rs` doc-comments — there is no README. New Rust crates go into the `src/crates/Cargo.toml` workspace. Reference impl: `src/crates/netflow-plugin/`. |
| 422 | - **Internal C plugins** — mirror an adjacent collector under `src/collectors/<name>.plugin/`; reuse `src/libnetdata/`. `libnetdata.h` includes most of libnetdata so individual headers are usually unnecessary. Allocators with the `z` suffix (`mallocz`, `callocz`, `strdupz`, `freez`) handle failures via `fatal()`; `freez(NULL)` is safe. JSON parsing: json-c. JSON generation: `buffer_json_*`. Linked lists: `DOUBLE_LINKED_LIST_*` macros. |
| 423 | - **PLUGINSD external plugins (any language)** — spec at `src/plugins.d/README.md`. Useful when implementation language is dictated by an SDK that go.d / ibm.d / Rust cannot accommodate. |
| 424 | |
| 425 | **Don't:** |
| 426 | - write new go.d modules against V1 |
| 427 | - add modules to `charts.d.plugin` or `python.d.plugin` |
| 428 | - run `go generate` for go.d (no `//go:generate` directives — uses `//go:embed`) |
| 429 | - add new third-party Go modules or system-library dependencies casually — they ship to every Netdata install; check with maintainers if non-trivial |
| 430 | |
| 431 | ### 5.5 Build / dev loop |
| 432 | |
| 433 | - go.d unit tests: `cd src/go && go test ./plugin/go.d/collector/<name>/...` |
| 434 | - Single-module dev run: `timeout 15s go run ./cmd/godplugin -m <name> -d` |
| 435 | from `src/go`; success means the module registers, starts a job, and keeps |
| 436 | running until the timeout stops it. |
| 437 | - Rust: `cargo test -p <crate>` |
| 438 | - Whole-project install: `./netdata-installer.sh` |
| 439 | |
| 440 | ## 6. Dealing with data types |
| 441 | |
| 442 | A collector ingests one or more of these data types. Each has its own pattern. |
| 443 | |
| 444 | ### 6.1 Metrics (time-series numeric data) |
| 445 | |
| 446 | The default. Streams as `BEGIN/SET/END` (PLUGINSD) or framework equivalents. Shape via NIDL (§3). Storage is the dbengine; alerts bind to chart `context`; anomaly detection / ML jobs run continuously. Every metric travels via streaming to parents and to Netdata Cloud — cardinality matters everywhere. |
| 447 | |
| 448 | ### 6.2 Logs |
| 449 | |
| 450 | Two paths: |
| 451 | |
| 452 | - **Structured journaling.** `src/collectors/log2journal/` parses application/access logs (configurable YAML rules in `log2journal.d/`, e.g. `nginx-json.yaml`, `default.yaml`) and writes structured fields into the systemd journal. The `systemd-journal.plugin` then exposes the entries via a Function (the log explorer in the Netdata UI). |
| 453 | - **OTEL log signals.** `src/crates/netdata-log-viewer/` ingests OTEL logs and exposes them as Functions in the dashboard. |
| 454 | |
| 455 | Platform-specific events: `windows-events.plugin` (Windows event log). |
| 456 | |
| 457 | Logs are **not metrics**. Don't try to derive metrics from logs in the collection loop — emit logs as logs, then build metrics separately if needed. |
| 458 | |
| 459 | ### 6.3 Live snapshots (Functions) |
| 460 | |
| 461 | Interactive, on-demand tabular data: process lists, network connections, FDB tables, log entries, journal queries, topology snapshots, flow records. Functions complement metrics; they don't replace them. |
| 462 | |
| 463 | Build a Function when the answer is **interactive/tabular live data**. If the answer is a numeric time series, that's a metric. |
| 464 | |
| 465 | Response shape is one of `info_response`, `data_response`, `topology_response`, `flows_response`, `error_response`, `not_modified_response` (defined in `src/plugins.d/FUNCTION_UI_SCHEMA.json`). New topology payloads use the dedicated production topology contract in `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`. For Go, use builders in `src/go/pkg/funcapi/`; Go topology producers should use `src/go/pkg/topology/v1` for the v1 response model and compact-table helpers. For Rust, implement the `FunctionHandler` trait from the SDK runtime (`src/crates/netdata-plugin/rt/`). |
| 466 | |
| 467 | Functions run concurrently with the collection loop — they must not block it. Validate during development with `src/go/tools/functions-validation/`. |
| 468 | |
| 469 | Reference implementations: `src/collectors/network-viewer.plugin/` (topology + connections), `src/collectors/systemd-journal.plugin/` (log explorer), `src/collectors/apps.plugin/` (processes). |
| 470 | |
| 471 | Backend docs: `src/go/plugin/framework/functions/README.md` (Go), `src/crates/netdata-plugin/rt/src/lib.rs` (Rust `FunctionHandler`). UI/protocol: `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_UI_REFERENCE.md`. Topology contract: `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`. |
| 472 | |
| 473 | ### 6.4 Topology / interconnections / links |
| 474 | |
| 475 | Topology is its own data type — directed/undirected graphs of nodes and links. Sources and consumers: |
| 476 | |
| 477 | - **SNMP-discovered topology** (`src/go/plugin/go.d/collector/snmp_topology/`) — LLDP/CDP neighbors, BRIDGE-MIB FDB, Q-BRIDGE FDB, ARP tables, STP. Builds on SNMP profiles; extending profiles is usually the right starting point. |
| 478 | - **Live socket topology** (`src/collectors/network-viewer.plugin/`) — local L3/L4 sockets and their inferred connections. |
| 479 | - **Streaming graph** (`src/streaming/`) — Netdata parent/child topology. |
| 480 | - **Topology library** at `src/go/pkg/topology/v1` — production Go payload |
| 481 | helpers for new topology producers. The non-v1 root |
| 482 | `src/go/pkg/topology/` payload model has been retired and must not be |
| 483 | reintroduced for topology work. |
| 484 | |
| 485 | Topology is consumed via Functions (`topology:*` family), not via metrics. The cardinality of network edges is too high for time-series storage and the use case is interactive lookup. |
| 486 | |
| 487 | ### 6.5 Data enrichment via netipc |
| 488 | |
| 489 | When a collector needs data from another collector to enrich its output (a network collector wanting cgroup labels, an `apps` collector wanting cgroup PIDs, a flow collector wanting interface metadata), use **netipc**. Don't shell out, don't open private sockets, don't poll log files. |
| 490 | |
| 491 | Both client and server roles exist in C, Go, and Rust: |
| 492 | |
| 493 | - C: `src/libnetdata/netipc/` |
| 494 | - Go: `src/go/pkg/netipc/` |
| 495 | - Rust: `src/crates/netipc/` |
| 496 | |
| 497 | `cgroups.plugin` (`src/collectors/cgroups.plugin/cgroup-netipc.c`) is a real example of a netipc server offering cgroup metadata to other plugins. Upstream spec, tests, fuzz suite: <https://github.com/netdata/plugin-ipc>. |
| 498 | |
| 499 | ## 7. Common practices per collector domain |
| 500 | |
| 501 | These are descriptive patterns — what existing Netdata collectors do. Use them as defaults; deviate with reason. |
| 502 | |
| 503 | ### 7.1 Database collectors |
| 504 | |
| 505 | DB collectors often pair metrics (uptime, connections, query rates, replication lag, lock counts, cache hit ratios) with **Functions for live query analysis**: top queries, slow queries, currently-running queries, locks. Real examples: |
| 506 | |
| 507 | - **MySQL** (`src/go/plugin/go.d/collector/mysql/`) — metrics + `mysqlfunc/top_queries.go` + processlist via `collect_process_list.go`. |
| 508 | - **PostgreSQL** (`src/go/plugin/go.d/collector/postgres/`) — metrics + `func_top_queries.go` + `func_running_queries.go`, dispatched through `func_router.go`. |
| 509 | - MongoDB / Redis are metrics-only today, but the same Function pattern fits if the use case demands it. |
| 510 | |
| 511 | Before adding a query Function, decide whether it is in scope for the current |
| 512 | work and record the product/design decision. The operator value of seeing |
| 513 | "what's slow right now" is high and the pattern is established, but Functions |
| 514 | are still a feature surface, not something to add accidentally during unrelated |
| 515 | metric work. |
| 516 | |
| 517 | ### 7.2 Network and SNMP collectors |
| 518 | |
| 519 | Network/SNMP collectors typically pair metrics with **topology Functions** and FDB / ARP / LLDP enrichment: |
| 520 | |
| 521 | - **`snmp` + `snmp_topology`** (`src/go/plugin/go.d/collector/snmp_topology/`) — topology Functions (`func_topology.go`, `func_topology_handler.go`, `func_topology_managed_focus.go`, `func_topology_options.go`, `func_topology_presentation.go`, `func_topology_depth.go`) on top of SNMP profile data. |
| 522 | - **`network-viewer.plugin`** (`src/collectors/network-viewer.plugin/`) — `topology:` Functions for live socket-level topology. |
| 523 | |
| 524 | Per-device metrics need **vnode wiring** (each managed device is a vnode). FDB/ARP/STP data lands as topology Functions, not metrics — the cardinality is too high for metrics and the use case is interactive lookup. |
| 525 | |
| 526 | ### 7.3 Container / orchestration collectors |
| 527 | |
| 528 | Container collectors pair container metrics with **enrichment via netipc**: |
| 529 | |
| 530 | - `cgroups.plugin` exposes a netipc server (`src/collectors/cgroups.plugin/cgroup-netipc.c`) that other plugins query to map PIDs/cgroups to container/pod identity. |
| 531 | - `apps.plugin` and `network-viewer.plugin` consume this enrichment to label processes and connections with container metadata. |
| 532 | |
| 533 | When adding a new orchestration source (Kubernetes API, Docker events, Nomad, etc.), think about who downstream needs the labels and whether to expose them via netipc. |
| 534 | |
| 535 | ### 7.4 Web servers and reverse proxies |
| 536 | |
| 537 | Web server collectors pair metrics (requests, status codes, latency, upstream errors) with **access-log Functions** when the access log is structured: |
| 538 | |
| 539 | - `log2journal` parses NGINX/Apache/HAProxy access logs (rules under `src/collectors/log2journal/log2journal.d/`). |
| 540 | - The journal explorer Function makes the parsed entries searchable in the dashboard. |
| 541 | |
| 542 | If the application's log format is closed or unstructured, only metrics are practical. |
| 543 | |
| 544 | ### 7.5 Flow protocols (NetFlow / sFlow / IPFIX) |
| 545 | |
| 546 | The Rust `netflow-plugin` (`src/crates/netflow-plugin/`) ingests flows and exposes them via Functions (`flows_response` shape). Flows are per-record, high-cardinality, and not suitable for traditional metric storage. Reference fixtures and provenance discipline live under `src/crates/netflow-plugin/testdata/`. Topology enrichment (interface names, AS metadata) typically comes from netipc or from SNMP-collected interface data. |
| 547 | |
| 548 | ### 7.6 Application servers and middleware |
| 549 | |
| 550 | Java app servers, message queues, application middleware — JMX/HTTP/protobuf metrics are the default; some pair with log exploration via journal or OTEL log signals when the workflow benefits from it. Mirror the closest existing collector. |
| 551 | |
| 552 | ### 7.7 OS/kernel collectors |
| 553 | |
| 554 | Internal C plugins under `src/collectors/`. Reuse shared metric definitions from `src/collectors/common-contexts/`; follow chart-priority conventions in `src/collectors/all.h`; lean on `src/libnetdata/` rather than reimplementing utilities. |
| 555 | |
| 556 | ## 8. Canonical documentation pointers |
| 557 | |
| 558 | | Topic | Open when | Path | |
| 559 | |---|---|---| |
| 560 | | NIDL framework | designing metrics, labels, charts | `docs/NIDL-Framework.md` | |
| 561 | | Chart types and dimension algorithms | choosing chart shape and metric algorithm | `src/database/rrdset-type.h`, `src/database/rrd-algorithm.h` | |
| 562 | | Chart priorities (C) | dashboard ordering convention | `src/collectors/all.h` | |
| 563 | | Shared metric definitions (C) | reusing common contexts | `src/collectors/common-contexts/` | |
| 564 | | Plugin types and privileges | choosing where to add a collector | `src/collectors/README.md` | |
| 565 | | External plugin protocol | non-Go external plugin | `src/plugins.d/README.md` | |
| 566 | | go.d V2 authoring | adding a `go.d` module | `src/go/plugin/go.d/docs/how-to-write-a-collector.md` | |
| 567 | | go.d V1-to-V2 migration | migrating existing go.d collector | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md` | |
| 568 | | Functions backend (Go / Rust) | implementing a Function | `src/go/plugin/framework/functions/README.md`, `src/crates/netdata-plugin/rt/src/lib.rs` | |
| 569 | | Functions UI schema & guides | response shapes and patterns | `src/plugins.d/FUNCTION_UI_SCHEMA.json`, `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_UI_REFERENCE.md` | |
| 570 | | Topology Function schema & guide | topology actors, links, evidence, overlays | `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`, `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` | |
| 571 | | Functions validator | E2E + schema validation | `src/go/tools/functions-validation/README.md` | |
| 572 | | ibm.d framework | starting `ibm.d` work | `src/go/plugin/ibm.d/AGENTS.md`, `src/go/plugin/ibm.d/framework/README.md` | |
| 573 | | Rust plugin SDK | new Rust plugin | `src/crates/netdata-plugin/` (`rt/`, `protocol/`, `bridge/`, `charts-derive/`, `schema/`, `types/`, `error/`) | |
| 574 | | Rust NetFlow plugin | NetFlow / sFlow / IPFIX work | `src/crates/netflow-plugin/` | |
| 575 | | OTEL ingestion mappings | per-metric YAML routing | `src/crates/netdata-otel/otel-plugin/` (configs under `configs/otel.d/v1/metrics/`) | |
| 576 | | SNMP profile format | adding/extending an SNMP profile | `src/go/plugin/go.d/collector/snmp/profile-format.md` | |
| 577 | | SNMP stock profiles | starting from a known device | `src/go/plugin/go.d/config/go.d/snmp.profiles/default/` | |
| 578 | | statsd synthetic_charts | operator-curated dashboards | `src/collectors/statsd.plugin/README.md` (lines 397-639) | |
| 579 | | Prometheus mapping | generic exposition scrape | `src/go/plugin/go.d/collector/prometheus/README.md` | |
| 580 | | log2journal | parsing application logs into the journal | `src/collectors/log2journal/log2journal.d/` | |
| 581 | | Auto-discovery rules | adding service-detection rules | `src/go/plugin/go.d/config/go.d/sd/{net_listeners,docker,snmp,http}.conf` | |
| 582 | | Topology library | topology producers in Go | `src/go/pkg/topology/v1` | |
| 583 | | netipc cross-plugin enrichment | C / Go / Rust | `src/libnetdata/netipc/`, `src/go/pkg/netipc/`, `src/crates/netipc/` | |
| 584 | | DYNCFG protocol | dynamic configuration | `src/plugins.d/DYNCFG.md`, `docs/developer-and-contributor-corner/dyncfg.md` | |
| 585 | | Health alerts reference | alert template authoring | `src/health/REFERENCE.md`, `src/health/alert-configuration-ordering.md` | |
| 586 | | Integrations pipeline | doc generation from `metadata.yaml` | `integrations/README.md` | |
| 587 | | Go framework changes | changing shared Go collector/runtime framework code | `src/go/plugin/framework/docs/changing-framework-code.md` | |
| 588 | | go.d V1-to-V2 migration | migrating existing go.d collectors | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md` | |
| 589 | | Credentials in config | `${env:}/${file:}/${cmd:}/${store:}` | `src/collectors/SECRETS.md` | |
| 590 | | Privileged operations | restricted setuid helper | `src/collectors/utils/ndsudo.c` | |
| 591 | |
| 592 | ## 9. Maintaining this skill |
| 593 | |
| 594 | This skill is **live**. When you find a gap, an outdated pointer, a new pattern, or a bad practice not yet captured, propose changes to this file in the same PR that exposed the issue. When fixing a wrong pointer, also record what was misleading about the prior text — future readers see both the corrected map and the failure mode that produced it. Mention the change in the PR description so it gets reviewed consciously rather than skimmed. |