@cryptotaxi247 / netdata-1 / commits / ff0bc1756

docs: update Go area instructions (#22600)

Ilya Mashchenko committed Jun 2, 2026 at 12:43 UTC ff0bc17560a77d0a2927b04352f3b6c1bd5c6f51

23 files changed +2019 -1822

.agents/skills/integrations-lifecycle/artifacts-and-banners.md

+3 -3

@@ -7,8 +7,8 @@ banner conventions and edit rules.

9 |---|---|---|---|

@@ -23,7 +23,7 @@ banner conventions and edit rules.

28 ## Banner conventions per file kind

.agents/skills/integrations-lifecycle/consistency.md

+29 -5

@@ -22,6 +22,12 @@ the durable review checklist; a given PR may legitimately not

22 touch every file, but it must explain why an affected artifact

23 does not need a matching edit.

25 +Because most cross-artifact checks are not enforced by CI, collector PR

26 +descriptions MUST enumerate the relevant consistency artifacts and justify every

27 +artifact that did not need a matching change. Any SHOULD-level exception or

28 +escape hatch used by the implementation MUST be visible in the PR description

29 +or design note, not only in a code comment.

30 +

31 The rule covers obvious cases (units change in code -> update

32 metadata.yaml; new config option -> update schema, stock conf,

33 and docs; new metric -> update metadata.yaml, taxonomy.yaml,

@@ -52,10 +58,9 @@ stock conf example and the documented default value).

58 trigger touched-collector taxonomy coverage by themselves, although

59 the global taxonomy validation still runs.

60 - **`integrations/check_collector_metadata.py`** is broken

55 - (see `gotchas.md` and `validators.md` for details). Its

56 - imports refer to symbols that no longer exist in

57 - `gen_integrations.py`. ImportError on first run. NOT

58 - invoked from any workflow.

61 + (see `gotchas.md` for details). Its imports refer to symbols that no longer

62 + exist in `gen_integrations.py`. ImportError on first run. NOT invoked from any

63 + workflow.

64 - **No CI workflow** runs a "verify metric names in

65 metadata.yaml exist in the collector code" check.

66 - **No CI workflow** runs a "verify `health.d/*.conf` alert

@@ -127,6 +132,25 @@ When reviewing a PR that touches a collector, verify:

132 `SERVICE-DISCOVERY.md` (service-discovery changes -- but

133 note this one is NOT in CI; manual regen required).

134

135 +8. **Generated artifacts are outputs, not source.** Files with

136 + `DO NOT EDIT THIS FILE DIRECTLY` or `<!--startmeta ... message:

137 + "DO NOT EDIT..." -->` banners must be regenerated from their source

138 + artifacts. Do not hand-edit generated files to fix prose, links, setup text,

139 + or metric descriptions.

140 +

141 +9. **Gitignored generated catalogs are absent from the PR.** Before opening the

142 + PR, run:

143 +

144 + ```bash

145 + git status --porcelain |

146 + rg '^(\?\?|!!| M|M |A |AM) integrations/(integrations\.(js|json)|taxonomy\.json)$' || true

147 + ```

148 +

149 + The command MUST print no output. If it prints

150 + `integrations/integrations.js`, `integrations/integrations.json`, or

151 + `integrations/taxonomy.json`, remove the local generated artifact from the

152 + commit/worktree state rather than committing it.

153 +

154 ## Why the policy is unenforced

155

156 Investigation of the repo found that

@@ -152,7 +176,7 @@ For now, the consistency rule is a review-time policy.

176 ## Anti-patterns to flag in review

177

178 - "I only changed the code; the docs can be a follow-up PR."

155 - -> No. Five files, one PR.

179 + -> No. Affected collector consistency artifacts move in one PR.

180 - "The integration page on Learn doesn't show my new option."

181 -> Author forgot to update `metadata.yaml` AND regenerate

182 `integrations/<slug>.md`.

.agents/skills/integrations-lifecycle/gotchas.md

+1 -1

@@ -331,7 +331,7 @@ edit the Python script and commit.

331 regen:

332

333 ```bash

334 -python3 integrations/gen_docs_integrations.py -c go.d/snmp

334 +python3 integrations/gen_docs_integrations.py -c go.d.plugin/snmp

335 ```

336

337 NOT used by CI (CI always does full regen). Useful for fast

.agents/skills/integrations-lifecycle/pipeline.md

+5 -3

@@ -192,7 +192,7 @@ Two files are written (`gen_integrations.py:1311-1325`):

192 - `integrations/integrations.json` -- pure JSON with the

193 `clean` variant of `{categories, integrations}`. No banner.

194

195 -Both are gitignored (`.gitignore:159-160`). They are produced

195 +Both are gitignored (`.gitignore:162-163`). They are produced

196 fresh on every run; in CI, the workflow `rm`s them after the

197 downstream scripts read them so they are NOT included in the

198 auto-PR.

@@ -319,7 +319,7 @@ The script accepts `-c plugin/module` to scope cleanup and

319 regen to one collector (`:578-583`). Useful locally:

320

321 ```bash

322 -python3 integrations/gen_docs_integrations.py -c go.d/snmp

322 +python3 integrations/gen_docs_integrations.py -c go.d.plugin/snmp

323 ```

324

325 NOT used by CI; CI always runs without `-c` (full regen).

@@ -495,7 +495,9 @@ scripts directly during active development.

495 ./integrations/pip.sh

496 python3 integrations/gen_integrations.py

497 python3 integrations/gen_taxonomy.py --check-only

498 - python3 integrations/gen_docs_integrations.py -c go.d/foo

498 + python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD

499 + python3 -m unittest integrations.tests.test_taxonomy

500 + python3 integrations/gen_docs_integrations.py -c go.d.plugin/foo

501 python3 integrations/gen_doc_collector_page.py

502 python3 integrations/gen_doc_secrets_page.py

503 ```

.agents/skills/integrations-lifecycle/recipes/INDEX.md

+1 -1

@@ -31,7 +31,7 @@ scoping on `gen_docs_integrations.py` to skip cleaning/

31 regenerating other directories:

33 ```bash

34 -python3 integrations/gen_docs_integrations.py -c go.d/<your-module>

34 +python3 integrations/gen_docs_integrations.py -c go.d.plugin/<your-module>

35 ```

37 ## When in doubt

.agents/skills/integrations-lifecycle/recipes/add-go-collector.md

+65 -28

@@ -9,23 +9,32 @@ called `<name>`. For modifying an existing collector, see

9 - `<repo>/.agents/skills/project-writing-collectors/SKILL.md`

10 -- the broader "how to write a collector" context (NIDL

11 contexts, dashboard shaping, plugin landscape).

12 +- `<repo>/src/go/plugin/go.d/docs/how-to-write-a-collector.md`

13 + -- the canonical framework V2 code/layout guide for new go.d collectors.

14 - `../SKILL.md` -- this skill's overview.

15 - `../schema-reference.md` -- the `collector.json` schema

16 fields you will be filling in.

18 ## 1. Create the module skeleton

18 -Standard go.d layout:

20 +New go.d collectors MUST use framework V2. Code layout details live in

21 +`src/go/plugin/go.d/docs/how-to-write-a-collector.md`; this recipe covers the

22 +integration artifact side. A normal V2 collector directory includes:

24 ```

25 src/go/plugin/go.d/collector/<name>/

22 -├── <name>.go # Module entrypoint, Init/Check/Collect

23 -├── config.go # Config struct

24 -├── config_schema.json # DYNCFG schema

25 -├── metadata.yaml # Integration metadata (this skill's territory)

26 -├── README.md # Will become a symlink to integrations/<slug>.md once gen runs

27 -├── testdata/ # Fixtures

28 -└── ...other .go files

26 +|-- collector.go # Register/CreateV2/New/public lifecycle

27 +|-- config.go # Config struct

28 +|-- collect.go # Collect orchestration

29 +|-- metrix.go # Typed metrix instruments

30 +|-- write_metrics.go # Metric writes

31 +|-- charts.yaml # V2 chart template

32 +|-- config_schema.json # DYNCFG schema

33 +|-- metadata.yaml # Integration metadata (this skill's territory)

34 +|-- taxonomy.yaml # Dashboard TOC placement

35 +|-- README.md # Will become a symlink to integrations/<slug>.md once gen runs

36 +|-- testdata/ # Fixtures

37 +`-- ...other .go files

38 ```

40 Plus stock conf:

@@ -133,13 +142,13 @@ modules:

142

143 The first sentence of `metrics_description` is also used as the

144 description in generated catalog-style pages such as

136 -`src/collectors/COLLECTORS.md`. Keep it product-facing and stable:

137 -start with an action phrase, describe the integration, and do not

145 +`src/collectors/COLLECTORS.md`. It SHOULD stay product-facing and stable:

146 +start with an action phrase, describe the integration, and MUST NOT

147 describe configuration variables, defaults, limits, or setup steps.

148 Put those details in the setup, default-behavior, examples, or

149 troubleshooting fields.

150

142 -Hit every required field. The validator is strict (fatal on

151 +Hit every REQUIRED field. The validator is strict (fatal on

152 warnings). Refer to `../schema-reference.md` for the

153 exhaustive field list.

154

@@ -153,7 +162,8 @@ or add a new one under the appropriate parent (typically

162

163 ## 4. Taxonomy, stock `.conf`, `config_schema.json`, alerts, README

164

156 -These files are the rest of the collector consistency rule:

165 +These files are the rest of the collector consistency rule and MUST stay

166 +synchronized with the collector code:

167

168 - `src/go/plugin/go.d/collector/<name>/taxonomy.yaml` --

169 dashboard TOC placement for chart contexts. Static collectors

@@ -162,7 +172,7 @@ These files are the rest of the collector consistency rule:

172 `context_prefix:` or `collect_plugin:` and matching

173 `metadata.yaml.metrics.dynamic_*` declarations. Display widgets

174 use `type: context` with `contexts:` and `chart_library`; those

165 - referenced contexts must also be owned by structural items.

175 + referenced contexts MUST also be owned by structural items.

176 Pick `--section-id` from

177 `integrations/taxonomy/sections.yaml`; `section_id` is a stable

178 registry ID, not a path to invent in the collector file.

@@ -170,23 +180,23 @@ These files are the rest of the collector consistency rule:

180 ```bash

181 python3 integrations/gen_taxonomy_seed.py src/go/plugin/go.d/collector/<name>/metadata.yaml --module-name <name> --section-id <section.id> --placement-id <name> --icon <icon>

182 ```

173 - For a rich example with summary grids, table widgets, nested

174 - groups, and ownership leaves, read

175 - `src/go/plugin/go.d/collector/mysql/taxonomy.yaml`.

183 + For a rich recent example with groups, context ownership, and

184 + generated integration docs, read

185 + `src/go/plugin/go.d/collector/cato_networks/taxonomy.yaml`.

186 - `src/go/plugin/go.d/config/go.d/<name>.conf` -- the stock

187 config users will see at

178 - `/etc/netdata/go.d/<name>.conf`. Keep it minimal but

188 + `/etc/netdata/go.d/<name>.conf`. It SHOULD stay minimal but

189 representative. Show every common option with a comment.

190 - `src/go/plugin/go.d/collector/<name>/config_schema.json` --

181 - the DYNCFG schema. Each option in the stock `.conf` should

191 + the DYNCFG schema. Each option in the stock `.conf` SHOULD

192 have a corresponding entry here, with the same default.

193 - `src/health/health.d/<name>.conf` -- alerts on the metrics

184 - declared in `metadata.yaml`. Each alert in this file should

194 + declared in `metadata.yaml`. Each alert in this file SHOULD

195 have a matching entry under `metadata.yaml.modules[0].alerts[]`.

196 - `src/go/plugin/go.d/collector/<name>/README.md` -- this is the

197 USER-FACING documentation. After step 5, this file will be

198 REPLACED with a symlink to

189 - `integrations/<slug>.md`. So you do NOT hand-write the

199 + `integrations/<slug>.md`. You MUST NOT hand-write the

200 README; the generator does. Stub it as empty initially.

201

202 ## 5. Run the pipeline locally

@@ -197,20 +207,27 @@ From the repo root:

207 ./integrations/pip.sh # once

208 python3 integrations/gen_integrations.py

209 python3 integrations/gen_taxonomy.py --check-only

200 -python3 integrations/gen_docs_integrations.py -c go.d/<name>

210 +python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD

211 +python3 -m unittest integrations.tests.test_taxonomy

212 +python3 integrations/gen_docs_integrations.py -c go.d.plugin/<name>

213 python3 integrations/gen_doc_collector_page.py

214 python3 integrations/gen_doc_secrets_page.py

215 +# If service-discovery rules or sdext metadata changed:

216 +python3 integrations/gen_doc_service_discovery_page.py

217 ```

218

219 Expected outputs:

220

221 - `integrations/integrations.js` and `integrations/integrations.json`

208 - regenerated (gitignored, do NOT commit them).

222 + regenerated (gitignored, MUST NOT be committed).

223 - Collector taxonomy validated. If `gen_taxonomy.py` fails, fix

224 `taxonomy.yaml` or the matching `metadata.yaml.metrics.dynamic_*`

225 declaration before continuing.

226 +- Touched-collector taxonomy coverage validated in the same `--pr-diff` mode

227 + used by CI. If your local base branch is not `master`, adjust the diff range

228 + to the PR base.

229 - `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md`

213 - CREATED. Inspect: it should contain the `<!--startmeta`

230 + CREATED. Inspect: it SHOULD contain the `<!--startmeta`

231 banner with your `sidebar_label` and `learn_rel_path`, then

232 the rendered overview / setup / metrics / alerts /

233 troubleshooting sections.

@@ -219,6 +236,8 @@ Expected outputs:

236 exactly one integration in this directory).

237 - `src/collectors/COLLECTORS.md` updated to include your new

238 collector in its category section.

239 +- If service-discovery rules or `sdext` metadata changed,

240 + `src/collectors/SERVICE-DISCOVERY.md` updated.

241

242 If `gen_integrations.py` exits non-zero, read the warning

243 output -- a schema validation failed. Fix `metadata.yaml` and

@@ -230,22 +249,40 @@ re-run.

249 every section reads correctly.

250 - Open `src/collectors/COLLECTORS.md` and find your collector

251 in the table.

233 -- Run `python3 integrations/check_collector_taxonomy.py` before

234 - opening the PR. In CI this also runs with `--pr-diff` to enforce

235 - touched-collector taxonomy coverage.

252 +- Run `python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD`

253 + before opening the PR. If your local base branch is not `master`, adjust the

254 + diff range to the PR base.

255 +- From `src/go`, run `timeout 15s go run ./cmd/godplugin -m <name> -d` to

256 + confirm go.d can load the module after the `init.go`, `go.d.conf`, stock

257 + config, and README wiring changes. Success means the module is registered, a

258 + job starts, and the command keeps running until timeout stops it; `unknown

259 + module`, `no jobs started`, config-load errors, or immediate exit are

260 + failures. Use `-c <config-dir>` when testing a non-standard config path.

261 +- If service-discovery rules or `sdext` metadata changed, run

262 + `python3 integrations/gen_doc_service_discovery_page.py` and commit

263 + `src/collectors/SERVICE-DISCOVERY.md`.

264 +- Run

265 + `git status --porcelain | rg '^(\?\?|!!| M|M |A |AM) integrations/(integrations\.(js|json)|taxonomy\.json)$' || true`

266 + and make sure it prints no output; these generated runtime catalogs are

267 + gitignored and MUST NOT be committed.

268 - Run `git diff` and confirm the only changes are in:

269 - `src/go/plugin/go.d/collector/<name>/...` (your new module

270 files).

271 + - `src/go/plugin/go.d/collector/init.go` (registration import).

272 + - `src/go/plugin/go.d/config/go.d.conf` (module toggle).

273 + - `src/go/plugin/go.d/config/go.d/<name>.conf` (stock config).

274 + - `src/go/plugin/go.d/README.md` (collector list).

275 - `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md`

276 (the generated integration page).

277 - `src/go/plugin/go.d/collector/<name>/README.md` (now a

278 symlink).

279 - `src/collectors/COLLECTORS.md` (umbrella page updated).

280 - `src/health/health.d/<name>.conf` (alerts file).

281 + - `src/collectors/SERVICE-DISCOVERY.md` if service-discovery changed.

282 - Possibly `integrations/categories.yaml` if you added a

283 category.

284 - NOT `integrations/integrations.js` or

248 - `integrations.json` (gitignored).

285 + `integrations/integrations.json` (gitignored).

286 - NOT `integrations/taxonomy.json` (gitignored).

287

288 ## 7. Commit and push

@@ -262,7 +299,7 @@ artifacts were updated together.

299 committed integration page diverges from CI's regen, the

300 workflow fails -- fix locally and re-push.

301 - After merge, `generate-integrations.yml` triggers on master.

265 - Since you already committed the regen, this should not

302 + Since you already committed the regen, this SHOULD NOT

303 produce changes. If it does, the auto-PR

304 (`Regenerate integrations docs`) catches the drift -- merge

305 it.

.agents/skills/integrations-lifecycle/recipes/update-collector.md

+20 -3

@@ -63,7 +63,9 @@ Rules of thumb:

63 `context_prefix:` or `collect_plugin:`;

64 - pick section IDs from `integrations/taxonomy/sections.yaml`.

66 -For a rich reference, compare against

66 +For a modern go.d V2 ownership reference, compare against

67 +`src/go/plugin/go.d/collector/cato_networks/taxonomy.yaml`. If the change needs

68 +grid or table widget examples, compare against

69 `src/go/plugin/go.d/collector/mysql/taxonomy.yaml`.

71 ## 4. Update the remaining collector artifacts

@@ -84,13 +86,22 @@ From the repo root:

86 ```bash

87 python3 integrations/gen_integrations.py

88 python3 integrations/gen_taxonomy.py --check-only

87 -python3 integrations/check_collector_taxonomy.py

89 +python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD

90 python3 -m unittest integrations.tests.test_taxonomy

91 python3 integrations/gen_docs_integrations.py -c go.d.plugin/<module>

92 +python3 integrations/gen_doc_collector_page.py

93 +python3 integrations/gen_doc_secrets_page.py

94 +# If service-discovery rules or sdext metadata changed:

95 +python3 integrations/gen_doc_service_discovery_page.py

96 ```

98 Use the repo-local `.venv/bin/python` when one exists for the current

93 -worktree.

99 +worktree. If your local base branch is not `master`, adjust the `--pr-diff`

100 +range to the PR base.

101 +

102 +If service-discovery rules or `sdext` metadata changed, run

103 +`python3 integrations/gen_doc_service_discovery_page.py` and commit

104 +`src/collectors/SERVICE-DISCOVERY.md`; this is not handled by CI for you.

105

106 ## 6. Before opening the PR

107

@@ -98,9 +109,15 @@ Run:

109

110 ```bash

111 git status --short

112 +git status --porcelain |

113 + rg '^(\?\?|!!| M|M |A |AM) integrations/(integrations\.(js|json)|taxonomy\.json)$' || true

114 ```

115

116 Commit source changes, generated docs, and taxonomy updates together.

117 Do not commit gitignored runtime artifacts such as

118 `integrations/integrations.js`, `integrations/integrations.json`, or

119 `integrations/taxonomy.json`.

120 +

121 +The generated-artifact status grep MUST return no output before the PR is

122 +opened. If it reports one of those files, remove the local generated artifact

123 +from the commit/worktree state rather than committing it.

.agents/skills/project-create-topology/SKILL.md

@@ -24,6 +24,7 @@ Read these before designing or changing topology payloads:

24 | `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` | Backend/frontend/aggregator migration scope |

25 | `.agents/sow/specs/topology-function-schema.md` | Durable project spec for topology semantics |

26 | `.agents/sow/specs/topology-modes-correlation-aggregation.md` | Mode, correlation, aggregation, and actor modal identification contract |

27 +| `src/go/pkg/topology/v1` | Go production topology payload builders and compact-table helpers |

28 | `.agents/skills/project-writing-collectors/SKILL.md` | Collector quality, Function, validation, and cardinality rules |

30 For transport-level Function behavior, also read:

@@ -40,6 +41,8 @@ developer-facing and must stay in this project skill, not under

41 ## Core Rules

43 - Production payloads carry canonical topology facts for the aggregator and UI.

44 +- Go producers MUST use `src/go/pkg/topology/v1`. The non-v1

45 + `src/go/pkg/topology` payload model is legacy for new producers.

46 - Test-only projection code may reconstruct compatibility payload shapes to

47 prove parity.

48 - Never add compatibility reconstruction fields, old-schema adapter names, or

.agents/skills/project-writing-collectors/SKILL.md

+117 -53

@@ -10,7 +10,28 @@ type: project

11 You are about to add or modify data collection in the Netdata Agent. This skill is a manifesto and a routing map. It tells you the mindset to apply, the principles you cannot violate, the ways the dashboard gets shaped from upstream data, the quality bar that separates a draft from a shippable collector, and where to look for depth. It is not a tutorial — the deep references already exist in the repo. Your job is to know they exist, pick the right one, and produce work that blends with the patterns the maintainers already accept.

13 -The skill is organized as: mental model → best practices → dashboard shaping → quality bar → environment reference → applied per data type → applied per domain. Read top to bottom on your first pass; come back to specific sections as the task narrows.

13 +The skill is organized as: AI fast path → mental model → best practices → dashboard shaping → quality bar → environment reference → applied per data type → applied per domain. For go.d work, follow the fast path first; for other collector families, read top to bottom on your first pass and come back to specific sections as the task narrows.

14 +

15 +## AI Fast Path

16 +

17 +For implementation agents, route to the concrete workflow first and use the

18 +rest of this skill as background:

19 +

20 +- New go.d collector: read `src/go/AGENTS.md`, then

21 + `src/go/plugin/go.d/docs/how-to-write-a-collector.md`,

22 + `.agents/skills/project-writing-go-modules-framework-v2/SKILL.md`, and

23 + `.agents/skills/integrations-lifecycle/recipes/add-go-collector.md`.

24 +- Existing go.d collector update: read `src/go/AGENTS.md`, the collector's

25 + local files, `.agents/skills/integrations-lifecycle/consistency.md`, and

26 + `.agents/skills/integrations-lifecycle/recipes/update-collector.md`.

27 +- V1-to-V2 migration: read `src/go/AGENTS.md`,

28 + `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`, and the V2 skill before

29 + changing code.

30 +- Framework or shared-helper change: stop and satisfy

31 + `src/go/plugin/framework/docs/changing-framework-code.md` before writing

32 + code.

33 +

34 +Do not use this broad skill as the only implementation guide for go.d work.

36 ## 1. Mental model

@@ -40,7 +61,7 @@ This is a truthfulness principle, not a cardinality one. It applies at any cardi

62 Mechanics:

63 - C: `rrdset_is_obsolete___safe_from_collector_thread()` in `src/database/rrdset.c:116` flags `RRDSET_FLAG_OBSOLETE`. Reverse with `rrdset_isnot_obsolete()` (line 140) when the entity reappears.

43 -- go.d: `c.Obsolete = true` on the chart struct; the framework appends `obsolete` to the CHART command. Documented at `src/go/BEST-PRACTICES.md:94-108`.

64 +- go.d V1: `c.Obsolete = true` or `MarkRemove()` on the chart marks it obsolete. go.d V2: chart lifetime is controlled by `charts.yaml` lifecycle policy and `chartengine`; start from `src/go/plugin/go.d/docs/how-to-write-a-collector.md` for new collectors and `src/go/plugin/go.d/docs/migrate-v1-to-v2.md` for migrations.

65 - Anti-flip-flop: if an entity may disappear and reappear quickly, wait roughly 1 minute of absence before obsoleting. Thrashing charts hurt streaming and ML.

67 ### 1.6 Your knowledge is stale — research the current spec

@@ -66,7 +87,11 @@ This is how you avoid shipping a parser that fails on the first real device. If

88 ### 1.8 Mirror an existing Netdata collector

69 -The repo holds 132 go.d modules and 24 internal C plugins. Maintainer patterns live there, not in any prose doc. After you've reality-checked the upstream protocol, pick the closest existing Netdata collector by domain and mirror its structure. Caveat: only 5 go.d modules use V2 — see §5.3.

90 +The repo holds many go.d modules and internal C plugins. Maintainer patterns

91 +live there, not in any prose doc. After you've reality-checked the upstream

92 +protocol, pick the closest existing Netdata collector by domain and mirror its

93 +structure. New go.d modules MUST use framework V2 and start from the current

94 +V2 authoring guide — see §5.3.

96 ### 1.9 Remote-monitored systems are vnodes

@@ -90,6 +115,16 @@ Per-job source priority: `stock < discovered < user < dyncfg`, matched by job id

115

116 Framework-agnostic, ordered by impact.

117

118 +### 2.0 Mandatory clean end state and scope discipline

119 +

120 +You MUST aim for the clean end state, not the smallest diff. While

121 +implementing, keep checking whether the design still looks like the structure

122 +maintainers should want after the work is complete.

123 +

124 +At each coherent batch, you MUST check for scope drift. If the work exposes an

125 +independent collector cleanup, framework change, docs correction, or migration,

126 +either defer it explicitly or submit it as its own step before continuing.

127 +

128 ### 2.1 Test against reality

129

130 Source test data based on what you're collecting:

@@ -109,9 +144,12 @@ order-independent.

144

145 ### 2.2 Hot-path discipline

146

112 -`Collect()` runs every `update_every` seconds. It must:

147 +`Collect()` runs every `update_every` seconds. It MUST:

148

114 -- Allocate buffers, maps, slices, parsed regexes once at `Init()` and reuse them. Reset at the top of `Collect()` if needed; see `ping/collect.go` for a V2 reference.

149 +- Allocate buffers, maps, slices, parsed regexes, matchers, and metric

150 + instruments once at `Init()` / `New()` and reuse them. Reset at the top of

151 + `Collect()` only when needed; see `cato_networks/metrix.go` for the typed

152 + V2 metric-instrument pattern.

153 - Hold persistent connections; reconnect only on failure with backoff.

154 - Cache anything stable between iterations: schema, capabilities, profile selections.

155 - Finish well under one cycle even on a slow target.

@@ -137,9 +175,9 @@ Past pain: an `ebpf.plugin` regression flooded logs because the collection loop

175

176 When a collector emits one chart per discovered entity (process, connection, profile target, container, schema, queue, route), bound the count and let the operator scope it. (Obsoletion of entities the collector knows have gone is a separate concern — see §1.5.)

177

140 -**`max_*` is mandatory for entities that may grow without bounds.** Without a cap, a single misbehaving target (a runaway log rotator, a container churn loop, a vendor-specific deep table) can produce thousands of charts.

178 +**`max_*` is REQUIRED for entities that may grow without bounds.** Without a cap, a single misbehaving target (a runaway log rotator, a container churn loop, a vendor-specific deep table) can produce thousands of charts.

179

142 -**`max_*` must be coupled with selectors.** A cap alone silently truncates whatever happens to land in the first N entries — the operator has no say in *which* entities survive. A selector lets the operator pick what's actually important. Cap and selector together: cap protects the system, selector lets the operator drive.

180 +**`max_*` MUST be coupled with selectors.** A cap alone silently truncates whatever happens to land in the first N entries — the operator has no say in *which* entities survive. A selector lets the operator pick what's actually important. Cap and selector together: cap protects the system, selector lets the operator drive.

181

182 **Where to filter — depends on what the monitored application exposes:**

183

@@ -153,19 +191,26 @@ When a collector emits one chart per discovered entity (process, connection, pro

191 - Histogram / percentile splits with high-cardinality labels (per-IP, per-tenant, per-trace) → multiplicative blow-up.

192 - Per-PID charts with no obsolete handler → growth at process churn rate (the bound is here in §2.5; the obsolete handler is the §1.5 concern).

193

156 -Pattern reference: `src/go/BEST-PRACTICES.md` (search `max`).

194 +For go.d V2 collectors, keep selector/cap behavior in the collector design and

195 +document the public config only when the operator has a real decision to make.

196 +Start from `src/go/plugin/go.d/docs/how-to-write-a-collector.md`.

197

198 ### 2.6 Configuration discipline

199

160 -Tunables live in `config_schema.json` (DYNCFG schema rendered by the dashboard) and `metadata.yaml` (integration page) — both must be complete and mutually consistent. The stock `.conf` shows safe, representative examples — not necessarily every tunable.

200 +Public tunables are part of the collector consistency contract. When a config

201 +option is added, removed, renamed, or given a new default, you MUST follow

202 +`.agents/skills/integrations-lifecycle/consistency.md`; you MUST NOT update

203 +only the Go struct or only the docs. The stock `.conf` shows safe,

204 +representative examples -- not necessarily every tunable.

205

162 -Don't hardcode timeouts, paths, ports, or credentials. Don't let stock conf and schema contradict each other.

206 +Collectors MUST NOT hardcode timeouts, paths, ports, or credentials. Stock

207 +config and schema MUST NOT contradict each other.

208

209 Credentials use the `${env:}/${file:}/${cmd:}/${store:}` indirection — see `src/collectors/SECRETS.md`. Privileged operations route through `src/collectors/utils/ndsudo.c`.

210

211 ### 2.7 Generated artifacts are not source

212

168 -Several artifacts are produced from upstream definitions and must never be hand-edited:

213 +Several artifacts are produced from upstream definitions and MUST NOT be hand-edited:

214

215 - `integrations/<name>.md` — generated from `metadata.yaml` (banner: `DO NOT EDIT THIS FILE DIRECTLY`).

216 - `ibm.d` modules — generated `README.md`, `metadata.yaml`, `config.go`, `zz_generated_*.go` from `contexts.yaml` via `go generate`.

@@ -175,20 +220,14 @@ When a generated file looks wrong, fix the source of truth (`metadata.yaml`, `co

220

221 ### 2.8 Documentation/configuration consistency

222

178 -A new or modified collector ships these in sync:

223 +Collector consistency has one detailed checklist:

224 +`.agents/skills/integrations-lifecycle/consistency.md`. Treat code,

225 +integration metadata, taxonomy, config, stock examples, alerts, and generated

226 +documentation as one unit, but do not maintain a second artifact matrix here.

227

180 -- the code

181 -- `metadata.yaml` — drives integration pages, in-app help, alert references

182 -- `taxonomy.yaml` — places emitted chart contexts in the dashboard TOC

183 - with an ordered `items:` tree; structural strings/`owned_context`

184 - entries own contexts, widgets reference them

185 -- `config_schema.json` — DYNCFG schema rendered by the dashboard

186 -- stock `.conf` — safe, representative example

187 -- `health.d/*.conf` — alert templates bound to chart `context`

188 -- `README.md` — concise narrative

189 -- if exposing a Function: response shape conforming to `src/plugins.d/FUNCTION_UI_SCHEMA.json`

190 -

191 -Treat them as one unit. Change a unit in code → update `metadata.yaml` in the same commit. Add or rename a chart context → update `taxonomy.yaml` or a declared dynamic selector. Add a config knob → update schema, stock conf, and metadata together.

228 +If a collector exposes a Function, its response shape MUST also conform to the

229 +relevant Function schema, such as `src/plugins.d/FUNCTION_UI_SCHEMA.json` or

230 +`src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.

231

232 ### 2.9 Cross-plugin enrichment via netipc

233

@@ -202,7 +241,12 @@ Both clients (consume) and servers (offer) exist in all three languages. Real ex

241

242 ### 2.10 Vnodes for remote targets

243

205 -Set `Vnode` in job config; respect it in `Init()` and DYNCFG handlers. See `src/go/plugin/framework/vnodes/` and `src/go/BEST-PRACTICES.md` (search `Vnode`). Past pain: an older refactor had to retroactively split job-name validation per vnode/domain because earlier collectors hadn't accounted for it.

244 +Set `Vnode` in job config when the collector has one remote target. For Go V2

245 +collectors that emit multiple remote nodes from one job, use

246 +`metrix.HostScope`; see `.agents/sow/specs/go-v2-host-scope.md` and

247 +`src/go/plugin/go.d/docs/how-to-write-a-collector.md`. Past pain: an older

248 +refactor had to retroactively split job-name validation per vnode/domain because

249 +earlier collectors had not accounted for it.

250

251 ## 3. Structuring dashboards

252

@@ -284,22 +328,19 @@ A collector is *production-quality* when it satisfies all of:

328 4. Are gaps preserved (no zero defaults for missing values)?

329 5. Does the collection cycle allocate, log per iteration, or reconnect every cycle?

330 6. Do error logs answer *what operation, what target, what was expected vs observed*?

287 -7. Are config knobs in `config_schema.json` and `metadata.yaml`? Does the stock `.conf` show a representative example?

288 -8. Does `taxonomy.yaml` cover every emitted chart context, or are dynamic contexts declared with `metrics.dynamic_context_prefixes` / `metrics.dynamic_collect_plugins`?

289 -9. Are alerts present in `health.d/`?

290 -10. Is `README.md` updated? (Not the generated `integrations/<name>.md`.)

291 -11. For remote targets: is vnode wiring done?

292 -12. For SNMP: did I extend a profile rather than hardcode OIDs?

293 -13. For statsd / OTEL: did I document and ship the operator-side config (synthetic_charts file or OTEL mapping YAML)?

294 -14. For Prometheus scraping: are selectors correct? Are untyped metrics handled?

295 -15. For cross-plugin enrichment: am I using netipc?

296 -16. For Functions: does the response conform to one of the six shapes? Non-blocking with respect to the collection loop? Schema-validated?

297 -17. For ibm.d only: did I run `go generate` after touching `contexts.yaml`?

298 -18. For new go.d modules: are all four wiring steps done (init.go, go.d.conf, stock conf, README)?

299 -19. Tests: real fixtures or real instances? Would they catch the bug I just fixed?

300 -20. High-cardinality labels / instances: bounded by `max_*` + selectors? Aggregated "Other" bucket or upstream-supplied aggregation present where applicable?

301 -21. Entities that can go away: obsoleted when the collector knows they're gone? Anti-flip-flop window applied where churn is expected?

302 -22. Production-quality criteria above — would this collector survive hours of target outage without leaks or log floods?

331 +7. Did I run the collector consistency checklist in `.agents/skills/integrations-lifecycle/consistency.md`, including the rule that generated integration pages are not hand-authored sources?

332 +8. For remote targets: is vnode wiring done?

333 +9. For SNMP: did I extend a profile rather than hardcode OIDs?

334 +10. For statsd / OTEL: did I document and ship the operator-side config (synthetic_charts file or OTEL mapping YAML)?

335 +11. For Prometheus scraping: are selectors correct? Are untyped metrics handled?

336 +12. For cross-plugin enrichment: am I using netipc?

337 +13. For Functions: does the response conform to one of the six shapes? Non-blocking with respect to the collection loop? Schema-validated?

338 +14. For ibm.d only: did I run `go generate` after touching `contexts.yaml`?

339 +15. For new go.d modules: are all four runtime-load wiring steps done (`collector/init.go` import, `go.d.conf`, stock conf, README)?

340 +16. Tests: real fixtures or real instances? Would they catch the bug I just fixed?

341 +17. High-cardinality labels / instances: bounded by `max_*` + selectors? Aggregated "Other" bucket or upstream-supplied aggregation present where applicable?

342 +18. Entities that can go away: obsoleted when the collector knows they're gone? Anti-flip-flop window applied where churn is expected?

343 +19. Production-quality criteria above — would this collector survive hours of target outage without leaks or log floods?

344

345 ## 5. Plugins and frameworks — what's available and where

346

@@ -321,7 +362,7 @@ Reference section. Use it after the mental model and best practices have framed

362 | `statsd.plugin` | C | All | `src/collectors/statsd.plugin/` | StatsD ingestion + synthetic_charts |

@@ -334,12 +375,13 @@ Path conventions: internal C plugins → `src/collectors/<name>.plugin/`; Go orc

375

376 | If you are doing… | Start with |

377 |---|---|

337 -| New off-the-shelf application integration (no CGO) | `src/go/plugin/go.d/docs/how-to-write-a-collector.md`; V2 reference: `src/go/plugin/go.d/collector/ping/` |

378 +| New off-the-shelf application integration (no CGO) | `src/go/plugin/go.d/docs/how-to-write-a-collector.md`; primary V2 reference: `src/go/plugin/go.d/collector/cato_networks/` |

379 +| Migrating existing go.d collector to V2 | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`; V2 mechanics: `.agents/skills/project-writing-go-modules-framework-v2/SKILL.md` |

380 | New IBM workload integration (CGO) | `src/go/plugin/ibm.d/AGENTS.md`, `src/go/plugin/ibm.d/framework/README.md` |

381 | New Rust plugin | SDK at `src/crates/netdata-plugin/`; reference: `src/crates/netflow-plugin/` |

382 | New SNMP profile (no code change) | `src/go/plugin/go.d/collector/snmp/profile-format.md` |

383 | New interactive Function | `src/go/plugin/framework/functions/README.md`, `src/plugins.d/FUNCTION_UI_SCHEMA.json`, `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md` |

342 -| Topology work | `src/go/pkg/topology/`, `src/go/plugin/go.d/collector/snmp_topology/`, `src/collectors/network-viewer.plugin/` |

384 +| Topology work | `.agents/skills/project-create-topology/SKILL.md`, `src/go/pkg/topology/v1`, `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json` |

385 | Auto-discovery for a new go.d module | rules under `src/go/plugin/go.d/config/go.d/sd/`; engine: `src/go/plugin/agent/discovery/` |

386 | OTEL ingestion | `src/crates/netdata-otel/otel-plugin/` |

387 | Log ingestion (parse → journal) | `src/collectors/log2journal/` and `log2journal.d/` rules |

@@ -351,15 +393,27 @@ Path conventions: internal C plugins → `src/collectors/<name>.plugin/`; Go orc

393

394 ### 5.3 go.d V1 / V2 reality check

395

354 -Only **5 of 132** go.d collectors use V2: `ping`, `mysql`, `azure_monitor`, `powerstore`, `powervault`. The big reference docs (`src/go/BEST-PRACTICES.md`, `src/go/COLLECTOR-LIFECYCLE.md`) describe V1. V2 building blocks have framework READMEs (`src/go/plugin/framework/charttpl/README.md`, `src/go/plugin/framework/chartengine/README.md`, `src/go/pkg/metrix/README.md`); there is no end-to-end V2 tutorial beyond `how-to-write-a-collector.md` plus the `ping/` source.

396 +Most go.d collectors are still V1, but the broad V1 authoring docs have been

397 +retired because they taught stale patterns from general Go paths. Do not use

398 +existing V1 collectors as the shape for new work.

399 +

400 +**New go.d modules MUST use V2.** Start with

401 +`src/go/plugin/go.d/docs/how-to-write-a-collector.md`. Use

402 +`src/go/plugin/go.d/collector/cato_networks/` as the primary modern reference,

403 +but copy focused responsibilities rather than the entire collector. Copying a V1

404 +module mirrors legacy patterns and the maintainers will ask you to migrate.

405

356 -**For new go.d modules: use V2.** Mirror `src/go/plugin/go.d/collector/ping/` (or `mysql/` for V2 + Functions). Copying any other module mirrors V1 and the maintainers will ask you to migrate.

406 +For migrating an existing V1 collector, start with

407 +`src/go/plugin/go.d/docs/migrate-v1-to-v2.md`. Migration is compatibility work;

408 +do not use the new-collector guide to justify chart, config, or lifecycle

409 +contract changes. Temporary V1 parity bridges can help during development, but

410 +the finished collector MUST NOT run through a V1-to-V2 bridge.

411

412 V2 imports: `github.com/netdata/netdata/go/plugins/plugin/framework/collectorapi` and `.../pkg/metrix`. The `CollectorV2` interface lives at `src/go/plugin/framework/collectorapi/collector.go`.

413

414 Lifecycle semantics: `Init()` is one-time setup (failure disables permanently); `Check()` is auto-detection probe (failure disables, retried later); `Collect()` is the hot path (every `update_every` seconds); `Cleanup()` is guaranteed on shutdown.

415

362 -**Silent-failure trap (go.d).** A new go.d module compiles and tests pass even when it is *not loaded* by the plugin at runtime. Loading requires four wiring steps: import in `src/go/plugin/go.d/collector/init.go`, `modules:` toggle in `src/go/plugin/go.d/config/go.d.conf`, stock job config at `src/go/plugin/go.d/config/go.d/<name>.conf`, and entry in `src/go/plugin/go.d/README.md`. Same trap applies to `ibm.d`.

416 +**Silent-failure trap (go.d).** A new go.d module compiles and tests pass even when it is *not loaded* by the plugin at runtime. Runtime loading requires four wiring steps: import in `src/go/plugin/go.d/collector/init.go`, `modules:` toggle in `src/go/plugin/go.d/config/go.d.conf`, stock job config at `src/go/plugin/go.d/config/go.d/<name>.conf`, and entry in `src/go/plugin/go.d/README.md`. Same trap applies to `ibm.d`.

417

418 ### 5.4 ibm.d, Rust SDK, internal C, PLUGINSD

419

@@ -377,7 +431,9 @@ Lifecycle semantics: `Init()` is one-time setup (failure disables permanently);

431 ### 5.5 Build / dev loop

432

433 - go.d unit tests: `cd src/go && go test ./plugin/go.d/collector/<name>/...`

380 -- Single-module dev run: `go run ./cmd/godplugin -m <name> -d`

434 +- Single-module dev run: `timeout 15s go run ./cmd/godplugin -m <name> -d`

435 + from `src/go`; success means the module registers, starts a job, and keeps

436 + running until the timeout stops it.

437 - Rust: `cargo test -p <crate>`

438 - Whole-project install: `./netdata-installer.sh`

439

@@ -421,7 +477,9 @@ Topology is its own data type — directed/undirected graphs of nodes and links.

477 - **SNMP-discovered topology** (`src/go/plugin/go.d/collector/snmp_topology/`) — LLDP/CDP neighbors, BRIDGE-MIB FDB, Q-BRIDGE FDB, ARP tables, STP. Builds on SNMP profiles; extending profiles is usually the right starting point.

478 - **Live socket topology** (`src/collectors/network-viewer.plugin/`) — local L3/L4 sockets and their inferred connections.

479 - **Streaming graph** (`src/streaming/`) — Netdata parent/child topology.

424 -- **Topology library** at `src/go/pkg/topology/` — shared types and providers consumed by the topology collectors.

480 +- **Topology library** at `src/go/pkg/topology/v1` — production Go payload

481 + helpers for new topology producers. The non-v1 `src/go/pkg/topology/` payload

482 + model is legacy and must not be used for new topology work.

483

484 Topology is consumed via Functions (`topology:*` family), not via metrics. The cardinality of network edges is too high for time-series storage and the use case is interactive lookup.

485

@@ -443,13 +501,17 @@ These are descriptive patterns — what existing Netdata collectors do. Use them

501

502 ### 7.1 Database collectors

503

446 -DB collectors typically pair metrics (uptime, connections, query rates, replication lag, lock counts, cache hit ratios) with **Functions for live query analysis**: top queries, slow queries, currently-running queries, locks. Real examples:

504 +DB collectors often pair metrics (uptime, connections, query rates, replication lag, lock counts, cache hit ratios) with **Functions for live query analysis**: top queries, slow queries, currently-running queries, locks. Real examples:

505

506 - **MySQL** (`src/go/plugin/go.d/collector/mysql/`) — metrics + `mysqlfunc/top_queries.go` + processlist via `collect_process_list.go`.

507 - **PostgreSQL** (`src/go/plugin/go.d/collector/postgres/`) — metrics + `func_top_queries.go` + `func_running_queries.go`, dispatched through `func_router.go`.

508 - MongoDB / Redis are metrics-only today, but the same Function pattern fits if the use case demands it.

509

452 -If you build a DB collector with metrics only, expect the maintainers to ask why you didn't add a query Function — the operator value of seeing "what's slow right now" is high and the pattern is established.

510 +Before adding a query Function, decide whether it is in scope for the current

511 +work and record the product/design decision. The operator value of seeing

512 +"what's slow right now" is high and the pattern is established, but Functions

513 +are still a feature surface, not something to add accidentally during unrelated

514 +metric work.

515

516 ### 7.2 Network and SNMP collectors

517

@@ -501,7 +563,7 @@ Internal C plugins under `src/collectors/`. Reuse shared metric definitions from

563 | Plugin types and privileges | choosing where to add a collector | `src/collectors/README.md` |

564 | External plugin protocol | non-Go external plugin | `src/plugins.d/README.md` |

565 | go.d V2 authoring | adding a `go.d` module | `src/go/plugin/go.d/docs/how-to-write-a-collector.md` |

504 -| go.d V1 best practices / lifecycle | working in legacy V1 module | `src/go/BEST-PRACTICES.md`, `src/go/COLLECTOR-LIFECYCLE.md` |

566 +| go.d V1-to-V2 migration | migrating existing go.d collector | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md` |

567 | Functions backend (Go / Rust) | implementing a Function | `src/go/plugin/framework/functions/README.md`, `src/crates/netdata-plugin/rt/src/lib.rs` |

568 | Functions UI schema & guides | response shapes and patterns | `src/plugins.d/FUNCTION_UI_SCHEMA.json`, `src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_UI_REFERENCE.md` |

569 | Topology Function schema & guide | topology actors, links, evidence, overlays | `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`, `src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md`, `src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md` |

@@ -516,11 +578,13 @@ Internal C plugins under `src/collectors/`. Reuse shared metric definitions from

578 | Prometheus mapping | generic exposition scrape | `src/go/plugin/go.d/collector/prometheus/README.md` |

579 | log2journal | parsing application logs into the journal | `src/collectors/log2journal/log2journal.d/` |

580 | Auto-discovery rules | adding service-detection rules | `src/go/plugin/go.d/config/go.d/sd/{net_listeners,docker,snmp,http}.conf` |

519 -| Topology library | topology providers in Go | `src/go/pkg/topology/` |

581 +| Topology library | topology producers in Go | `src/go/pkg/topology/v1` |

582 | netipc cross-plugin enrichment | C / Go / Rust | `src/libnetdata/netipc/`, `src/go/pkg/netipc/`, `src/crates/netipc/` |

583 | DYNCFG protocol | dynamic configuration | `src/plugins.d/DYNCFG.md`, `docs/developer-and-contributor-corner/dyncfg.md` |

584 | Health alerts reference | alert template authoring | `src/health/REFERENCE.md`, `src/health/alert-configuration-ordering.md` |

585 | Integrations pipeline | doc generation from `metadata.yaml` | `integrations/README.md` |

586 +| Go framework changes | changing shared Go collector/runtime framework code | `src/go/plugin/framework/docs/changing-framework-code.md` |

587 +| go.d V1-to-V2 migration | migrating existing go.d collectors | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md` |

588 | Credentials in config | `${env:}/${file:}/${cmd:}/${store:}` | `src/collectors/SECRETS.md` |

589 | Privileged operations | restricted setuid helper | `src/collectors/utils/ndsudo.c` |

590

.agents/skills/project-writing-go-modules-framework-v2/SKILL.md

+98 -40

@@ -11,78 +11,132 @@ source files for evidence.

11 ## Read First

13 - Contract: `src/go/plugin/framework/collectorapi/collector.go`

14 +- Framework-change workflow:

15 + `src/go/plugin/framework/docs/changing-framework-code.md`

16 +- Canonical new-collector guide:

17 + `src/go/plugin/go.d/docs/how-to-write-a-collector.md`

18 +- Helper-package guide:

19 + `src/go/plugin/go.d/docs/helper-packages.md`

20 +- V1-to-V2 migration guide:

21 + `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`

22 - Runtime/chart lifecycle: `src/go/plugin/framework/chartengine/README.md`

23 - Template format: `src/go/plugin/framework/charttpl/README.md`

24 - Host scopes/vnodes: `.agents/sow/specs/go-v2-host-scope.md`

17 -- Closest examples:

18 - - `src/go/plugin/go.d/collector/azure_monitor/` for dynamic scopes/profiles.

19 - - `src/go/plugin/go.d/collector/ping/` for the smallest V2 shape.

20 - - `src/go/plugin/go.d/collector/mysql/` for migration compatibility.

21 - - `src/go/plugin/go.d/collector/powervault/` and `powerstore/` for remote

22 - discovery, labels, and chart templates.

25 +- Primary modern example: `src/go/plugin/go.d/collector/cato_networks/`.

26 + Use focused pieces from it, not the whole collector shape.

27 +- Older V2 collectors can still be useful for local patterns, but review them

28 + for stale style before treating them as examples.

29 +

30 +## Decision Discipline

31 +

32 +- You MUST aim for the clean end state, not the smallest collector diff. If a

33 + framework capability is missing and the problem is general, design the

34 + framework change instead of hiding the issue in collector-local glue.

35 +- If any framework-scope package changes, stop and satisfy

36 + `src/go/plugin/framework/docs/changing-framework-code.md` before writing code.

37 +- You MUST re-check scope after each coherent batch. If the work reveals an

38 + independent collector cleanup, framework fix, or integration-doc change,

39 + either defer it explicitly or land it separately before continuing.

41 ## Core Style

26 -- Register with `CreateV2`; expose `Config: func() any { return &Config{} }`.

27 -- `New()` owns defaults, `metrix.NewCollectorStore()`, and test seams.

28 -- Store `metrix.CollectorStore`; implement `MetricStore()`.

29 -- Implement `ChartTemplateYAML()`; prefer embedded `charts.yaml`.

30 -- `Collect(ctx)` returns `error` and writes metrics to `metrix`; it does not

43 +- New collectors MUST implement `collectorapi.CollectorV2` from

44 + `src/go/plugin/framework/collectorapi/collector.go` and register via

45 + `CreateV2`.

46 +- `New()` SHOULD own defaults, `metrix.NewCollectorStore()`, typed metric

47 + instruments, and test seams.

48 +- V2 collectors MUST write metrics through `metrix.CollectorStore` during

49 + `Collect()` and provide chart template YAML through `ChartTemplateYAML()`;

50 + embedded `charts.yaml` is RECOMMENDED.

51 +- `Collect(ctx)` MUST return `error` and write metrics to `metrix`; it MUST NOT

52 return a V1 `map[string]int64`.

32 -- Keep files boring: `collector.go`, `collect.go`, `metrics.go`,

33 - `charts.yaml`, focused domain helpers, focused tests.

53 +- Files SHOULD stay boring: public lifecycle methods in `collector.go`, setup

54 + helpers in `init.go` when needed, orchestration in `collect.go`, distinct

55 + upstream operations in `collect_<operation>.go`, metrics in `metrix.go` /

56 + `write_metrics.go`, focused tests.

57 +- Before adding custom HTTP, selector, logging, command-execution, SQL, ping,

58 + or log-file plumbing, check `src/go/plugin/go.d/docs/helper-packages.md` and

59 + reuse an existing helper when it fits.

60 +- If Functions exist, isolate them in a `<name>func/` subpackage with a narrow

61 + `Deps` interface declared there. The Function package MUST NOT import the

62 + collector package or hold `*Collector`.

63 +- Public config options SHOULD stay small and justified. A proposed config

64 + option MUST name the concrete operator decision it enables; "operators may

65 + want to tune it" is not enough. Internal tuning SHOULD use constants unless

66 + the operator has a real decision to make.

68 ## Metrics And Charts

37 -- Create instruments once when the metric surface is known.

70 +- Instruments SHOULD be created once when the metric surface is known.

71 - Use `store.Write().SnapshotMeter("")` for normal metrics.

72 - Use `Vec(...)` for labels, `Gauge` for current values,

73 `Counter.ObserveTotal()` for source counters, and `StateSet` for fixed

74 one-active-state values.

42 -- Use stable metric names that `charts.yaml` selects.

75 +- Metric names MUST be stable and selected by `charts.yaml`.

76 - In `charts.yaml`: use `version: v1`, `context_namespace`, `instances.by_labels`,

44 - `algorithm: incremental` for counters, and `absolute` for gauges.

77 + `label_promotion`, `algorithm: incremental` for counters, and `absolute` for

78 + gauges.

79 - Put multipliers, divisors, hidden flags, and float formatting in the chart

80 template, not ad hoc chart-emission code.

82 ## Compatibility Rules

84 +- For V1-to-V2 migrations, start with

85 + `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`.

86 +

87 +### Migration Hard Stops

88 +

89 +- A collector using V1 chart `Vars` is blocked until framework support, an

90 + approved equivalent design, or explicit breaking-alert approval exists.

91 +- `collecttest.AssertChartCoverage` is not chart-identity parity; it cannot

92 + prove old chart IDs, family, priority, lifecycle, labels, or alert variables.

93 +- A finished migration MUST pass an import/runtime-path audit proving no V1

94 + collection path or V1 map-to-`metrix` bridge remains reachable from normal

95 + execution.

96 +

97 +- Temporary V1-to-V2 parity bridges MAY be used during development, but the

98 + finished collector MUST NOT keep a runtime V1 map-to-`metrix` bridge.

99 - For migrations, first create a compatibility manifest covering chart IDs,

100 contexts, dimension IDs/names, labels, config keys, DynCfg schema keys,

101 stock config, alerts, docs, and lifecycle behavior.

53 -- Preserve existing public contracts unless the SOW records an explicit breaking

54 - decision.

55 -- Keep old YAML/JSON field names. Add new config as opt-in when cardinality,

56 - cost, or user-visible identity could surprise existing users.

57 -- Keep `metadata.yaml`, `config_schema.json`, stock config, health alerts, and

58 - README synchronized with code.

59 -- Never log raw secrets, DSNs, bearer tokens, or URLs with embedded credentials.

102 +- Migrations MUST preserve existing public contracts unless the SOW records an

103 + explicit breaking decision.

104 +- Migrations MUST keep old YAML/JSON field names. Add new config as opt-in when

105 + cardinality, cost, or user-visible identity could surprise existing users.

106 +- Collector integration artifacts MUST follow

107 + `.agents/skills/integrations-lifecycle/consistency.md`; do not preserve a

108 + partial local artifact checklist in V2 collector work.

109 +- MUST NOT log raw secrets, DSNs, bearer tokens, or URLs with embedded

110 + credentials.

111

112 ## Hot-Path Logging

113

63 -- Do not emit `Warningf`/`Errorf` every collection cycle for a recoverable

64 - partial failure. Use the built-in logger limiter:

114 +- Collectors MUST NOT emit `Warningf`/`Errorf` every collection cycle for a

115 + recoverable partial failure. Use the built-in logger limiter:

116 `c.Limit("collector:stable-operation-key", 1, time.Hour).Warningf(...)`.

66 -- Keep limiter keys stable and low-cardinality. Use operation names, not entity

67 - IDs, labels, URLs, raw errors, or user-controlled values.

117 +- Limiter keys MUST be stable and low-cardinality. Use operation names, not

118 + entity IDs, labels, URLs, raw errors, or user-controlled values.

119 - `Once()` is reset by `JobV2.runOnce()`, so it is useful inside one cycle only;

120 it is not cross-cycle spam protection.

70 -- Full collection failure should still return an error with context so the job

121 +- Full collection failure SHOULD still return an error with context so the job

122 retry path handles it. Limit only fail-soft warnings/errors where collection

123 continues with partial or stale data.

124

125 ## Host Scopes

126

76 -- Use host scopes only after a product decision says the data belongs on a

77 - generated vnode.

78 -- Keep `ScopeKey` and `GUID` deterministic.

79 -- Add `_vnode_type=<source>` on collector-generated vnodes.

80 -- Bound and document cardinality. Do not create VM/disk/NIC/path/sensor scopes

81 - by default.

127 +- Host scopes SHOULD be used only after a product decision says the data belongs

128 + on a generated vnode.

129 +- `ScopeKey` and `GUID` MUST be deterministic.

130 +- Collector-generated vnodes MUST set `_vnode_type=<source>`.

131 +- Host-scope cardinality MUST be bounded and documented. Collectors SHOULD NOT

132 + create VM/disk/NIC/path/sensor scopes by default.

133 +- Scope identity MUST use stable IDs. Human-readable names SHOULD be hostnames

134 + or promoted labels only.

135

136 ## Tests

137

85 -At minimum, V2 work needs:

138 +At minimum, V2 work MUST include these tests, or the PR/SOW MUST justify why a

139 +specific item does not apply:

140

141 - config YAML/JSON serialization compatibility;

142 - `Init`, `Check`, `Collect`, and `Cleanup` lifecycle coverage;

@@ -94,8 +148,12 @@ At minimum, V2 work needs:

148

149 ## Pre-PR Check

150

97 -- No V1 `map[string]int64` collection path remains unless intentionally kept for

98 - a compatibility bridge.

99 -- Existing public chart/metric/config identity is preserved.

100 -- New labels and scopes are bounded and documented.

101 -- Enrichment is split from the V2 compatibility migration when possible.

151 +- A finished V1-to-V2 migration MUST NOT keep a runtime

152 + `map[string]int64` collection path or V1 map-to-`metrix` bridge.

153 +- The PR description or design note MUST enumerate affected collector

154 + consistency artifacts and justify every artifact that did not need a matching

155 + change. SHOULD-level exceptions and escape hatches MUST be reviewer-visible.

156 +- Existing public chart/metric/config identity MUST be preserved unless the SOW

157 + records an explicit breaking decision.

158 +- New labels and scopes MUST be bounded and documented.

159 +- Enrichment SHOULD be split from the V2 compatibility migration when possible.

.gitignore

@@ -238,3 +238,6 @@ install.sh

238 .mcpregistry_github_token

239 .mcpregistry_registry_token

240 .playwright-mcp/

241 +

242 +# Local AI/agent planning notes. Durable SOW/spec work lives under .agents/.

243 +TODO-*.md

AGENTS.md

+34 -26

@@ -6,6 +6,17 @@ This repository is the Netdata Agent codebase. It is a large, multi-language, mu

7 Work in this repository must prioritize root-cause understanding, correctness, performance, maintainability, portability, security, and consistency with existing project conventions.

9 +## Requirement Language

10 +

11 +This repository uses RFC-style requirement language:

12 +

13 +- **MUST** / **REQUIRED**: mandatory. Work that violates it is not acceptable

14 + unless the user explicitly changes the requirement.

15 +- **MUST NOT**: prohibited.

16 +- **SHOULD** / **RECOMMENDED**: expected default. Deviate only with evidence

17 + and explain the trade-off.

18 +- **MAY** / **OPTIONAL**: allowed, not required.

19 +

20 CRITICAL RULES:

22 1. You MUST ALWAYS find the root cause of a problem, before offering/giving a solution.

@@ -17,6 +28,24 @@ CRITICAL RULES:

28 3. Do not duplicate code.

29 First check if similar code already exists and reuse it.

31 +## Mandatory Development Principles

32 +

33 +These principles are mandatory for every task in this repository:

34 +

35 +1. **Clean end state over less churn.**

36 + You MUST always aim for the clean end state, not the smallest diff. While

37 + designing and implementing, actively search for the structure that should

38 + exist after the work is complete. You MUST periodically re-evaluate

39 + already-written changes against that target; do not keep a compromise only

40 + because it already exists in the branch.

41 +

42 +2. **Scope discipline at every step.**

43 + At each milestone, you MUST check whether the work has drifted outside the

44 + approved scope. If the new work is valid but independent, you MUST defer it

45 + to a later step or pause and submit the independent work first, then rebase

46 + the current branch after it merges. Complex features MUST be delivered in

47 + coherent steps where each step builds on the previous one.

48 +

49 USER COMMUNICATION:

51 1. ALWAYS DO YOUR HOMEWORK BEFORE ASKING QUESTIONS OR REQUESTING USER DECISIONS.

@@ -336,7 +365,7 @@ Runtime input skills:

365 Purpose: mirror maintainer-preferred framework V2 patterns from accepted collectors so new or migrated modules blend with repository style.

366

367 - `.agents/skills/integrations-lifecycle/`

339 - Trigger: editing any `metadata.yaml` or collector `taxonomy.yaml`; modifying `integrations/` generators, schemas, taxonomy registries, or templates; working with `integrations.js` / `integrations.json` / `integrations/taxonomy.json` / per-integration `.md` files / `COLLECTORS.md` / `SECRETS.md` / `SERVICE-DISCOVERY.md`; ibm.d module generation (`contexts.yaml` -> `metadata.yaml`); CI workflows `generate-integrations.yml` and `check-markdown.yml`; the collector-consistency rule.

368 + Trigger: editing any `metadata.yaml` or collector `taxonomy.yaml`; modifying `integrations/` generators, schemas, taxonomy registries, or templates; debugging generated gitignored integration outputs (`integrations.js`, `integrations.json`, `integrations/taxonomy.json`); working with committed per-integration `.md` files / `COLLECTORS.md` / `SECRETS.md` / `SERVICE-DISCOVERY.md`; ibm.d module generation (`contexts.yaml` -> `metadata.yaml`); CI workflows `generate-integrations.yml` and `check-markdown.yml`; the collector-consistency rule.

369 Status: live. SKILL.md plus per-domain guides (`pipeline.md`, `schema-reference.md`, `per-type-matrix.md`, `artifacts-and-banners.md`, `ibm-d.md`, `consistency.md`, `in-app-contract.md`, `gotchas.md`) and `recipes/`, `how-tos/` directories.

370

371 - `.agents/skills/learn-site-structure/`

@@ -417,31 +446,10 @@ All existing project-specific instructions in this file remain active. The SOW f

446

447 ## Collector Consistency Requirements

448

420 -When working on collectors (especially Go collectors), ALL of the following files MUST be kept in sync before creating a PR:

421 -

422 -1. **The code** - All .go files implementing the collector

423 -2. **metadata.yaml** - Proper information for the Netdata integrations page, including:

424 - - Metric descriptions with correct units

425 - - Alert definitions

426 - - Setup instructions

427 - - Configuration examples

428 -3. **config_schema.json** - Schema for dynamic configuration in the dashboard

429 -4. **Stock config file** (.conf file) - Example configuration users edit manually

430 -5. **Health alerts** (health.d/*.conf) - Alert definitions for the collector metrics

431 -6. **README.md** - Comprehensive documentation describing:

432 - - What the collector monitors

433 - - How it works

434 - - Configuration options

435 - - Troubleshooting

436 -7. **taxonomy.yaml** - Dashboard table-of-contents placement for the collector's chart contexts

437 -

438 -These files MUST be consistent with each other. For example:

439 -- If units change in code, they MUST be updated in metadata.yaml

440 -- If new metrics are added, they MUST be documented in metadata.yaml and README.md

441 -- If configuration options change, they MUST be updated in config_schema.json, stock config, and documentation

442 -- If chart contexts are added, removed, or renamed, taxonomy.yaml MUST still resolve to real metadata.yaml contexts or declared dynamic selectors

443 -

444 -Unlike the other consistency artifacts, taxonomy.yaml coverage is enforced fatally in CI by `integrations/check_collector_taxonomy.py` running in `.github/workflows/check-markdown.yml`.

449 +When working on collectors, runtime behavior, metrics, charts, configuration,

450 +alerts, taxonomy, and generated documentation MUST stay consistent in one PR.

451 +The detailed collector consistency checklist and CI enforcement notes live in

452 +`.agents/skills/integrations-lifecycle/consistency.md`.

453

454 ## C code

455 - gcc, clang, glibc and muslc

integrations/gen_docs_integrations.py

+1 -1

@@ -609,7 +609,7 @@ def main():

609 parser.add_argument(

610 "-c",

611 "--collector",

612 - help="Generate docs only for this collector (plugin/module), e.g. 'go.d/snmp' or 'apps.plugin/groups'",

612 + help="Generate docs only for this collector (plugin/module), e.g. 'go.d.plugin/snmp' or 'apps.plugin/groups'",

613 default=None,

614 )

615 args = parser.parse_args()

src/go/AGENTS.md

+108 -13

@@ -1,19 +1,114 @@

1 -# Collector Authoring Checklist

1 +# Go Area Instructions

3 -CRITICAL: Never write raw sensitive data to durable artifacts. This includes passwords, API keys, bearer tokens, SNMP communities, private keys, connection strings with embedded credentials, session cookies, community member names, customer names, customer identifiers, personal data, non-private IP addresses that can identify customers, private endpoints, account IDs, and proprietary incident details.

3 +This file routes Go-specific work under `src/go/`. Repository-wide rules in

4 +the repo-root `AGENTS.md` still apply. More specific `AGENTS.md` files under

5 +subdirectories override this file for that subtree. Paths below are

6 +repo-relative unless stated otherwise.

5 -This file is a quick landing page for humans and AI assistants contributing to the IBM.d plugin. For the full documentation follow these links:

8 +## Sensitive Data

7 -- [Plugin overview & build instructions](README.md)

8 -- [IBM.D framework internals](framework/README.md)

9 -- [Go collector best practices](BEST-PRACTICES.md)

10 +The repo-root `AGENTS.md` sensitive-data policy applies in full to all durable

11 +Go artifacts, including code comments, docs, specs, skills, and SOWs.

11 -When implementing or modifying a collector:

13 +## Mandatory Development Principles

13 -1. Read the framework guide to understand contexts, generators, and the module layout.

14 -2. Keep `contexts.yaml`, `config.go`, metadata, schema, README, and health alerts synchronized by running `go generate`.

15 -3. Use conservative defaults – configuration always overrides auto-detection.

16 -4. Log clearly, avoid spamming, and never fabricate metric values.

17 -5. Verify with `ibm.d.plugin --dump` before sending a pull request.

15 +The repo-root clean-end-state and scope-discipline principles are mandatory and

16 +apply in full. For Go work, enforce them explicitly when changing collectors,

17 +framework code, runtime behavior, `metrix`, chart templates, Functions,

18 +topology, tests, specs, or skills.

19 -That’s it! Dive into the module directories for concrete examples.

20 +- You MUST prefer the clean framework or collector shape over preserving a

21 + smaller diff.

22 +- You MUST re-check scope after each coherent batch. If a separate framework fix,

23 + collector cleanup, or docs rewrite becomes necessary, split it into its own

24 + step or submit it independently before continuing.

25 +- You SHOULD use RFC-style requirement language (`MUST`, `SHOULD`, `MAY`) in

26 + Go-area specs, skills, and instructions when documenting enforceable rules.

27 +

28 +## Task Routing

29 +

30 +| Work area | Start here | Notes |

31 +|---|---|---|

32 +| New go.d collector | `src/go/plugin/go.d/docs/how-to-write-a-collector.md` | New go.d collectors use framework V2. |

33 +| Migrating go.d V1 collector to V2 | `src/go/plugin/go.d/docs/migrate-v1-to-v2.md` | Preserve public contracts unless a breaking change is explicitly approved. |

34 +| go.d V2 implementation details | `.agents/skills/project-writing-go-modules-framework-v2/SKILL.md`, `src/go/pkg/metrix/README.md`, `src/go/plugin/framework/charttpl/README.md`, `src/go/plugin/framework/chartengine/README.md` | Use the skill for maintainer style and the READMEs for framework API contracts. Editing `metrix` or framework packages is framework-gated work. |

35 +| go.d helper packages | `src/go/plugin/go.d/docs/helper-packages.md` | Check existing HTTP, config-option, matcher, logger, socket, command, SQL, ping, log-file, and cloud-auth helpers before adding custom plumbing. |

36 +| Collector design across plugins | `.agents/skills/project-writing-collectors/SKILL.md` | Use for NIDL, cardinality, obsoletion, missing data, logging, and config discipline. |

37 +| Integration metadata, taxonomy, generated docs | `.agents/skills/integrations-lifecycle/SKILL.md`, `.agents/skills/integrations-lifecycle/consistency.md` | Source artifacts and generated artifacts MUST stay synchronized. |

38 +| IBM.d work | `src/go/plugin/ibm.d/AGENTS.md` | IBM.d has a generator-driven workflow; go.d V2 layout rules MUST NOT be applied there. |

39 +| Function handlers | `src/go/plugin/framework/functions/README.md`, `src/go/tools/functions-validation/README.md` | Collector Functions SHOULD be isolated behind narrow dependencies. |

40 +| Topology payloads | `.agents/skills/project-create-topology/SKILL.md`, `.agents/sow/specs/topology-function-schema.md`, `src/go/pkg/topology/v1` | New topology producers MUST use the production `netdata.topology.v1` schema. |

41 +| Host scopes / vnodes | `.agents/sow/specs/go-v2-host-scope.md`, `src/go/plugin/go.d/collector/azure_monitor/` | Use host scopes when one job emits metrics for resources that SHOULD appear as separate Netdata nodes. |

42 +| Matchers/selectors | `src/go/pkg/matcher/README.md` | Prefer existing matcher APIs over custom selector grammars. |

43 +| Core framework changes | `src/go/plugin/framework/docs/changing-framework-code.md` and `Core Framework Change Gate` below | The applicable approval tier MUST be satisfied before implementation. |

44 +

45 +## New go.d Collector Rules

46 +

47 +- New go.d collectors MUST implement `collectorapi.CollectorV2` from

48 + `src/go/plugin/framework/collectorapi/collector.go` and register via

49 + `CreateV2`. This includes writing metrics through `metrix.CollectorStore` and

50 + providing `ChartTemplateYAML()`.

51 +- New go.d collector guidance MUST NOT teach or copy the V1

52 + `Collect() map[string]int64` pattern for new

53 + collectors.

54 +- Collector runtime, metric, chart, config, alert, taxonomy, and documentation

55 + changes MUST follow the repository collector consistency policy. The detailed

56 + checklist lives in `.agents/skills/integrations-lifecycle/consistency.md`.

57 +- Public config options SHOULD be added only when they represent a real

58 + operator decision. Implementation tuning such as page sizes, scan windows,

59 + retry limits, and cadence SHOULD use internal constants unless user control is

60 + clearly justified.

61 +- New collectors MUST NOT inherit unsupported config knobs from adjacent

62 + collectors or generic templates.

63 +- Collector-local globals, singletons, adapters, caches, or glue layers that

64 + substitute for missing shared framework/helper capabilities are

65 + framework-scope work and MUST follow

66 + `src/go/plugin/framework/docs/changing-framework-code.md` before

67 + implementation.

68 +- If the collector exposes Functions, put Function code in a dedicated

69 + `<name>func/` package behind a narrow `Deps` interface declared in that

70 + package. The Function package MUST NOT import the collector package or hold

71 + `*Collector`.

72 +- If the collector emits topology, it MUST use `netdata.topology.v1`, the Go

73 + producer model in `src/go/pkg/topology/v1`, and validate payloads against

74 + `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`. New producers MUST NOT use

75 + legacy topology payloads.

76 +- If one job emits metrics for multiple remote resources that SHOULD appear as

77 + separate Netdata nodes, it MUST use V2 host scopes/vnodes.

78 +

79 +## go.d V1-to-V2 Migration Rules

80 +

81 +- V1-to-V2 migrations MUST start with

82 + `src/go/plugin/go.d/docs/migrate-v1-to-v2.md`.

83 +- Migrations MUST preserve chart IDs, contexts, dimensions, config keys,

84 + defaults, health lookups, metadata, taxonomy, stock config, and service

85 + discovery behavior unless the user explicitly approves a breaking change.

86 +- Compatibility migration SHOULD be separate from enrichment such as new labels,

87 + host scopes, topology, Functions, or config expansion.

88 +- Completed migrations MUST NOT keep a runtime V1-to-V2 bridge. Temporary V1

89 + logic can be used during development for parity checks, but it MUST be

90 + removed from the final migrated collector.

91 +

92 +## Core Framework Change Gate

93 +

94 +Changes to shared Go framework code are high-blast-radius work. Before changing

95 +these areas, read `src/go/plugin/framework/docs/changing-framework-code.md`.

96 +That guide is the canonical owner of the framework-change scope list, required

97 +design note, validation expectations, and artifact checks.

98 +

99 +Framework-change implementation MUST NOT begin until the applicable approval

100 +tier in that guide is satisfied. You SHOULD prefer a clean framework extension

101 +over collector-local glue, global variables, or private package coupling when

102 +the problem is general.

103 +

104 +## Batching And Review

105 +

106 +- Changes SHOULD stay atomic. If a collector or framework task grows, split it

107 + into coherent batches before review becomes difficult.

108 +- At every batch boundary, you MUST re-evaluate clean end state and scope. If

109 + the branch now contains independent work, pause and submit that work

110 + separately or defer it before continuing.

111 +- Changes MUST NOT mix framework changes, collector migrations, and

112 + integration-doc regeneration unless they are required for one coherent

113 + behavior change.

114 +- For Go test style, follow the repo-root `AGENTS.md` "Go test style" section.

src/go/BEST-PRACTICES.md deleted

-387

@@ -1,387 +0,0 @@

1 -# Writing Go Collectors for Netdata

2 -

3 -This guide documents the patterns, conventions, and best practices for writing Go collectors (modules) for Netdata's go.d.plugin framework.

4 -

5 -## File Organization

6 -

7 -Each collector MUST have a directory structure like this:

8 -

9 -```text

10 -collector/mymodule/

11 -├── README.md # Developer documentation about the plugin

12 -├── collector.go # Main collector implementation

13 -├── collect.go # Collection logic (optional)

14 -├── charts.go # Chart definitions

15 -├── config_schema.json # Configuration schema

16 -├── metadata.yaml # Netdata marketplace metadata (includes user documentation about the plugin)

17 -├── init.go # Init helpers (optional)

18 -└── testdata/ # Test fixtures

19 -```

20 -

21 -- **collector.go**: Module struct, Init(), Check(), configuration

22 -- **collect.go**: Collect() implementation (can be in collector.go for simple modules)

23 -- **collect_*.go**: Split collection logic for complex collectors (e.g., collect_databases.go)

24 -- **charts.go**: Chart templates and creation functions

25 -- **config_schema.json**: JSON schema for web UI configuration

26 -- **metadata.yaml**: Marketplace metadata, metric descriptions

27 -- **init.go**: Validation and initialization helpers (optional)

28 -

29 -## Stock Config Files

30 -

31 -Each collector MUST have a stock configuration file in go.d/config or ibm.d/config following a simple pattern:

32 -

33 -```yaml

34 -## All available configuration options, their descriptions and default values:

35 -## https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/collector/mymodule#configuration

36 -

37 -#jobs:

38 -# - name: local

39 -# url: http://localhost:8080

40 -#

41 -# - name: remote

42 -# url: http://203.0.113.0

43 -# username: username

44 -# password: password

45 -```

46 -

47 -For more detailed documentation, create self-documenting configs in custom plugins:

48 -

49 -```yaml

50 -## netdata configuration for MyModule

51 -## This collector monitors...

52 -

53 -## Prerequisites:

54 -## 1. First requirement

55 -## 2. Second requirement

56 -

57 -jobs:

58 - ## Example: Basic configuration

59 - # - name: local

60 - # url: http://localhost:8080

61 - # update_every: 5

62 -

63 - ## Example: With authentication

64 - # - name: prod

65 - # url: https://prod.example.com

66 - # username: monitor

67 - # password: secret

68 -```

69 -

70 -## Dynamic charts with activity check

71 -

72 -Collectors must NEVER filter the monitored items based on activity or volume. Netdata expects a stable number of metrics. Of course charts and dimensions can come and go, but it is important this NOT to be too frequent for a large number of metrics.

73 -

74 -The best practice is to have 2 configuration parameters:

75 -

76 -- `maxXXX`, as a number

77 -- `selectXXX` as a simpler pattern (glob)

78 -

79 -The combination of the two works like this: "If the number of items is less than {maxXXX} chart them all, otherwise chart only the ones matched by {selectXXX}".

80 -

81 -This ensures that small deployments are monitored in full, while bigger deployments require configuration in order to enable monitoring this kind of items.

82 -

83 -The default `maxXXX` MUST be 100-500.

84 -The default `selectXXX` should be unset or empty to match nothing.

85 -

86 -## Chart Obsolescence Mechanism

87 -

88 -Collectors MUST obsolete charts when they are no longer collected.

89 -

90 -IMPORTANT: Obsoletion flushes Netdata memory. Unecessary obsolation and recreation leads to increased storage footprint for the metrics, because TSTB flushes them prematurely before having enough samples for Gorilla compression + ZSTD to efficiently compress them.

91 -

92 -Creating charts and obsoleting them in a flip-flot fashion is a bad practice.

93 -

94 -If there is the risk for charts to flip-flop active-obsolete, the best practice is to obsolete them after 1 minute of absence.

95 -

96 -The go.d framework provides a built-in mechanism for marking charts as obsolete when they are no longer needed. This is available for ALL charts - both static (base) charts and dynamic instance charts.

97 -

98 -1. **Marking a Chart as Obsolete**:

99 - ```go

100 - // Method available on any chart

101 - func (c *Chart) MarkRemove() {

102 - c.Obsolete = true // Adds "obsolete" to CHART command options

103 - c.remove = true // Flags for removal from charts slice

104 - }

105 - ```

106 -

107 -2. **Framework Behavior**:

108 - - When a chart is marked obsolete, the framework appends "obsolete" to the options field in the CHART command

109 - - Example output: `CHART 'module.metric' '' 'Title' 'units' 'family' 'context' 'line' '100' '1' 'obsolete' 'plugin' 'module'`

110 - - Netdata sees the "obsolete" flag and knows to clean up the chart

111 -

112 -3. **No Dimension Values Sent**:

113 - - The framework still sends BEGIN/END for obsolete charts

114 - - But it sends SETEMPTY for all dimensions (no values)

115 - - This creates empty updates that signal the chart should be removed

116 -

117 -## Configuration Precedence and Auto-Detection

118 -

119 -For collectors that support multiple versions or editions of the monitored application, follow this critical principle:

120 -

121 -Admin configuration MUST always take precedence over auto-detection.

122 -

123 -1. **Version detection can be wrong**: Vendors backport features, custom builds exist

124 -2. **Admins know their environment**: They might have special configurations

125 -3. **Testing new versions**: Admins need to test features on versions we haven't validated

126 -4. **Enterprise flexibility**: Production environments often have unique requirements

127 -

128 -## Floating Point Precision Handling

129 -

130 -### Understanding Netdata's Precision System

131 -

132 -Netdata's Go collectors MUST send **integer values** to Netdata, but many metrics are naturally floating-point (percentages, response times, load averages, etc.). Netdata uses a precision system to handle this conversion while preserving decimal places in the database.

133 -

134 -**How It Works:**

135 -

136 -1. **Collection**: Get floating-point value from source

137 -2. **Multiply by precision**: Convert to integer for transmission

138 -3. **Send to Netdata**: Integer value via protocol

139 -4. **Chart definition**: Specify division to restore original value

140 -5. **Database storage**: Netdata stores the restored float value

141 -

142 -### Precision Best Practices

143 -

144 -1. **Use consistent precision**: Most Go collectors use `precision = 1000`

145 -2. **Apply precision once**: During collection phase only

146 -3. **Always divide by precision**: In chart definitions for float metrics

147 -4. **Separate concerns**: Unit conversion (`Mul`) and precision (`Div`) are independent

148 -5. **Test the math**: Verify the final value matches the original float

149 -

150 -## Missing Data is Data

151 -

152 -**CRITICAL**: Never fill gaps. For Netdata, the absence of data collection samples is crucial. Netdata visualizes gaps in data collection, marking missing points as empty. This happens automatically when collectors DO NOT SEND samples at the predefined collection interval. The Golden Rule: When data collection fails, DO NOT SEND ANY VALUE. Skip the metric entirely.

153 -

154 -The ONLY exception is for metrics derived from events (logs, message queues, etc.) where sparse data is expected.

155 -

156 -## Persistent Connections

157 -

158 -When possible Connect **ONCE** and maintain the connection for the collector's lifetime.

159 -

160 -When reconnection is necessary:

161 -1. **Always log the error** - Users need to know about connection issues

162 -2. **Implement backoff** - Don't hammer the target with reconnection attempts

163 -3. **Maintain connection state** - Track connection health

164 -

165 -## Minimize Application Impact

166 -

167 -**CRITICAL**: Reuse temporary objects between collection cycles. Don't create new temporary resources for each collection.

168 -

169 -## Ideal collector lifecycle

170 -

171 -1. **One connection per collector** - Not per collection cycle

172 -2. **One set of temporary resources** - Created at init, cleaned at shutdown

173 -3. **Minimal queries** - Batch requests when possible

174 -4. **Respect rate limits** - Don't overload the monitored application

175 -5. **Clean shutdown** - Always clean up resources in Cleanup()

176 -

177 -## Documentation

178 -

179 -### README.md Structure (Developer/User documentation)

180 -

181 -```markdown

182 -# Module name collector

183 -

184 -## Overview

185 -What this collector does and what it monitors.

186 -

187 -## Collected metrics

188 -List of all metrics with descriptions.

189 -

190 -## Configuration

191 -Configuration examples and options.

192 -

193 -## Requirements

194 -Any prerequisites or dependencies.

195 -

196 -## Troubleshooting

197 -Common issues and solutions.

198 -```

199 -

200 -### metadata.yaml Key Sections (source of truth for User Documentation)

201 -

202 -metadata.yaml drives the integrations list presented in Netdata dashboard and sites

203 -

204 -```yaml

205 -plugin_name: go.d.plugin

206 -modules:

207 - - meta:

208 - module_name: mymodule

209 - monitored_instance:

210 - name: My Service

211 - link: https://example.com

212 - categories:

213 - - data-collection.category

214 - icon_filename: "icon.svg"

215 - overview:

216 - data_collection:

217 - metrics_description: |

218 - Detailed description of what metrics are collected.

219 - method_description: |

220 - How the collector gathers these metrics.

221 - setup:

222 - prerequisites:

223 - list:

224 - - title: Requirement

225 - description: What needs to be done

226 - metrics:

227 - folding:

228 - title: Metrics

229 - enabled: false

230 - description: ""

231 - availability: []

232 - scopes:

233 - - name: global

234 - description: These metrics refer to the entire instance.

235 - labels: []

236 - metrics:

237 - - name: module.metric_name

238 - description: Metric description

239 - unit: units

240 - chart_type: line

241 - dimensions:

242 - - name: dimension1

243 - - name: dimension2

244 -```

245 -

246 -### Vnode Support

247 -

248 -Collectors MUST support vnode for configuration management:

249 -

250 -```go

251 -type Config struct {

252 - Vnode string `yaml:"vnode,omitempty" json:"vnode"`

253 - UpdateEvery int `yaml:"update_every,omitempty" json:"update_every"`

254 - // ... other fields

255 -}

256 -```

257 -

258 -## Alert Best Practices

259 -

260 -Alerts should be stores in `src/health/health.d/{module}.conf`

261 -

262 -Netdata uses alert templates that are automatically applied to a single instance (chart in the code).

263 -

264 -### Threshold Selection

265 -

266 -Netdata follows specific patterns for alert thresholds:

267 -

268 -1. **Avoid fixed thresholds** except for:

269 - - Error count metrics (e.g., deadlocks > 0)

270 - - Status/state metrics (e.g., backup failed = 1)

271 - - Long-running operations (e.g., queries > X seconds)

272 - - **Percentage-based metrics** (e.g., CPU usage %, memory usage %, disk usage %)

273 -

274 -2. **Preferred approaches**:

275 - - **Fixed thresholds on percentages**: Simple thresholds (e.g., > 80%, > 90%) work well for percentage metrics

276 - - **Rolling windows with predictions**: For non-percentage metrics, compare current values to predicted normal ranges

277 - - **Dynamic baselines**: Use historical data to establish normal behavior when percentages aren't available

278 - - **Percentile-based thresholds**: Adapt to workload patterns

279 - - **Rate of change detection**: Alert on sudden spikes or drops

280 -

281 -**Note**: The dynamic threshold rule primarily applies when we cannot express the metric as a percentage. For percentage-based metrics (0-100%), fixed thresholds are appropriate and preferred for their simplicity and clarity.

282 -

283 -### Alert Severity and Routing

284 -

285 -1. **Silent alerts**: Use `to: silent` for alerts that don't require immediate human action

286 - - Performance degradation that's not critical

287 - - Utilization approaching limits but not critical

288 - - Informational alerts for capacity planning

289 -

290 -2. **Non-silent alerts**: Only for issues requiring immediate action

291 - - Service failures

292 - - Critical resource exhaustion

293 - - Data corruption risks

294 - - Security breaches

295 -

296 -### Alert Configuration Structure

297 -

298 -```yaml

299 -template: collector_metric_condition

300 - on: collector.metric

301 - class: Utilization|Errors|Latency|Workload|Availability

302 - type: System|Database|Web Server|Application

303 -component: ServiceName

304 - lookup: average -5m unaligned of dimension # for time-based, the result in $this

305 - calc: $dimension * 100 / $total # for calculated metrics, can use $this, the result in $this

306 - units: %|ms|requests|errors

307 - every: 10s

308 - warn: # warning condition, can use $this

309 - crit: # critical condition, can use $this

310 - delay: down 5m multiplier 1.5 max 1h

311 - summary: Short description

312 - info: Detailed description with ${value} placeholder

313 - to: role|silent

314 -```

315 -

316 -## Charts design

317 -

318 -The collectors MUST always (when possible) add labels for the versions of the application. This enables users to filter by version, group metrics by version, and understand which features are available.

319 -

320 -1. **Dimensions of chart must be additive or comparable** - they should make sense when summed (exception: percentages, boolean statuses)

321 -2. **All dimensions must share the same units** - never mix bytes with milliseconds, or gauges with rates

322 -3. **Context names should describe what's measured** - `thread_pools.threads` clearly indicates thread counts

323 -4. **Group metrics that tell the same story** - min/current/max belong together when they have same units

324 -5. **Use chart families** to group related metrics

325 -6. **Calculate derived values** (like "other" = total - specific) to maintain additivity

326 -7. **Separate different measurement types** into different charts

327 -8. **Historical/peak values** should be in separate charts from current values

328 -9. **Gauges and rates must be in separate contexts** - point-in-time values vs incremental counters

329 -

330 -## Collect Everything Available

331 -

332 -**Netdata's distributed architecture enables ingestion of SIGNIFICANTLY more metrics than centralized systems. Collect EVERYTHING unless there's a compelling reason not to.**

333 -

334 -**DEFAULT: COLLECT EVERYTHING**

335 -

336 -Only exclude metrics when:

337 -1. **100% redundant** - Exact duplicate of another metric already collected

338 -2. **Completely useless** - Provides zero operational value (very rare)

339 -3. **Performance impact** - Collection significantly affects the monitored application

340 -4. **Resource intensive** - Metric calculation is expensive on the target system

341 -5. **Configuration constants** - Metrics that never or rarely change (although some are useful)

342 -

343 -## MOCK DATA

344 -

345 -Review the code for any mock data, remaining TODOs, frictional logic, frictional API endpoints, frictional response formats, frictional members, etc. An independent reality check **MUST** be performed.

346 -

347 -## LOGS

348 -

349 -Review all logs generated by the code to ensure users a) have enough information to understand the selections the code made, b) have descriptive and detailed logs on failures, c) they are not spammed by repeated logs on failures that are supposed to be permanent (like a not supported feature by the monitored application)

350 -

351 -## CONFIGURATION

352 -

353 -The code, the stock configuration, the metadata.yaml, the config_schema.json and any stock alerts **MUST** match 100%. Even the slightest variation between them in config keys, possible values, enum values, contexts, dimension names, etc LEADS TO A NON WORKING SOLUTION. So, at the end of every change, an independent sync check **MUST** be performed. While doing this work, ensure also README.md reflects the facts.

354 -

355 -## USE DOCUMENTATION AND INFORMATION

356 -

357 -metadata.yaml is the PRIMARY documentation source shown on the Netdata integrations page - ensure troubleshooting information, setup instructions, and all documentation is up-to-date in BOTH README.md AND metadata.yaml.

358 -

359 -## CRITICAL: Data Integrity Rules

360 -

361 -- **NEVER FAKE DATA COLLECTION VALUES!** YOU ARE NEVER ALLOWED TO SET DATA COLLECTION VALUES TO ZERO, OR ANY VALUE! THIS IS A DATA COLLECTION SYSTEM AND IT SHOULD REFLECT THE ACTUAL DATA COLLECTED VALUES! MISSING DATA = DATA! WHEN A VALUE IS MISSING, NETDATA CREATES GAPS ON THE CHARTS. THIS IS IMPORTANT INFORMATION FOR THE MONITORED APPLICATIONS AND SHOULD NEVER BE FILLED WITH ZEROS OR MOCK DATA!

362 -- When we create charts for arrays of items, we never use activity based filtering to control cardinality (e.g. create charts for the tables that read > X rows, create charts for the connections that transfer > X bytes, etc). But we can use activity based filtering when we count items (e.g. a metric with the number of queries running for more than X minutes).

363 -

364 -## CRITICAL: NIDL-Framework

365 -

366 -The file /docs/NIDL-Framework.md describes how Netdata metrcis, charts and dashboards work. You **MUST** read it before working with metrics in collectors.

367 -IMPORTANT: "chart" for go.d modules is "instance" for NIDL and "context" for go.d modules in "chart" for NIDL.

368 -

369 -

370 -## METADATA UPDATE

371 -

372 -### CONFIGURATION

373 -

374 -1. Build a list of all the configuration options supported by the module

375 -2. Verify that exactly these (no more, no less) configuration options are used:

376 - - in `config_schema.json`, so that dynamic configuration of the collector will work on the dashboards

377 - - in `metadata.yaml`, so that users can see them

378 - - in stock config, so that users editing the config can see the options

379 -

380 -### CONTEXTS/METRICS

381 -

382 -1. Build a list of all static and dynamic contexts, units, titles and dimensions the collector may collect

383 -2. Verify that exactly these contexts (no more, no less) are used:

384 - - in `metadata.yaml`, so that users can see them

385 - - in `README.md`, so that developers and users can see them

386 - - in any stock alerts

387 -

src/go/COLLECTOR-LIFECYCLE.md deleted

-1210

@@ -1,1210 +0,0 @@

1 -# go.d Collector Lifecycle Guide

2 -

3 -This document provides a comprehensive guide to the lifecycle and data flows of go.d framework collectors. It covers everything a new developer needs to understand how to build robust, efficient collectors that integrate seamlessly with Netdata.

4 -

5 -## Table of Contents

6 -

7 -1. [Overview](#overview)

8 -2. [Module Interface](#module-interface)

9 -3. [Job Lifecycle](#job-lifecycle)

10 -4. [Configuration Management](#configuration-management)

11 -5. [Chart Management](#chart-management)

12 -6. [Data Collection](#data-collection)

13 -7. [Connection Management](#connection-management)

14 -8. [Error Handling](#error-handling)

15 -9. [Performance Optimization](#performance-optimization)

16 -10. [Best Practices](#best-practices)

17 -

18 -## Overview

19 -

20 -The go.d framework provides a structured approach to data collection that ensures reliability, performance, and maintainability. Every collector follows a well-defined lifecycle managed by the job framework.

21 -

22 -### Core Principles

23 -

24 -1. **Missing Data is Data**: Never fake or cache values - gaps indicate real issues

25 -2. **Graceful Degradation**: Collect what you can, warn about what you can't

26 -3. **Minimal Impact**: Optimize for minimal resource usage on monitored systems

27 -4. **Admin Override**: User configuration always takes precedence over auto-detection

28 -

29 -## Module Interface

30 -

31 -Every collector must implement the `module.Module` interface:

32 -

33 -```go

34 -type Module interface {

35 - Init(context.Context) error

36 - Check(context.Context) error

37 - Charts() *Charts

38 - Collect(context.Context) map[string]int64

39 - Cleanup(context.Context)

40 - Configuration() any

41 -}

42 -```

43 -

44 -### Module Registration

45 -

46 -Register your module in the `init()` function:

47 -

48 -```go

49 -//go:embed "config_schema.json"

50 -var configSchema string

51 -

52 -func init() {

53 - module.Register("mymodule", module.Creator{

54 - JobConfigSchema: configSchema,

55 - Create: func() module.Module { return New() },

56 - Config: func() any { return &Config{} },

57 - })

58 -}

59 -```

60 -

61 -## Job Lifecycle

62 -

63 -### 1. Initialization Phase

64 -

65 -**Job Creation → Auto-Detection → Start**

66 -

67 -```mermaid

68 -graph TD

69 - A[Job Created] --> B[Init()]

70 - B --> C[Check()]

71 - C --> D[Charts()]

72 - D --> E[Start Collection Loop]

73 - B --> F[Init Failed]

74 - C --> G[Check Failed - Retry]

75 - F --> H[Job Disabled]

76 - G --> I{Retries Left?}

77 - I -->|Yes| C

78 - I -->|No| H

79 -```

80 -

81 -### Auto-Detection Process

82 -

83 -```go

84 -func (j *Job) AutoDetection() error {

85 - // 1. Initialize module

86 - if err := j.init(); err != nil {

87 - return err // Fatal - job disabled

88 - }

89 -

90 - // 2. Check connectivity/validity

91 - if err := j.check(); err != nil {

92 - return err // Retryable - decrements AutoDetectTries

93 - }

94 -

95 - // 3. Validate charts

96 - if err := j.postCheck(); err != nil {

97 - return err // Fatal - job disabled

98 - }

99 -

100 - return nil

101 -}

102 -```

103 -

104 -**Auto-Detection Behavior:**

105 -- **Stock jobs**: Muted during auto-detection to reduce noise

106 -- **Custom jobs**: Normal logging throughout

107 -- **Retry mechanism**: Controlled by `AutoDetectEvery` and `AutoDetectTries`

108 -- **Panic recovery**: Automatic with stack traces in debug mode

109 -

110 -### 2. Collection Phase

111 -

112 -**Continuous Collection Loop with Penalty System**

113 -

114 -```go

115 -func (j *Job) Start() {

116 - for {

117 - select {

118 - case <-j.stop:

119 - break

120 - case t := <-j.tick:

121 - // Apply penalty for failed collections

122 - if t%(j.updateEvery+j.penalty()) == 0 {

123 - j.runOnce()

124 - }

125 - }

126 - }

127 -}

128 -```

129 -

130 -**Penalty System:**

131 -- **Progressive backoff**: Failed collections increase collection interval

132 -- **Maximum penalty**: 600 seconds (10 minutes)

133 -- **Recovery**: Successful collections reset penalty

134 -

135 -### 3. Cleanup Phase

136 -

137 -**Graceful Shutdown with Resource Cleanup**

138 -

139 -```go

140 -func (j *Job) Cleanup() {

141 - // Mark charts as obsolete

142 - for _, chart := range *j.charts {

143 - if chart.created {

144 - chart.MarkRemove()

145 - j.createChart(chart) // Send CHART command with "obsolete" flag

146 - }

147 - }

148 -}

149 -```

150 -

151 -## Configuration Management

152 -

153 -### Configuration Structure

154 -

155 -```go

156 -type Config struct {

157 - Vnode string `yaml:"vnode,omitempty" json:"vnode"`

158 - UpdateEvery int `yaml:"update_every,omitempty" json:"update_every"`

159 -

160 - // Connection configuration

161 - DSN string `yaml:"dsn" json:"dsn"`

162 - Timeout confopt.Duration `yaml:"timeout,omitempty" json:"timeout"`

163 -

164 - // Feature flags

165 - CollectAdvancedMetrics *bool `yaml:"collect_advanced_metrics" json:"collect_advanced_metrics"`

166 -

167 - // Filtering

168 - InstanceSelector matcher.SimpleExpr `yaml:"instance_selector,omitempty" json:"instance_selector"`

169 -}

170 -```

171 -

172 -### Configuration Precedence

173 -

174 -**CRITICAL**: Admin configuration always overrides auto-detection:

175 -

176 -```go

177 -func (c *Collector) Init() {

178 - // 1. Detect capabilities

179 - c.detectVersion()

180 -

181 - // 2. Set defaults ONLY if admin hasn't configured

182 - if c.Config.CollectAdvancedFeatures == nil {

183 - defaultValue := c.supportsAdvancedFeatures()

184 - c.Config.CollectAdvancedFeatures = &defaultValue

185 - }

186 -}

187 -

188 -func (c *Collector) collect() {

189 - // 3. Always attempt what admin configured

190 - if *c.Config.CollectAdvancedFeatures {

191 - if err := c.collectAdvancedFeatures(); err != nil {

192 - c.Warningf("Advanced features failed: %v", err)

193 - // Continue collection - don't fail entire job

194 - }

195 - }

196 -}

197 -```

198 -

199 -### Validation Patterns

200 -

201 -```go

202 -func (c *Collector) Init(context.Context) error {

203 - if err := c.validateConfig(); err != nil {

204 - return fmt.Errorf("config validation: %v", err) // Fatal error

205 - }

206 -

207 - // Initialize expensive resources once

208 - if err := c.initClient(); err != nil {

209 - return fmt.Errorf("init client: %v", err)

210 - }

211 -

212 - return nil

213 -}

214 -

215 -func (c *Collector) validateConfig() error {

216 - if c.URL == "" {

217 - return errors.New("url is required")

218 - }

219 - return nil

220 -}

221 -```

222 -

223 -## Chart Management

224 -

225 -### Static Charts (Always Present)

226 -

227 -```go

228 -var baseCharts = module.Charts{

229 - {

230 - ID: "cpu_usage",

231 - Title: "CPU Usage",

232 - Units: "percentage",

233 - Fam: "cpu",

234 - Ctx: "mymodule.cpu_usage",

235 - Priority: prioCPUUsage,

236 - Type: module.Stacked,

237 - Dims: module.Dims{

238 - {ID: "user", Name: "user"},

239 - {ID: "system", Name: "system"},

240 - },

241 - },

242 -}

243 -

244 -func (c *Collector) Charts() *module.Charts {

245 - return c.charts // Return base charts

246 -}

247 -```

248 -

249 -### Dynamic Charts (Instance-Based)

250 -

251 -```go

252 -func newInstanceCharts(instanceName string) *module.Charts {

253 - charts := instanceChartsTmpl.Copy()

254 -

255 - for _, chart := range *charts {

256 - chart.ID = fmt.Sprintf(chart.ID, cleanName(instanceName))

257 - chart.Labels = []module.Label{

258 - {Key: "instance", Value: instanceName},

259 - }

260 - for _, dim := range chart.Dims {

261 - dim.ID = fmt.Sprintf(dim.ID, cleanName(instanceName))

262 - }

263 - }

264 -

265 - return charts

266 -}

267 -```

268 -

269 -### Chart Lifecycle Management

270 -

271 -**Basic Pattern (Simple boolean tracking):**

272 -

273 -```go

274 -func (c *Collector) collectInstances(mx map[string]int64) {

275 - seen := make(map[string]bool)

276 -

277 - for _, instance := range getInstances() {

278 - seen[instance.Name] = true

279 -

280 - // Add charts for new instances

281 - if !c.collected[instance.Name] {

282 - c.collected[instance.Name] = true

283 - charts := newInstanceCharts(instance.Name)

284 - c.charts.Add(*charts...)

285 - }

286 -

287 - // Collect metrics

288 - collectInstanceMetrics(instance, mx)

289 - }

290 -

291 - // Remove stale instances

292 - for name := range c.collected {

293 - if !seen[name] {

294 - delete(c.collected, name)

295 - c.removeInstanceCharts(name)

296 - }

297 - }

298 -}

299 -```

300 -

301 -**Advanced Pattern (Staleness detection with tolerance):**

302 -

303 -```go

304 -type cacheEntry struct {

305 - seen bool

306 - notSeenTimes int // Track consecutive misses

307 - charts []*module.Chart

308 -}

309 -

310 -func (c *Collector) collectInstances(mx map[string]int64) {

311 - // Reset seen flags

312 - for _, entry := range c.cache.entries {

313 - entry.seen = false

314 - }

315 -

316 - for _, instance := range getInstances() {

317 - entry := c.cache.entries[instance.Name]

318 - if entry == nil {

319 - entry = &cacheEntry{charts: newInstanceCharts(instance.Name)}

320 - c.cache.entries[instance.Name] = entry

321 - c.charts.Add(*entry.charts...)

322 - }

323 -

324 - entry.seen = true

325 - entry.notSeenTimes = 0

326 - collectInstanceMetrics(instance, mx)

327 - }

328 -

329 - // Remove stale instances with tolerance

330 - for name, entry := range c.cache.entries {

331 - if entry.seen {

332 - continue

333 - }

334 -

335 - // Allow N missed collections before removal

336 - if entry.notSeenTimes++; entry.notSeenTimes >= maxNotSeenTimes {

337 - for _, chart := range entry.charts {

338 - chart.MarkRemove()

339 - chart.MarkNotCreated()

340 - }

341 - delete(c.cache.entries, name)

342 - }

343 - }

344 -}

345 -```

346 -

347 -### Chart Obsolescence

348 -

349 -```go

350 -func (c *Collector) removeInstanceCharts(instanceName string) {

351 - cleanName := cleanName(instanceName)

352 -

353 - for _, chart := range *c.charts {

354 - if strings.HasPrefix(chart.ID, fmt.Sprintf("instance_%s_", cleanName)) {

355 - if !chart.Obsolete {

356 - chart.Obsolete = true

357 - chart.MarkNotCreated() // Trigger CHART command with obsolete flag

358 - c.Debugf("Marked chart %s as obsolete", chart.ID)

359 - }

360 - }

361 - }

362 -}

363 -```

364 -

365 -### Dimension ID Management

366 -

367 -**CRITICAL**: Understanding dimension IDs vs Names:

368 -

369 -1. **Dimension IDs must be unique across the entire job** (all charts, all contexts)

370 -2. **The framework sends dimension Names as both ID and name to Netdata**

371 -3. **NIDL compliance is achieved through shared dimension Names, not IDs**

372 -

373 -```go

374 -// For dynamic instances - unique dimension IDs required

375 -chart.Dims = append(chart.Dims, &module.Dim{

376 - ID: "thread_pool_instance1_active", // Must be unique across job

377 - Name: "active", // This is what Netdata sees

378 -})

379 -

380 -// In your mx map - must match dim.ID exactly

381 -mx["thread_pool_instance1_active"] = 42

382 -```

383 -

384 -### Advanced Chart Creation Patterns

385 -

386 -**Configuration-based chart modification:**

387 -

388 -```go

389 -func (c *Collector) addContainerCharts(name, image string) {

390 - charts := containerChartsTmpl.Copy()

391 -

392 - // Remove charts based on configuration

393 - if !c.CollectContainerSize {

394 - _ = charts.Remove(containerWritableLayerSizeChartTmpl.ID)

395 - }

396 -

397 - if !c.CollectContainerHealth {

398 - _ = charts.Remove(containerHealthStatusChartTmpl.ID)

399 - }

400 -

401 - // Modify charts for specific container types

402 - if strings.Contains(image, "database") {

403 - // Add database-specific dimensions

404 - for _, chart := range *charts {

405 - if chart.ID == "container_cpu" {

406 - chart.Dims = append(chart.Dims,

407 - &module.Dim{ID: "db_cpu_time", Name: "database"})

408 - }

409 - }

410 - }

411 -

412 - c.charts.Add(*charts...)

413 -}

414 -```

415 -

416 -**Multiple metric type handling:**

417 -

418 -```go

419 -func (c *Collector) collectPrometheusMetrics(mx map[string]int64) {

420 - for _, mf := range c.metricFamilies {

421 - switch mf.Type() {

422 - case model.MetricTypeGauge:

423 - c.collectGauge(mx, mf)

424 - case model.MetricTypeCounter:

425 - c.collectCounter(mx, mf)

426 - case model.MetricTypeSummary:

427 - c.collectSummary(mx, mf)

428 - case model.MetricTypeHistogram:

429 - c.collectHistogram(mx, mf)

430 - case model.MetricTypeUnknown:

431 - // Fallback type detection

432 - if c.isFallbackTypeGauge(mf.Name()) {

433 - c.collectGauge(mx, mf)

434 - } else if c.isFallbackTypeCounter(mf.Name()) ||

435 - strings.HasSuffix(mf.Name(), "_total") {

436 - c.collectCounter(mx, mf)

437 - }

438 - }

439 - }

440 -}

441 -```

442 -

443 -**Hard limits (NOT RECOMMENDED):**

444 -

445 -Some collectors implement hard limits, but this approach has significant drawbacks:

446 -

447 -```go

448 -// PROBLEMATIC: All-or-nothing approach

449 -func (c *Collector) collectWithHardLimits(mx map[string]int64) {

450 - for _, metric := range getAllMetrics() {

451 - // This skips the ENTIRE metric family if over limit

452 - if c.MaxTSPerMetric > 0 && len(metric.TimeSeries()) > c.MaxTSPerMetric {

453 - c.Debugf("metric '%s' num of time series (%d) > limit (%d), skipping",

454 - metric.Name(), len(metric.TimeSeries()), c.MaxTSPerMetric)

455 - continue // Loses ALL data for this metric!

456 - }

457 - }

458 -}

459 -```

460 -

461 -**Why hard limits are problematic:**

462 -- If limit is 100 and metric has 110 time series, you get 0 instead of 100

463 -- No way to determine which instances are "important"

464 -- Unpredictable data loss from user perspective

465 -- Better to use selector-based filtering (see Cardinality Protection section)

466 -

467 -## Data Collection

468 -

469 -### Collection Return Values

470 -

471 -**Different return scenarios and their meanings:**

472 -

473 -```go

474 -func (c *Collector) Collect(context.Context) map[string]int64 {

475 - mx, err := c.collect()

476 - if err != nil {

477 - c.Error(err) // Log error

478 - }

479 -

480 - if len(mx) == 0 {

481 - return nil // Complete failure - shows gaps

482 - }

483 -

484 - return mx // Partial or complete success

485 -}

486 -```

487 -

488 -**Return Value Guidelines:**

489 -- **`nil`**: Complete collection failure - creates gaps

490 -- **Empty map**: No data available - creates gaps

491 -- **Partial map**: Some metrics missing - missing dimensions get `SETEMPTY`

492 -- **Complete map**: All metrics collected successfully

493 -

494 -### Gap Handling Philosophy

495 -

496 -**NEVER cache or fake data - missing data is meaningful:**

497 -

498 -```go

499 -// CORRECT: Only send metrics actually collected

500 -func (c *Collector) collect() map[string]int64 {

501 - mx := make(map[string]int64)

502 -

503 - currentMetrics := c.fetchFromApplication()

504 - for name, value := range currentMetrics {

505 - mx[name] = value // Only real values

506 - }

507 -

508 - return mx // Gaps will show if metrics missing

509 -}

510 -

511 -// WRONG: Never cache old values

512 -func (c *Collector) collect() map[string]int64 {

513 - mx := make(map[string]int64)

514 -

515 - currentMetrics := c.fetchFromApplication()

516 - if len(currentMetrics) == 0 {

517 - // NEVER DO THIS!

518 - mx["metric"] = c.lastValue // Hides real problems

519 - }

520 -

521 - return mx

522 -}

523 -```

524 -

525 -### Partial Collection Patterns

526 -

527 -```go

528 -func (c *Collector) collect() (map[string]int64, error) {

529 - mx := make(map[string]int64)

530 -

531 - // Core metrics - must succeed

532 - if err := c.collectCoreMetrics(mx); err != nil {

533 - return nil, fmt.Errorf("core metrics failed: %v", err)

534 - }

535 -

536 - // Optional metrics - failures are warnings

537 - if c.Config.CollectAdvanced {

538 - if err := c.collectAdvancedMetrics(mx); err != nil {

539 - c.Warningf("advanced metrics failed: %v", err)

540 - // Continue with core metrics

541 - }

542 - }

543 -

544 - return mx, nil

545 -}

546 -```

547 -

548 -### Floating Point Precision

549 -

550 -**Basic precision handling:**

551 -

552 -```go

553 -const precision = 1000

554 -

555 -// Collection: Convert float to int64

556 -floatValue := 1.345

557 -mx["metric"] = int64(floatValue * precision) // 1345

558 -

559 -// Chart definition: Restore precision

560 -{

561 - Dims: module.Dims{

562 - {ID: "metric", Name: "value", Div: precision}, // 1345 / 1000 = 1.345

563 - },

564 -}

565 -```

566 -

567 -**Advanced precision for different metric types:**

568 -

569 -```go

570 -func (c *Collector) collectPrometheusMetrics(mx map[string]int64) {

571 - for _, mf := range c.metricFamilies {

572 - switch mf.Type() {

573 - case model.MetricTypeGauge:

574 - // Standard precision for gauges

575 - mx[id] = int64(mf.Gauge().Value() * precision)

576 -

577 - case model.MetricTypeSummary:

578 - // Double precision for quantiles (more granular)

579 - for _, q := range mf.Summary().Quantile() {

580 - dimID := fmt.Sprintf("%s_quantile_%s", id, formatQuantile(q.Quantile()))

581 - mx[dimID] = int64(q.Value() * precision * precision)

582 - }

583 -

584 - case model.MetricTypeHistogram:

585 - // Different precision for different bucket types

586 - for _, bucket := range mf.Histogram().Bucket() {

587 - bucketID := fmt.Sprintf("%s_bucket_%s", id, formatBound(bucket.UpperBound()))

588 - mx[bucketID] = int64(bucket.CumulativeCount()) // No precision - count is integer

589 - }

590 - }

591 - }

592 -}

593 -```

594 -

595 -### Cardinality Protection

596 -

597 -**Best Practice: Selector-Based Filtering**

598 -

599 -The recommended approach for cardinality protection is **selective monitoring using regular expressions** rather than hard limits. This pattern is already widely used in production go.d collectors.

600 -

601 -**Common Implementations:**

602 -

603 -1. **Simple Pattern Matcher** (most common):

604 -```go

605 -type Config struct {

606 - // Empty by default - monitor everything

607 - // Users explicitly define what's important to them

608 - InstanceSelector string `yaml:"instance_selector,omitempty"`

609 -}

610 -

611 -func (c *Collector) Init() error {

612 - if c.InstanceSelector != "" {

613 - // Supports glob patterns: *, ?, and exclusions with !

614 - matcher, err := matcher.NewSimplePatternsMatcher(c.InstanceSelector)

615 - if err != nil {

616 - return fmt.Errorf("invalid instance selector: %w", err)

617 - }

618 - c.instanceMatcher = matcher

619 - }

620 - // By default, no filter - collect everything

621 -}

622 -

623 -func (c *Collector) collectInstances() {

624 - for _, instance := range getAllInstances() {

625 - // Skip if selector is defined and doesn't match

626 - if c.instanceMatcher != nil && !c.instanceMatcher.MatchString(instance.Name) {

627 - continue

628 - }

629 -

630 - // Collect the instance

631 - c.collectInstance(instance)

632 - }

633 -}

634 -```

635 -

636 -2. **Include/Exclude Arrays** (MongoDB style):

637 -```go

638 -type Config struct {

639 - Databases matcher.SimpleExpr `yaml:"databases,omitempty"`

640 -}

641 -

642 -// Configuration:

643 -// databases:

644 -// includes:

645 -// - "prod_*"

646 -// - "staging_*"

647 -// excludes:

648 -// - "*_test"

649 -```

650 -

651 -3. **Regular Expression** (MQ PCF style):

652 -```go

653 -type Config struct {

654 - QueueSelector string `yaml:"queue_selector"`

655 -}

656 -

657 -func (c *Collector) Init() error {

658 - if c.QueueSelector != "" {

659 - c.queueSelectorRegex, err = regexp.Compile(c.QueueSelector)

660 - if err != nil {

661 - return fmt.Errorf("invalid queue_selector regex: %w", err)

662 - }

663 - }

664 -}

665 -```

666 -

667 -**Real-World Examples:**

668 -- **Docker**: `container_selector` to filter containers

669 -- **PostgreSQL**: `collect_databases_matching` for database selection

670 -- **MongoDB**: `databases` include/exclude arrays

671 -- **DB2**: Multiple selectors for different object types

672 -- **MQ PCF**: Regex-based queue and channel selectors

673 -- **VSphere**: Hierarchical path matching for hosts/VMs

674 -

675 -**Why This Approach?**

676 -1. **User Control**: Users explicitly choose what to monitor

677 -2. **Predictable**: No surprises about what gets collected

678 -3. **Flexible**: Supports glob patterns, regex, or include/exclude lists

679 -4. **Battle-tested**: Already proven in production collectors

680 -5. **No Arbitrary Decisions**: Avoids "which 100 out of 110?" problem

681 -

682 -**Configuration Examples:**

683 -```yaml

684 -# Simple patterns (glob-style)

685 -- name: postgres_prod

686 - dsn: postgresql://localhost

687 - collect_databases_matching: "prod_* staging_* !*_test"

688 -

689 -# Regular expressions

690 -- name: mq_queues

691 - queue_selector: "^(PROD|STAGE)\\..*"

692 -

693 -# Include/Exclude arrays

694 -- name: mongodb

695 - databases:

696 - includes:

697 - - "myapp_*"

698 - - "analytics_*"

699 - excludes:

700 - - "*_backup"

701 - - "*_temp"

702 -```

703 -

704 -### Advanced Filtering Patterns

705 -

706 -**Label-based filtering:**

707 -

708 -```go

709 -func (c *Collector) shouldCollectContainer(container containerInfo) bool {

710 - // Check ignore labels

711 - if container.Labels["netdata.cloud/ignore"] == "true" {

712 - return false

713 - }

714 -

715 - // Check namespace filtering

716 - if c.Config.NamespaceSelector != "" {

717 - if !c.namespaceSelector.MatchString(container.Namespace) {

718 - return false

719 - }

720 - }

721 -

722 - // Check image filtering

723 - if c.Config.ImageSelector != "" {

724 - if !c.imageSelector.MatchString(container.Image) {

725 - return false

726 - }

727 - }

728 -

729 - return true

730 -}

731 -```

732 -

733 -**Version-based feature detection:**

734 -

735 -```go

736 -func (c *Collector) collectVersionSpecificMetrics(mx map[string]int64) {

737 - if c.systemdVersion >= 230 {

738 - // Feature available in systemd 230+

739 - units, err := c.getLoadedUnitsByPatterns(conn)

740 - if err != nil {

741 - c.Warningf("failed to get units by patterns: %v", err)

742 - return

743 - }

744 - // Process newer API results

745 - } else {

746 - // Fallback for older versions

747 - units, err := c.getLoadedUnits(conn)

748 - if err != nil {

749 - c.Warningf("failed to get units: %v", err)

750 - return

751 - }

752 - // Process older API results

753 - }

754 -}

755 -```

756 -

757 -## Connection Management

758 -

759 -### Connection Establishment Patterns

760 -

761 -**HTTP-based collectors - Create in Init():**

762 -

763 -```go

764 -func (c *Collector) Init(context.Context) error {

765 - httpClient, err := web.NewHTTPClient(c.ClientConfig)

766 - if err != nil {

767 - return fmt.Errorf("failed initializing http client: %w", err)

768 - }

769 - c.httpClient = httpClient

770 - return nil

771 -}

772 -```

773 -

774 -**Database collectors - Lazy initialization in collect():**

775 -

776 -```go

777 -func (c *Collector) collect() (map[string]int64, error) {

778 - if c.db == nil {

779 - if err := c.openConnection(); err != nil {

780 - return nil, err

781 - }

782 - }

783 -

784 - return c.queryMetrics()

785 -}

786 -

787 -func (c *Collector) openConnection() error {

788 - db, err := sql.Open("postgres", c.DSN)

789 - if err != nil {

790 - return fmt.Errorf("error opening connection: %v", err)

791 - }

792 -

793 - // Configure connection pool

794 - db.SetMaxOpenConns(1)

795 - db.SetMaxIdleConns(1)

796 - db.SetConnMaxLifetime(10 * time.Minute)

797 -

798 - // CRITICAL: Always test connection

799 - ctx, cancel := context.WithTimeout(context.Background(), c.Timeout.Duration())

800 - defer cancel()

801 -

802 - if err := db.PingContext(ctx); err != nil {

803 - _ = db.Close() // Clean up failed connection

804 - return fmt.Errorf("connection test failed: %v", err)

805 - }

806 -

807 - c.db = db

808 - return nil

809 -}

810 -```

811 -

812 -**Non-persistent connections (for specific use cases):**

813 -

814 -```go

815 -func (c *Collector) collect() (map[string]int64, error) {

816 - // Create client each time - for APIs that don't benefit from persistence

817 - if c.client == nil {

818 - client, err := c.newClient(c.Config)

819 - if err != nil {

820 - return nil, err

821 - }

822 - c.client = client

823 - }

824 -

825 - // Close after use - not persistent for Docker/container APIs

826 - defer func() { _ = c.client.Close() }()

827 -

828 - // One-time setup operations

829 - if !c.verNegotiated {

830 - c.verNegotiated = true

831 - c.negotiateAPIVersion()

832 - }

833 -

834 - return c.queryMetrics()

835 -}

836 -```

837 -

838 -### Context-Aware Timeout Management

839 -

840 -```go

841 -func (c *Collector) collectWithTimeouts(mx map[string]int64) error {

842 - // Different timeouts for different operations

843 -

844 - // Quick operations

845 - ctx, cancel := context.WithTimeout(context.Background(), time.Second*2)

846 - defer cancel()

847 -

848 - c.Debugf("calling function 'ListUnits'")

849 - units, err := c.conn.ListUnitsContext(ctx)

850 - if err != nil {

851 - return fmt.Errorf("failed to list units: %v", err)

852 - }

853 -

854 - // Longer timeout for expensive operations

855 - ctx, cancel = context.WithTimeout(context.Background(), c.Timeout.Duration())

856 - defer cancel()

857 -

858 - c.Debugf("calling function 'GetUnitProperties'")

859 - for _, unit := range units {

860 - props, err := c.conn.GetUnitPropertiesContext(ctx, unit.Name)

861 - if err != nil {

862 - c.Warningf("failed to get properties for %s: %v", unit.Name, err)

863 - continue // Skip this unit, continue with others

864 - }

865 -

866 - c.collectUnitProperties(mx, unit.Name, props)

867 - }

868 -

869 - return nil

870 -}

871 -```

872 -

873 -### Reconnection Strategies

874 -

875 -```go

876 -func (c *Collector) collect() (map[string]int64, error) {

877 - // Try with existing connection

878 - if err := c.ping(); err != nil {

879 - c.Warningf("connection failed, attempting reconnection: %v", err)

880 -

881 - // Clean up and reconnect

882 - c.Cleanup(context.TODO())

883 - if err := c.openConnection(); err != nil {

884 - return nil, err

885 - }

886 -

887 - // Retry after reconnection

888 - if err := c.ping(); err != nil {

889 - return nil, err

890 - }

891 - }

892 -

893 - return c.queryMetrics()

894 -}

895 -```

896 -

897 -### Resource Cleanup

898 -

899 -```go

900 -func (c *Collector) Cleanup(context.Context) {

901 - if c.db != nil {

902 - if err := c.db.Close(); err != nil {

903 - c.Warningf("cleanup error: %v", err)

904 - }

905 - c.db = nil

906 - }

907 -

908 - if c.httpClient != nil {

909 - c.httpClient.CloseIdleConnections()

910 - }

911 -}

912 -```

913 -

914 -## Error Handling

915 -

916 -### Error Hierarchy

917 -

918 -**Init Errors (Fatal - Job Disabled):**

919 -

920 -```go

921 -func (c *Collector) Init(context.Context) error {

922 - if err := c.validateConfig(); err != nil {

923 - return fmt.Errorf("config validation: %v", err) // Fatal

924 - }

925 - return nil

926 -}

927 -```

928 -

929 -**Check Errors (Retryable during Auto-Detection):**

930 -

931 -```go

932 -func (c *Collector) Check(context.Context) error {

933 - mx, err := c.collect()

934 - if err != nil {

935 - return err // Retryable - decrements AutoDetectTries

936 - }

937 -

938 - if len(mx) == 0 {

939 - return errors.New("no metrics collected")

940 - }

941 -

942 - return nil

943 -}

944 -```

945 -

946 -**Collect Errors (Graceful - Shows Gaps):**

947 -

948 -```go

949 -func (c *Collector) Collect(context.Context) map[string]int64 {

950 - mx, err := c.collect()

951 - if err != nil {

952 - c.Error(err) // Log but don't fail job

953 - }

954 -

955 - if len(mx) == 0 {

956 - return nil // Show gaps

957 - }

958 -

959 - return mx

960 -}

961 -```

962 -

963 -### Logging Patterns

964 -

965 -**Use appropriate log levels:**

966 -

967 -```go

968 -// Error: Failures preventing collection

969 -c.Errorf("database connection failed: %v", err)

970 -

971 -// Warning: Features not available or partial failures

972 -c.Warningf("advanced metrics not available: %v", err)

973 -

974 -// Info: Important lifecycle events

975 -c.Info("check success")

976 -c.Infof("started, collection interval %ds", interval)

977 -

978 -// Debug: Detailed troubleshooting information

979 -c.Debugf("query completed in %v", duration)

980 -```

981 -

982 -### Panic Recovery

983 -

984 -The framework automatically handles panics with recovery and stack traces:

985 -

986 -```go

987 -// Framework provides automatic panic recovery

988 -func (j *Job) collect() (result map[string]int64) {

989 - defer func() {

990 - if r := recover(); r != nil {

991 - j.panicked = true

992 - j.Errorf("PANIC: %v", r)

993 - if logger.Level.Enabled(slog.LevelDebug) {

994 - j.Errorf("STACK: %s", debug.Stack())

995 - }

996 - }

997 - }()

998 -

999 - result = j.module.Collect(context.TODO())

1000 - return result

1001 -}

1002 -```

1003 -

1004 -## Performance Optimization

1005 -

1006 -### Resource Creation Strategy

1007 -

1008 -**Create Once in Init():**

1009 -

1010 -```go

1011 -func (c *Collector) Init(context.Context) error {

1012 - // Expensive one-time operations

1013 - c.httpClient = web.NewHTTPClient(c.ClientConfig)

1014 - c.selector, _ = matcher.NewSimpleExprMatcher(c.Selector)

1015 - c.responseRegex = regexp.MustCompile(c.ResponsePattern)

1016 - return nil

1017 -}

1018 -```

1019 -

1020 -**Reuse in collect():**

1021 -

1022 -```go

1023 -func (c *Collector) collect() (map[string]int64, error) {

1024 - // Only fresh data collection - reuse expensive objects

1025 - req, _ := web.NewHTTPRequest(c.RequestConfig)

1026 - resp, err := c.httpClient.Do(req) // Reuse client

1027 - // ...

1028 -}

1029 -```

1030 -

1031 -### Connection Optimization

1032 -

1033 -```go

1034 -// Single connection per collector

1035 -db.SetMaxOpenConns(1)

1036 -db.SetMaxIdleConns(1)

1037 -db.SetConnMaxLifetime(10 * time.Minute)

1038 -

1039 -// Reuse temporary resources

1040 -func (c *Collector) Init() error {

1041 - c.responseQueue = c.mq.CreateTemporaryQueue() // Once

1042 - return nil

1043 -}

1044 -

1045 -func (c *Collector) collect() {

1046 - response := c.mq.QueryMetrics(c.responseQueue) // Reuse

1047 -}

1048 -```

1049 -

1050 -### Caching Strategies

1051 -

1052 -```go

1053 -type Collector struct {

1054 - recheckSettingsTime time.Time

1055 - recheckSettingsEvery time.Duration

1056 - addChartsOnce *sync.Once

1057 -}

1058 -

1059 -func (c *Collector) collect() {

1060 - // Cache expensive queries with time-based invalidation

1061 - if time.Since(c.recheckSettingsTime) > c.recheckSettingsEvery {

1062 - c.recheckSettingsTime = time.Now()

1063 - c.recheckExpensiveSettings()

1064 - }

1065 -

1066 - // Add charts only once when feature becomes available

1067 - if c.featureDetected {

1068 - c.addChartsOnce.Do(func() {

1069 - c.charts.Add(*newFeatureCharts()...)

1070 - })

1071 - }

1072 -}

1073 -```

1074 -

1075 -## Best Practices

1076 -

1077 -### Module Structure

1078 -

1079 -**Basic structure:**

1080 -

1081 -```go

1082 -type Collector struct {

1083 - module.Base

1084 - Config `yaml:",inline" json:""`

1085 -

1086 - charts *module.Charts

1087 -

1088 - // Connections

1089 - client someClient

1090 - db *sql.DB

1091 -

1092 - // State tracking

1093 - collected map[string]bool

1094 - seen map[string]bool

1095 -

1096 - // Feature detection

1097 - version string

1098 - addFeatureChartsOnce *sync.Once

1099 -}

1100 -```

1101 -

1102 -**Advanced structure with comprehensive patterns:**

1103 -

1104 -```go

1105 -type Collector struct {

1106 - module.Base

1107 - Config `yaml:",inline" json:""`

1108 -

1109 - charts *module.Charts

1110 -

1111 - // Connection management

1112 - client someClient

1113 - db *sql.DB

1114 - connRetries int

1115 -

1116 - // Advanced instance tracking

1117 - cache struct {

1118 - entries map[string]*cacheEntry

1119 - maxAge time.Duration

1120 - }

1121 -

1122 - // Version and feature detection

1123 - version string

1124 - systemdVersion int

1125 - detectedFeatures map[string]bool

1126 - verNegotiated bool

1127 -

1128 - // Conditional chart addition

1129 - addFeatureChartsOnce *sync.Once

1130 - addExtendedChartsOnce *sync.Once

1131 - addAdvancedChartsOnce *sync.Once

1132 -

1133 - // Cardinality control

1134 - maxInstances int

1135 - maxTSPerMetric int

1136 -

1137 - // Filtering and selection

1138 - namespaceSelector matcher.Matcher

1139 - imageSelector matcher.Matcher

1140 -

1141 - // Periodic operations

1142 - recheckTime time.Time

1143 - recheckInterval time.Duration

1144 -

1145 - // Performance tracking

1146 - lastCollectionTime time.Time

1147 - collectionDuration time.Duration

1148 -}

1149 -```

1150 -

1151 -### Configuration Best Practices

1152 -

1153 -1. **Use pointer fields for tri-state configuration** (`*bool` instead of `bool`)

1154 -2. **Provide sensible defaults** in the constructor

1155 -3. **Always validate configuration** in `Init()`

1156 -4. **Support admin override** of auto-detection

1157 -

1158 -### Chart Design Rules

1159 -

1160 -1. **Non-overlapping dimensions**: Dimensions must be additive or comparable

1161 -2. **Appropriate chart types**: Stacked for volume, Area for bidirectional, Line for independent

1162 -3. **Dynamic creation**: Only create charts when metrics are available

1163 -4. **Proper obsolescence**: Mark charts obsolete when instances disappear

1164 -

1165 -### Data Collection Rules

1166 -

1167 -1. **Never fake data**: Missing data creates meaningful gaps

1168 -2. **Graceful degradation**: Collect what you can, warn about failures

1169 -3. **Proper precision**: Use consistent precision multipliers for floats

1170 -4. **Respect gaps**: Don't cache old values to fill missing data

1171 -5. **Advanced precision**: Use different precision for different metric types (gauges vs quantiles)

1172 -6. **Cardinality protection**: Use selector-based filtering, not hard limits

1173 -7. **Label-based filtering**: Support application-specific ignore patterns

1174 -

1175 -### Connection Management Rules

1176 -

1177 -1. **Persistent connections**: Create once, reuse throughout lifecycle (preferred)

1178 -2. **Non-persistent connections**: Use for APIs that don't benefit from persistence

1179 -3. **Proper validation**: Always test connections before storing

1180 -4. **Automatic recovery**: Recreate failed connections in collect()

1181 -5. **Clean shutdown**: Always implement proper cleanup

1182 -6. **Context-aware timeouts**: Use different timeouts for different operations

1183 -7. **API version negotiation**: Handle version-specific features gracefully

1184 -

1185 -### Error Handling Rules

1186 -

1187 -1. **Appropriate log levels**: Error for failures, Warning for missing features

1188 -2. **Meaningful messages**: Include context about what failed and why

1189 -3. **Graceful failures**: Continue with partial data when possible

1190 -4. **Admin respect**: Always attempt configured features

1191 -5. **Timeout handling**: Use context with appropriate timeouts

1192 -6. **Version-based fallbacks**: Implement fallback logic for older versions

1193 -

1194 -### Advanced Patterns

1195 -

1196 -1. **Staleness detection**: Use tolerance-based removal for unstable instances

1197 -2. **Multiple metric types**: Handle different Prometheus metric types appropriately

1198 -3. **Feature detection**: Implement version-based feature availability

1199 -4. **Configuration-based charts**: Modify charts based on user configuration

1200 -5. **Fallback type detection**: Handle ambiguous metric types with intelligent fallbacks

1201 -

1202 -### Performance Optimization Rules

1203 -

1204 -1. **Periodic expensive operations**: Cache expensive queries with time-based invalidation

1205 -2. **Batch operations**: Group related operations together

1206 -3. **Resource reuse**: Reuse buffers, parsers, and temporary objects

1207 -4. **Intelligent caching**: Use advanced caching with staleness detection

1208 -5. **Connection pooling**: Configure pools appropriately for collector patterns

1209 -

1210 -This comprehensive guide provides the foundation for building robust, efficient go.d collectors that integrate seamlessly with Netdata's monitoring ecosystem. The advanced patterns documented here are based on analysis of production collectors and represent battle-tested approaches for handling complex, real-world monitoring scenarios.

\ No newline at end of file

src/go/plugin/framework/docs/changing-framework-code.md new

+346

@@ -0,0 +1,346 @@

1 +# Changing Go Framework Code

2 +

3 +Requirement language follows the root `AGENTS.md` definitions.

4 +

5 +This guide applies to shared Go framework code, not one collector's private

6 +implementation. Framework changes have high blast radius and MUST be designed

7 +before implementation.

8 +

9 +## Scope

10 +

11 +This guide applies when changing or extending any of these areas:

12 +

13 +- `src/go/plugin/framework/collectorapi`

14 +- `src/go/plugin/framework/jobruntime`

15 +- `src/go/plugin/framework/charttpl`

16 +- `src/go/plugin/framework/chartengine`

17 +- `src/go/plugin/framework/chartemit`

18 +- `src/go/plugin/framework/functions`

19 +- `src/go/plugin/framework/vnodes`

20 +- `src/go/plugin/framework/vnoderegistry`

21 +- `src/go/plugin/framework/dyncfg`

22 +- `src/go/plugin/framework/confgroup`

23 +- `src/go/plugin/framework/runtimecomp`

24 +- `src/go/plugin/go.d/pkg/collecttest`

25 +- `src/go/pkg/netdataapi`

26 +- `src/go/pkg/metrix`

27 +- `src/go/pkg/funcapi`

28 +- `src/go/pkg/topology`

29 +- `src/go/pkg/matcher`

30 +- `src/go/pkg/stm`

31 +- shared collector/runtime helpers under `src/go/pkg/` when their semantics are

32 + used by go.d collectors or framework runtime code, such as `web`,

33 + `prometheus`, `tlscfg`, `netdataapi`, and `netipc`

34 +- shared go.d helper packages under `src/go/plugin/go.d/pkg/`, such as

35 + `collecttest`, `ndexec`, `logs`, `sqlquery`, `cloudauth`, `pinger`,

36 + `snmputils`, `k8sclient`, and `dockerhost`

37 +

38 +It also applies when a collector change requires a new shared framework

39 +capability instead of collector-local code.

40 +

41 +## Core Rule

42 +

43 +Framework changes MUST optimize for the clean end state, not the smallest local

44 +diff. If a collector exposes a general framework gap, the implementation MUST

45 +consider a framework change before adding collector-local glue, package-level

46 +globals, duplicate helpers, or private coupling.

47 +

48 +Framework changes MUST NOT begin until the applicable approval tier below is

49 +satisfied.

50 +

51 +## Framework Vs Collector-Local

52 +

53 +Use this split before designing:

54 +

55 +- Collector-local code is appropriate when the behavior depends on one upstream

56 + product, one collector's private model, or one collector's artifact set.

57 +- Calling existing framework APIs from a collector is collector-local work.

58 + Changing those APIs, or changing any package in the scope list above, is

59 + framework work.

60 +- Framework code is appropriate when the behavior affects lifecycle,

61 + chart/template semantics, metric storage, host scopes, Functions, topology,

62 + dynamic config, shared tests, shared matchers, or multiple collectors.

63 +- The test is the generality of the behavior, not only the directory touched.

64 + Collector-local globals, singletons, adapters, duplicated helpers, or package

65 + glue that substitute for a missing general framework capability are framework

66 + work for approval purposes.

67 +- A framework extension is usually appropriate when two collectors would

68 + otherwise need the same helper or workaround.

69 +- A collector-local workaround MUST NOT be used only because it is less churn.

70 +

71 +When uncertain, pause and ask for a design decision with evidence.

72 +

73 +## Approval Tiers

74 +

75 +Use the smallest tier that honestly fits the change. If the risk is unclear,

76 +use the full design gate.

77 +

78 +### Full Design Gate

79 +

80 +The full design gate is REQUIRED for framework changes that affect contracts,

81 +runtime behavior, compatibility, lifecycle, chart output, metric storage,

82 +Function protocol, topology payloads, host scopes/vnodes, dyncfg behavior, or

83 +multiple collectors.

84 +

85 +The full design gate requires the design note below and explicit user approval

86 +before implementation.

87 +

88 +### Short Decision Gate

89 +

90 +The short decision gate is allowed only for additive, backward-compatible

91 +framework changes that do not alter existing behavior or public contracts. This

92 +includes narrow cases such as exposing an existing helper, adding an extension

93 +interface that existing implementations do not need to satisfy, or adding a

94 +test helper that preserves all existing caller semantics.

95 +

96 +The short decision gate MUST NOT be used to disguise a collector-local

97 +workaround, avoid a full design discussion, or reduce the apparent blast radius

98 +of a change that really belongs under the full design gate. Before using this

99 +tier, verify that the change still serves the clean end state. If the change is

100 +a hack, it MUST NOT be implemented under this tier.

101 +

102 +Before implementation, record the short decision in the active TODO or SOW. The

103 +record MUST include:

104 +

105 +1. Root cause.

106 +2. Why collector-local code is the wrong place.

107 +3. Why the change is additive and backward-compatible.

108 +4. Why this is the clean framework shape rather than a tier-reducing hack.

109 +5. Approval source: either the exact user request that already approved this

110 + framework addition or the explicit approval response after presenting this

111 + short-gate note.

112 +6. Affected packages and callers searched.

113 +7. Representative collectors selected for validation, or why none apply.

114 +8. Tests that will prove no existing behavior changed.

115 +9. Documentation, spec, skill, and integration-artifact update decision.

116 +

117 +Ask for explicit user approval when the request does not already cover the

118 +decision, when compatibility is uncertain, or when another package or collector

119 +needs changes to consume the new framework capability.

120 +If the task began as collector work, the short gate still requires explicit

121 +user approval before implementation.

122 +

123 +## Required Design Note

124 +

125 +For full-gate changes, prepare a design note, record it in the active TODO or

126 +SOW, and get user approval. The design note MUST cover:

127 +

128 +1. Root cause.

129 + - What exactly is broken or missing?

130 + - Why is a collector-local fix insufficient?

131 +2. Clean end state.

132 + - What is the intended framework shape after the work is complete?

133 + - Which current compromise or workaround will be removed or avoided?

134 +3. Scope boundary.

135 + - What is included in this step?

136 + - What is explicitly deferred?

137 + - Does any independent prerequisite need to land first?

138 +4. Affected contracts.

139 + - Public interfaces, runtime behavior, chart template semantics,

140 + `metrix` read/write semantics, Function protocol, host scopes/vnodes,

141 + topology payloads, generated docs, tests, and collector compatibility.

142 +5. Compatibility.

143 + - Is this preserving existing contracts?

144 + - If not, what breaking change did the user explicitly accept?

145 +6. Existing patterns.

146 + - Which framework packages or collectors already solve something similar?

147 + - Which pattern is being reused?

148 +7. Implementation batches.

149 + - Split into coherent commits when the work is non-trivial.

150 + - Each batch SHOULD build on the previous batch and be reviewable alone.

151 +8. Validation.

152 + - Framework unit tests.

153 + - Representative collector tests.

154 + - Docs/spec/skill updates.

155 +

156 +## Scope Checkpoints

157 +

158 +At every coherent batch boundary, you MUST re-check scope:

159 +

160 +- If the branch now contains independent framework work, split it out or defer

161 + it.

162 +- If the collector change is complete but a framework cleanup is separate,

163 + submit the collector change first and continue later.

164 +- If a framework change blocks the clean end state, pause and get approval for

165 + the framework change before continuing.

166 +- If the current branch depends on an independent change, land that change first

167 + and rebase on master before continuing.

168 +

169 +## Contract Checklist

170 +

171 +Use this checklist when the changed package is involved.

172 +

173 +### collectorapi

174 +

175 +- Collector interfaces MUST stay compatible unless a breaking change is

176 + explicitly approved.

177 +- Backward-compatible new collector capabilities MUST be expressed as extension

178 + interfaces that existing collectors are not required to implement. Breaking

179 + collector contract changes require explicit approval.

180 +- Registration behavior MUST be covered by tests when changed.

181 +

182 +### jobruntime

183 +

184 +- Lifecycle semantics MUST be explicit: `Init`, `Check`, `Collect`, `Cleanup`,

185 + commit, abort, cancellation, retry, and runtime metrics.

186 +- Cancellation behavior MUST be tested when changed.

187 +- V1 and V2 behavior MUST be considered separately.

188 +

189 +### metrix

190 +

191 +- Read/write semantics MUST be documented and tested: snapshot vs stateful,

192 + gauges, counters, StateSet, labels, host scopes, flattening, and cycle abort.

193 +- The `BeginCycle`, `CommitCycleSuccess`, and `AbortCycle` contract MUST be

194 + preserved and tested when changed.

195 +- Identity and label behavior MUST be stable.

196 +- New instrument behavior MUST include tests for both typed and flattened

197 + readers when applicable.

198 +

199 +### charttpl and chartengine

200 +

201 +- Template schema changes MUST update docs, validation, and compile tests.

202 +- Runtime chart behavior MUST be covered by planner/engine tests.

203 +- Per-host-scope planning MUST keep chart coverage, lifecycle, and labels

204 + isolated per scope.

205 +- Generated chart IDs, contexts, dimensions, labels, and lifecycle behavior are

206 + public contracts and MUST be treated as stable unless a breaking change is

207 + approved.

208 +

209 +### chartemit

210 +

211 +- Emitted chart and host commands MUST remain compatible with the plugin

212 + protocol.

213 +- Host/vnode identity changes MUST be tested against invalid and edge-case

214 + host information.

215 +

216 +### host scopes and vnodes

217 +

218 +See `.agents/sow/specs/go-v2-host-scope.md`.

219 +

220 +- Scope identity MUST use deterministic stable IDs.

221 +- Framework changes MUST preserve collector-provided `_vnode_type` labels. The

222 + framework does not synthesize this label for collectors.

223 +- Cardinality MUST be bounded and documented.

224 +- Representative scoped and unscoped collectors MUST be checked when read/write

225 + behavior changes.

226 +

227 +### Functions

228 +

229 +- Function protocol changes MUST stay compatible with

230 + `src/plugins.d/FUNCTION_UI_SCHEMA.json` unless a breaking change is approved.

231 +- Function handlers MUST remain isolated from collector internals through narrow

232 + dependencies.

233 +- Manager, scheduler, cancellation, and cleanup behavior MUST be tested when

234 + touched.

235 +

236 +### topology

237 +

238 +See `.agents/skills/project-create-topology/SKILL.md` and

239 +`.agents/sow/specs/topology-function-schema.md`.

240 +

241 +- New topology producers MUST use `src/go/pkg/topology/v1`.

242 +- Payload changes MUST validate against

243 + `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.

244 +- Actor/link identity and table merge behavior MUST be treated as public

245 + contracts.

246 +

247 +### matcher

248 +

249 +- Existing matcher grammar and edge-case behavior MUST stay compatible unless a

250 + breaking change is approved.

251 +- New selector behavior SHOULD use matcher package APIs instead of adding

252 + collector-local grammar.

253 +

254 +### collecttest

255 +

256 +- Test-helper changes MUST remain compatible with every current caller and MUST

257 + be validated with compile coverage plus representative callers.

258 +- Shared assertions MUST NOT hide per-scope, per-label, or per-chart failures by

259 + over-aggregating results.

260 +

261 +## Representative Collector Checks

262 +

263 +When a framework change affects collectors, choose representative collectors

264 +from the affected surface. Broad tests MUST NOT be used blindly as a substitute

265 +for choosing the right representatives.

266 +

267 +Common representatives:

268 +

269 +- V2 metrics and chart templates: `cato_networks`, `azure_monitor`, `powerstore`,

270 + `powervault`, `ping`.

271 +- V2 Functions: `mysql`, `cato_networks`.

272 +- Host scopes/vnodes: `azure_monitor`, `cato_networks`.

273 +- Topology Functions: `cato_networks`, `snmp_topology` when the legacy topology

274 + path is affected.

275 +- Legacy V1 compatibility: pick a directly affected V1 collector and one simple

276 + V1 collector such as `apache` when changing shared V1/V2 runtime code.

277 +

278 +The exact list SHOULD be justified in the design note.

279 +

280 +## Validation

281 +

282 +Validation MUST match the changed contract.

283 +

284 +Examples:

285 +

286 +- Framework package tests:

287 + - `go test -count=1 ./plugin/framework/...`

288 + - `go test -count=1 ./pkg/metrix/...`

289 + - `go test -count=1 ./pkg/matcher/...`

290 + - `go test -count=1 ./pkg/topology/...`

291 + - `go test -count=1 ./plugin/go.d/pkg/collecttest`

292 + - `go test -count=1 ./plugin/go.d/pkg/...` when shared go.d helper semantics

293 + change.

294 +- Collector representatives:

295 + - `go test -count=1 ./plugin/go.d/collector/<name>/...`

296 + - `go test -race -count=1 ./plugin/go.d/collector/<name>/...` when

297 + concurrency, Functions, host scopes, or topology are involved.

298 + - HTTP/web helper changes: include at least one HTTP collector and its config

299 + serialization tests.

300 + - Matcher changes: include selector-using collectors.

301 + - `collecttest` changes: include several representative V2 collectors that use

302 + chart coverage, config serialization, and host scopes where relevant.

303 +- Runtime components:

304 + - `go test -count=1 ./plugin/framework/jobruntime ./plugin/framework/runtimecomp`

305 +- Runtime wiring and dyncfg lifecycle:

306 + - `go test -race -count=1 ./plugin/agent/jobmgr/...`

307 + - REQUIRED when changing `collectorapi`, `jobruntime`, `dyncfg`,

308 + `confgroup`, `vnoderegistry`, or runtime wiring behavior.

309 + - Representative files include `manager_v2_test.go`, `job_factory_test.go`,

310 + `sim_test.go`, `dyncfg_collector_test.go`, and `dyncfg_vnode_test.go`.

311 +- Function/topology payloads:

312 + - schema validation tests in the affected collector or package.

313 +

314 +Record exactly what ran. Full validation MUST NOT be claimed from a narrow

315 +command.

316 +

317 +## Artifact Updates

318 +

319 +Framework changes often require durable artifact updates. Check each class:

320 +

321 +- `AGENTS.md` and `src/go/AGENTS.md`

322 +- project skills under `.agents/skills/`

323 +- framework package READMEs

324 +- specs under `.agents/sow/specs/`

325 +- collector authoring docs under `src/go/plugin/go.d/docs/`

326 +- integrations-lifecycle skill and artifacts if collector metadata/taxonomy

327 + changes

328 +- public Function/topology schemas and guides if protocol behavior changes

329 +

330 +If no artifact update is needed, record why in the active TODO/SOW.

331 +

332 +If the framework work was discovered while writing a collector, return to

333 +`src/go/plugin/go.d/docs/how-to-write-a-collector.md` after the framework

334 +decision or change is complete.

335 +

336 +## Anti-Patterns

337 +

338 +- Framework behavior hidden behind a collector-local workaround.

339 +- Package-level globals used to share state between framework packages.

340 +- Shared test helpers that pass by aggregating away the failing dimension.

341 +- Public interface changes without a compatibility decision.

342 +- Runtime behavior changes without representative collector tests.

343 +- New topology work using legacy topology payloads.

344 +- Config knobs added to avoid designing the framework behavior.

345 +- Continuing implementation after discovering independent scope that SHOULD be

346 + landed separately.

src/go/plugin/go.d/docs/helper-packages.md new

+322

@@ -0,0 +1,322 @@

1 +# Go Helper Packages For go.d Collectors

2 +

3 +Use existing helper packages before adding collector-local plumbing. A helper is

4 +not better because it is shared; it is better when it gives users the same

5 +configuration shape, the same safety behavior, or the same testable parsing path

6 +as other collectors.

7 +

8 +This guide covers helper surfaces used by go.d collectors across:

9 +

10 +- `src/go/pkg/*` for shared Go packages used beyond go.d;

11 +- `src/go/plugin/go.d/pkg/*` for go.d-specific helpers;

12 +- `src/go/logger` for the logger embedded through `collectorapi.Base`.

13 +

14 +It is not an exhaustive API reference. Before adding a local helper, search

15 +these roots for an existing package that already owns the behavior.

16 +

17 +## Helper Roots

18 +

19 +| Need | Start with |

20 +|---|---|

21 +| V2 metrics, metric stores, host scopes | `src/go/pkg/metrix` |

22 +| Duration and tri-state config option types | `src/go/pkg/confopt` |

23 +| HTTP request/client config | `src/go/pkg/web` |

24 +| TLS config outside HTTP | `src/go/pkg/tlscfg` |

25 +| Prometheus exposition parsing | `src/go/pkg/prometheus` |

26 +| User selector/matcher grammar | `src/go/pkg/matcher` |

27 +| Collector logging and log limiting | `src/go/logger` |

28 +| Function request/response helpers | `src/go/pkg/funcapi` |

29 +| Topology payloads | `src/go/pkg/topology/v1` |

30 +| Agent API / chart emission payloads | `src/go/pkg/netdataapi` |

31 +| TCP/UDP/Unix line-protocol clients | `src/go/plugin/go.d/pkg/socket` |

32 +| Command execution | `src/go/plugin/go.d/pkg/ndexec` |

33 +| Log-file readers/parsers | `src/go/plugin/go.d/pkg/logs` |

34 +| IP range parsing | `src/go/plugin/go.d/pkg/iprange` |

35 +| SQL query/scan helpers | `src/go/plugin/go.d/pkg/sqlquery` |

36 +| Cloud auth config/credentials | `src/go/plugin/go.d/pkg/cloudauth` |

37 +| Ping probing | `src/go/plugin/go.d/pkg/pinger` |

38 +| SNMP utilities | `src/go/plugin/go.d/pkg/snmputils` |

39 +| Kubernetes client helpers | `src/go/plugin/go.d/pkg/k8sclient` |

40 +| Docker host helpers | `src/go/plugin/go.d/pkg/dockerhost` |

41 +| Test helpers for collectors | `src/go/plugin/go.d/pkg/collecttest` |

42 +| Legacy V1 metric helpers | `src/go/pkg/stm`, `src/go/plugin/go.d/pkg/oldmetrix` |

43 +

44 +## Config Option Types

45 +

46 +Use `src/go/pkg/confopt` for common configuration value types.

47 +

48 +When:

49 +

50 +- users configure durations that should accept strings such as `5s`, `30m`, or

51 + numeric seconds;

52 +- users need explicit `auto` / `enabled` / `disabled` behavior instead of a

53 + plain boolean;

54 +- a migration needs to preserve legacy pointer-boolean semantics without

55 + keeping pointer plumbing in new code.

56 +

57 +Why:

58 +

59 +- `confopt.Duration` and `confopt.LongDuration` centralize YAML/JSON duration

60 + parsing and formatting;

61 +- `confopt.AutoBool` makes tri-state behavior explicit and schema-friendly;

62 +- collectors avoid ad hoc parsers and inconsistent boolean defaults.

63 +

64 +## HTTP Collectors

65 +

66 +Use `src/go/pkg/web` for HTTP-based collectors.

67 +

68 +When:

69 +

70 +- the collector talks to an HTTP or HTTPS endpoint;

71 +- users need the normal Netdata HTTP options: `url`, timeout, redirects, proxy,

72 + basic auth, bearer token file, headers, body, method, and TLS fields;

73 +- the collector builds repeated requests against the same endpoint.

74 +

75 +Why:

76 +

77 +- `web.HTTPConfig` embeds `web.RequestConfig` and `web.ClientConfig` so HTTP

78 + collectors expose the same option surface;

79 +- `web.NewHTTPClient(c.ClientConfig)` applies timeout, TLS, proxy, redirect, and

80 + HTTP/2 behavior consistently;

81 +- `web.NewHTTPRequest(c.RequestConfig)` and

82 + `web.NewHTTPRequestWithPath(c.RequestConfig, path)` apply user agent,

83 + authentication, headers, body, and safe path joining.

84 +

85 +Pattern:

86 +

87 +```go

88 +type Config struct {

89 + web.HTTPConfig `yaml:",inline" json:""`

90 +}

91 +```

92 +

93 +Use `src/go/pkg/tlscfg` directly only when the collector is not HTTP-based but

94 +still needs TLS, such as Redis or x509-style checks. HTTP collectors should get

95 +TLS behavior through `web.HTTPConfig`.

96 +

97 +## Prometheus Endpoints

98 +

99 +Use `src/go/pkg/prometheus` when the upstream endpoint exposes Prometheus text

100 +format.

101 +

102 +When:

103 +

104 +- the collector scrapes `/metrics` or another Prometheus exposition endpoint;

105 +- the collector needs to parse metric families or sorted series;

106 +- the collector needs a bounded selector for metric names.

107 +

108 +Why:

109 +

110 +- it reuses `web.RequestConfig` and `*http.Client`;

111 +- it handles Prometheus text parsing and gzip responses;

112 +- selectors avoid parsing or processing metric families the collector will not

113 + use.

114 +

115 +Do not hand-roll text exposition parsing in a collector.

116 +

117 +## Selectors And Matchers

118 +

119 +Use `src/go/pkg/matcher` for user-facing include/exclude or selector fields.

120 +

121 +When:

122 +

123 +- users select entities by name, ID, interface, queue, topic, or similar labels;

124 +- the selector syntax can be glob, regexp, string, or simple patterns;

125 +- negative matches such as `!*test* *` are sufficient.

126 +

127 +Why:

128 +

129 +- users get one matcher grammar across collectors;

130 +- tests can cover selector behavior without custom parser logic;

131 +- existing logical matchers can combine conditions when needed.

132 +

133 +Do not invent a selector language unless the upstream API requires one. Prefer a

134 +single simple-pattern field for simple cases; add separate include/exclude fields

135 +only when the user problem needs that shape.

136 +

137 +Do not use `src/go/pkg/selectorcore` for user-facing collector selectors. It is

138 +the lower-level selector metadata/parser surface used by template and selector

139 +engines, not the normal collector selector helper.

140 +

141 +## Limited Logging

142 +

143 +Collectors embed `collectorapi.Base`, which embeds `*logger.Logger`. Use the

144 +logger's built-in limiting before adding collector-local rate-limit state.

145 +

146 +When:

147 +

148 +- an error can repeat every collection cycle;

149 +- a partial failure is useful to report but would spam logs;

150 +- a one-time notice or warning is enough.

151 +

152 +Why:

153 +

154 +- in go.d jobs, `c.Once(key).Warningf(...)` is cycle-local because the runtime

155 + resets `Once` state each `runOnce`; it is useful for suppressing duplicate

156 + messages inside one cycle only;

157 +- `c.Limit(key, n, window).Warningf(...)` logs at most `n` messages per key per

158 + window and is the right default for cross-cycle spam control;

159 +- the limiter is shared through the collector logger and already used by modern

160 + collectors such as Cato Networks, PAN-OS, and vSphere.

161 +

162 +Pattern:

163 +

164 +```go

165 +c.Limit("mycollector:operation:error", 1, time.Hour).

166 + Warningf("operation failed: %v", err)

167 +```

168 +

169 +Use stable keys. Include the operation and bounded error class when needed, but

170 +do not put unbounded IDs, URLs, query strings, customer names, or raw provider

171 +messages in the key.

172 +

173 +Custom warning gates are justified only when the built-in count-per-window

174 +semantics are not the right behavior, for example when logging only on state

175 +transitions. Document that reason in the PR description or design note so

176 +reviewers can see why the built-in limiter was not enough.

177 +

178 +## Socket Clients

179 +

180 +Use `src/go/plugin/go.d/pkg/socket` for simple TCP, UDP, or Unix-socket

181 +line-protocol collectors.

182 +

183 +When:

184 +

185 +- the collector connects to a local or remote socket and sends text commands;

186 +- the response is processed line by line;

187 +- the collector needs shared timeout, TLS, and max-read-line behavior.

188 +

189 +Why:

190 +

191 +- socket address parsing is shared across collectors;

192 +- connect, command, read, disconnect, deadline, and line-limit behavior stay

193 + consistent;

194 +- tests can use the helper's fake TCP/UDP/Unix servers instead of custom socket

195 + harnesses.

196 +

197 +Do not hand-roll socket dial/read loops for common line-oriented protocols.

198 +

199 +## External Commands

200 +

201 +Use `src/go/plugin/go.d/pkg/ndexec` for collectors that execute binaries.

202 +

203 +When:

204 +

205 +- the collector needs a local command output;

206 +- the command should run through Netdata's helper wrappers;

207 +- the command may need privilege through `ndsudo`;

208 +- tests need to stub helper paths.

209 +

210 +Why:

211 +

212 +- arguments are passed without a shell;

213 +- timeouts and context cancellation are handled;

214 +- stderr snippets are bounded;

215 +- helpers integrate with Netdata's execution model.

216 +

217 +Use:

218 +

219 +- `RunUnprivileged` / `RunUnprivilegedWithOptions...` for unprivileged commands;

220 +- `RunNDSudo` for commands exposed through `ndsudo`;

221 +- `RunDirect` only when direct execution is intentionally required;

222 +- `FindBinary` for PATH/default-path discovery.

223 +

224 +Do not call `exec.Command` directly unless the helper cannot support the case and

225 +the reason is documented.

226 +

227 +## Log File Collectors

228 +

229 +Use `src/go/plugin/go.d/pkg/logs` for collectors that parse application log

230 +files.

231 +

232 +When:

233 +

234 +- the collector tails files that can rotate;

235 +- the log format is CSV, LTSV, regexp, or JSON;

236 +- parser errors should be distinguishable from I/O errors.

237 +

238 +Why:

239 +

240 +- `logs.Reader` is log-rotation aware;

241 +- `logs.NewParser` centralizes supported parser types;

242 +- `logs.IsParseError` lets collection logic treat malformed rows differently

243 + from source failures.

244 +

245 +Do not open and seek log files manually unless the collector's source is not a

246 +normal file-tail workflow.

247 +

248 +## IP Ranges

249 +

250 +Use `src/go/plugin/go.d/pkg/iprange` when users configure address ranges.

251 +

252 +When:

253 +

254 +- the collector filters IPs, networks, peers, or hosts by ranges;

255 +- the config accepts CIDR, range, or other supported range syntax.

256 +

257 +Why:

258 +

259 +- range parsing and membership checks are shared;

260 +- invalid syntax handling is consistent;

261 +- collectors avoid slightly different IP matching semantics.

262 +

263 +## SQL Helpers

264 +

265 +Use `src/go/plugin/go.d/pkg/sqlquery` for repeated SQL row-scanning patterns.

266 +

267 +When:

268 +

269 +- the collector or Function scans rows into strings, integers, floats, or discard

270 + columns;

271 +- the collector needs table-column discovery with `?` or `$1` placeholders;

272 +- the row-to-value assignment is generic across queries.

273 +

274 +Why:

275 +

276 +- scan holders and null handling are centralized;

277 +- query duration measurement and row iteration behavior stay testable;

278 +- Function code can avoid custom one-off scanners.

279 +

280 +## Cloud Auth Helpers

281 +

282 +Use `src/go/plugin/go.d/pkg/cloudauth` when a cloud collector needs supported

283 +cloud-provider credentials.

284 +

285 +When:

286 +

287 +- the collector supports `cloud_auth` configuration;

288 +- Azure AD credential construction is needed.

289 +

290 +Why:

291 +

292 +- provider names normalize consistently;

293 +- validation is centralized;

294 +- unsupported providers fail with consistent errors.

295 +

296 +## Ping Helpers

297 +

298 +Use `src/go/plugin/go.d/pkg/pinger` for ping/latency probing.

299 +

300 +When:

301 +

302 +- a collector needs ICMP-style probing;

303 +- it needs shared latency/jitter derivation.

304 +

305 +Why:

306 +

307 +- probe config validation and derived metrics are shared;

308 +- collectors avoid reimplementing packet sampling and jitter math.

309 +

310 +## Legacy V1 Helpers

311 +

312 +`src/go/pkg/stm` converts structs into `map[string]int64`.

313 +`src/go/plugin/go.d/pkg/oldmetrix` provides V1 metric vector helper types such

314 +as counters, summaries, histograms, and boolean conversions used by existing V1

315 +collectors. Both helpers are V1-shaped. New V2 collectors MUST NOT use them as

316 +their metric path.

317 +

318 +Acceptable uses:

319 +

320 +- maintaining an existing V1 collector;

321 +- temporary parity tests during V1-to-V2 migration, provided the helper is not

322 + reachable from the final runtime path.

src/go/plugin/go.d/docs/how-to-write-a-collector.md

+402 -31

@@ -1,41 +1,159 @@

1 # How to Write a go.d Collector (V2)

3 -This guide is intentionally minimal. The source of truth is existing collectors in

4 -`src/go/plugin/go.d/collector/`.

3 +This is the canonical starting point for new go.d collectors. New collectors

4 +MUST use framework V2. V1 collectors remain in the tree for compatibility and

5 +maintenance only.

6 -## Scope

7 +For migrating an existing V1 collector, use

8 +`src/go/plugin/go.d/docs/migrate-v1-to-v2.md` instead. Migration is

9 +compatibility work and has different rules from new collector authoring.

8 -- New collectors should use `collectorapi.CollectorV2`.

9 -- `CollectorV1` is legacy compatibility for existing collectors; do not use it for new work.

11 +Use `src/go/plugin/go.d/collector/cato_networks/` as the primary modern example.

12 +It is large, so copy the pattern, not the whole shape. The useful references are

13 +called out below by responsibility.

11 -## Start from a Real Collector

15 +## Before Writing Code

13 -- Minimal V2 example: `src/go/plugin/go.d/collector/ping/collector.go`.

14 -- V2 with function methods: `src/go/plugin/go.d/collector/mysql/collector.go`.

17 +Do the design work first:

16 -## Required Collector Pieces

19 +1. Read the upstream API or protocol docs. Do not infer current behavior from

20 + memory or from generated SDK types alone.

21 +2. Check existing helper packages before implementing parser, HTTP, selector,

22 + command-execution, SQL, ping, log-reading, or log-limiting plumbing. Start

23 + with `src/go/plugin/go.d/docs/helper-packages.md`.

24 +3. You MUST aim for the clean end state, not the smallest initial diff. If the

25 + clean collector design requires a framework improvement, surface that as a

26 + design decision and follow

27 + `src/go/plugin/framework/docs/changing-framework-code.md` instead of hiding

28 + it behind collector-local glue.

29 +4. Decide the monitored entities and cardinality bounds. If one job collects

30 + remote resources that SHOULD be separate Netdata nodes, design V2 host scopes

31 + from the start.

32 +5. Decide the minimal public config surface. Public config is a compatibility

33 + contract. Use constants for internal tuning such as page limits, scan cadence,

34 + retry limits, fan-out concurrency, and cache TTLs unless the operator has a

35 + real decision to make. A proposed config option MUST name that concrete

36 + operator decision; "operators may want to tune it" is not enough.

37 +6. Decide whether the collector needs Functions or topology. Functions are

38 + interactive live/snapshot views; metrics are time series. New topology

39 + producers MUST use `src/go/pkg/topology/v1` and validate against

40 + `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.

41 +7. Plan collector consistency using

42 + `.agents/skills/integrations-lifecycle/consistency.md`. Generated

43 + integration pages and README symlinks are outputs, not hand-authored sources.

44 +8. Plan the first coherent batch and its boundaries. At each boundary, you MUST

45 + re-check whether new work has drifted out of scope; defer it or land it

46 + independently before continuing.

18 -For a new collector `foo`, create `src/go/plugin/go.d/collector/foo/` with:

48 +## Source References

20 -- `collector.go`:

21 - - `collectorapi.Register("foo", collectorapi.Creator{...})`

22 - - `CreateV2`, `Config`, optional `JobConfigSchema`, optional function wiring

23 - - collector struct embedding `collectorapi.Base`

24 - - `Configuration()`, `Init()`, `Check()`, `Collect()`, `Cleanup()`

25 - - `MetricStore()` and `ChartTemplateYAML()`

26 -- `charts.yaml`: chart template consumed by the V2 chart engine.

27 -- `config_schema.json`: JSON schema for collector job config.

28 -- Any helper files you need (`collect.go`, `init.go`, `types.go`, `testdata/`, tests).

50 +Primary V2 reference:

30 -## Repository Wiring Checklist

52 +- `src/go/plugin/go.d/collector/cato_networks/`

32 -1. Import the collector in `src/go/plugin/go.d/collector/init.go`.

33 -2. Add default config file: `src/go/plugin/go.d/config/go.d/foo.conf`.

34 -3. Add toggle in `src/go/plugin/go.d/config/go.d.conf` under `modules:` (legacy key name kept for compatibility).

35 -4. Add/update the entry in `src/go/plugin/go.d/README.md` (available collectors).

54 +Read these files by responsibility:

37 -## V2 Runtime Contract

56 +- `collector.go`: registration, defaults, public lifecycle methods,

57 + `MetricStore()`, `ChartTemplateYAML()`, and Function wiring.

58 +- `config.go`: config defaults, normalization, validation, and intentionally

59 + small public config.

60 +- `collect.go`, `collect_metrics.go`, `collect_bgp.go`: collection

61 + orchestration and split domain operations.

62 +- `metrix.go`, `write_metrics.go`, `charts.yaml`: typed instruments, metric

63 + writes, chart template, `StateSet`, `instances.by_labels`, and

64 + `label_promotion`. Audit every `instances.by_labels` identity choice instead

65 + of copying labels from the example blindly.

66 +- `host_scope.go`: deterministic per-site V2 host scopes/vnodes.

67 +- `func_deps.go`, `catofunc/`: Function subpackage boundary behind a narrow

68 + dependency interface.

69 +- `topology_store.go`, `topology.go`, `topology_test.go`: immutable topology

70 + snapshot publishing and topology v1 schema validation.

71 +- `config_test.go`, `collector_lifecycle_test.go`, `collector_collect_test.go`,

72 + `charts_test.go`: table-driven V2 tests and fixture validation.

74 +Framework/API references:

75 +

76 +- `src/go/plugin/framework/collectorapi/collector.go`

77 +- `src/go/plugin/framework/docs/changing-framework-code.md`

78 +- `src/go/plugin/go.d/docs/helper-packages.md`

79 +- `src/go/pkg/metrix/README.md`

80 +- `src/go/plugin/framework/charttpl/README.md`

81 +- `src/go/plugin/framework/chartengine/README.md`

82 +- `src/go/plugin/framework/functions/README.md`

83 +- `src/go/tools/functions-validation/README.md`

84 +- `.agents/sow/specs/go-v2-host-scope.md`

85 +- `.agents/skills/integrations-lifecycle/consistency.md`

86 +

87 +## File Layout

88 +

89 +Start with this layout and add focused files only when a responsibility needs

90 +its own boundary:

91 +

92 +```text

93 +src/go/plugin/go.d/collector/<name>/

94 +|-- collector.go # registration, New, public lifecycle, store/template

95 +|-- init.go # Init helper methods for clients/matchers/state

96 +|-- config.go # Config, defaults, validation

97 +|-- collect.go # Collect orchestration

98 +|-- metrix.go # typed metrix instruments built once in New

99 +|-- write_metrics.go # normalized state -> metrix observations

100 +|-- models.go # collector-local state/DTOs

101 +|-- client.go # API/client boundary

102 +|-- charts.yaml # V2 chart template

103 +|-- config_schema.json # DYNCFG schema

104 +|-- metadata.yaml # integration metadata source

105 +|-- taxonomy.yaml # dashboard TOC placement source

106 +|-- integrations/ # generated integration page

107 +|-- README.md # symlink to generated integration page

108 +|-- testdata/ # fixtures and config serialization files

109 +`-- *_test.go # table-driven tests

110 +```

111 +

112 +Common optional splits:

113 +

114 +Most collectors need none of these optional files. Add one only when the

115 +collector has the corresponding product surface or state boundary; do not create

116 +empty `host_scope.go`, `topology.go`, or `<name>func/` files just because Cato

117 +has them.

118 +

119 +- `init.go` when `Init()` needs helper setup for clients, matchers, caches, or

120 + other persistent state. Keep the public `Init()` method itself in

121 + `collector.go`; let it call focused helpers such as `initClient()` or

122 + `initSiteSelector()`.

123 +- `collect_<operation>.go` when the collector has multiple distinct collection

124 + operations, such as discovery, account metrics, BGP, or inventory.

125 +- `normalize_<operation>.go` when API payload normalization would otherwise

126 + dominate `collect.go`.

127 +- `host_scope.go` when the collector emits generated vnodes.

128 +- `<name>func/` plus `func_deps.go` when the collector exposes Functions.

129 +- `topology.go` and `topology_store.go` when the collector emits topology.

130 +

131 +Avoid files whose names hide their responsibility. For example, a file named

132 +`diagnostics.go` SHOULD NOT contain only error classification.

133 +

134 +## Registration And Lifecycle

135 +

136 +New collectors MUST implement `collectorapi.CollectorV2` from

137 +`src/go/plugin/framework/collectorapi/collector.go` and register via `CreateV2`.

138 +In practice, `collector.go` should:

139 +

140 +- embed `config_schema.json` for `JobConfigSchema`;

141 +- embed `charts.yaml` for `ChartTemplateYAML()`;

142 +- expose `Config: func() any { return &Config{} }`;

143 +- return a new collector from `CreateV2`;

144 +- add `Methods` and `MethodHandler` only when the collector has Functions.

145 +

146 +`New()` SHOULD own defaults and test seams:

147 +

148 +- create `metrix.NewCollectorStore()`;

149 +- build typed instruments once from that store;

150 +- set default config values;

151 +- set injected seams such as client factories or clocks;

152 +- create the Function router when Functions exist.

153 +

154 +Public lifecycle and framework-contract methods MUST stay in `collector.go`:

155 +

156 +- `Configuration() any`

157 - `Init(context.Context) error`

158 - `Check(context.Context) error`

159 - `Collect(context.Context) error`

@@ -43,14 +161,267 @@ For a new collector `foo`, create `src/go/plugin/go.d/collector/foo/` with:

161 - `MetricStore() metrix.CollectorStore`

162 - `ChartTemplateYAML() string`

163

46 -Collector output is produced via `metrix` + chart templates, not by returning raw metric maps.

164 +`Init()` validates config, prepares matchers/clients, and initializes persistent

165 +state. Explicit setup details SHOULD live in helper methods, preferably in

166 +`init.go`, so the public method reads as the lifecycle sequence. `Check()` MUST

167 +be a cheap auth/connectivity probe, not a full collection. `Collect()` MUST run

168 +the real write path through `metrix`. `Cleanup()` closes idle connections and

169 +forwards Function cleanup.

170 +

171 +## Config

172 +

173 +Config SHOULD stay small and operator-oriented:

174 +

175 +- connection identity and credentials;

176 +- endpoint and standard HTTP/TLS/proxy fields when applicable;

177 +- `update_every`, `timeout`, and `vnode` when relevant;

178 +- selectors that let users intentionally scope cardinality.

179 +

180 +Implementation tuning SHOULD use constants:

181 +

182 +- discovery refresh cadence;

183 +- page sizes and maximum pages;

184 +- per-cycle fan-out concurrency;

185 +- cache TTLs;

186 +- retry/backoff internals;

187 +- API batching constraints.

188 +

189 +You MUST NOT add a config option just because it is easy to expose. Once

190 +shipped, it is hard to remove and MUST stay synchronized across `Config`,

191 +`config_schema.json`, stock `.conf`, metadata, generated docs, and tests.

192 +A proposed config option MUST name the concrete operator decision it enables;

193 +"operators may want to tune it" is not enough.

194 +

195 +For SaaS/API credentials, examples SHOULD prefer secret indirection such as

196 +`${env:COLLECTOR_API_KEY}` or `${file:/run/secrets/collector_api_key}` instead

197 +of realistic-looking inline credentials. Schema fields that carry secrets MUST

198 +be marked sensitive and use password-style UI handling where the schema

199 +supports it.

200 +

201 +Selectors SHOULD use existing matcher packages such as `src/go/pkg/matcher`

202 +unless the upstream API forces a different grammar. Document the exact matching

203 +input, for example "site name when present, otherwise site ID."

204 +

205 +## Collect Flow

206 +

207 +`Collect()` SHOULD stay orchestration, not a large parser. A typical flow is:

208 +

209 +1. ensure the client is initialized;

210 +2. refresh stable discovery only when needed;

211 +3. fetch the current snapshot/state needed for this cycle;

212 +4. enrich with optional or slower data;

213 +5. normalize API payloads into collector-local state;

214 +6. publish any immutable Function/topology snapshot;

215 +7. write metrics to `metrix`.

216 +

217 +When the collector performs several upstream calls or collection operations,

218 +those operations SHOULD be split into focused files named by operation, for

219 +example `collect_metrics.go` or `collect_bgp.go`. `collect.go` SHOULD explain

220 +the cycle; the operation files SHOULD own the operation-specific API calls,

221 +fail-soft behavior, and merge rules.

222 +

223 +Fail-soft behavior MUST be used only when partial data is still truthful. If one

224 +optional operation fails, log a rate-limited warning and omit or preserve only

225 +values that remain honest. If the core operation fails, return an error with

226 +context.

227 +

228 +`Collect()` MUST preserve context cancellation. If the context is canceled

229 +during a partial path, `Collect()` MUST return the context error so the runtime

230 +aborts the cycle instead of committing a stale or partial frame.

231 +

232 +## Metrics And Charts

233 +

234 +Metric instruments SHOULD be built once in `New()` when the metric surface is

235 +known. Use a typed collector metrics struct so write code is a value mapping,

236 +not repeated dynamic instrument lookup.

237 +

238 +Use the right instrument:

239 +

240 +- `SnapshotGaugeVec` for labeled current values; use scalar

241 + `SnapshotMeter.Gauge` only when the metric is intentionally unlabeled;

242 +- `Counter.ObserveTotal()` for source counters;

243 +- `StateSet` SHOULD be used for fixed mutually exclusive states, such as

244 + connected vs disconnected or up vs down.

245 +

246 +`charts.yaml` is the chart contract. Every template MUST define:

247 +

248 +- `version: v1`;

249 +- `context_namespace`;

250 +

251 +Templates SHOULD also group charts by operational area, use

252 +`instances.by_labels` for stable instance identity when charts are

253 +entity-scoped, use `label_promotion` for descriptive labels that should not

254 +define uniqueness, and keep the default lifecycle unless a concrete reason

255 +exists to override it.

256 +

257 +Metric labels and chart instance labels MUST be bounded and stable. Use IDs for

258 +identity. Mutable display names SHOULD be promoted with `label_promotion`.

259 +Do not blindly copy `instances.by_labels` from Cato or any other example; audit

260 +every label used for chart identity and record why it is stable enough for that

261 +collector.

262 +

263 +## Host Scopes And Vnodes

264 +

265 +Use `metrix.HostScope` when one job emits data for remote entities that SHOULD

266 +appear as separate Netdata nodes. Labels alone are not enough for that product

267 +semantics.

268 +

269 +Rules:

270 +

271 +- `ScopeKey` and `GUID` are deterministic and based on stable IDs.

272 +- Hostname may use a human-readable name when safe, with stable fallback.

273 +- Add `_vnode_type=<source>` and useful source labels.

274 +- Route every metric for that remote entity through the same host scope.

275 +- Keep the default host scope empty unless the metric truly belongs to the

276 + agent/job host.

277 +

278 +Use `.agents/sow/specs/go-v2-host-scope.md` for the framework contract.

279 +

280 +## Functions

281 +

282 +Functions MUST NOT freely access collector internals. Put Function code in a

283 +dedicated subpackage, for example `<name>func/`, with a narrow `Deps` interface

284 +declared by that subpackage.

285 +

286 +Non-topology Function responses MUST conform to

287 +`src/plugins.d/FUNCTION_UI_SCHEMA.json`; topology Function responses MUST

288 +conform to `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`. Function payloads

289 +MUST be validated with `src/go/tools/functions-validation/` or an equivalent

290 +schema-validation test, and the validation method must be recorded.

291 +

292 +Pattern:

293 +

294 +- collector package owns state and implements a small adapter in `func_deps.go`;

295 +- Function package owns method IDs, router, handlers, presentation, and tests;

296 +- Function package MUST import only framework/function types and other allowed

297 + dependencies, not the collector package;

298 +- the Function package `Deps` interface MUST expose only the methods the

299 + Function needs and MUST NOT expose, return, or embed `*Collector`;

300 +- `Collector.Cleanup()` forwards cleanup to the Function router.

301 +

302 +Use `catofunc/` as the primary example. Test the Function package with fake

303 +deps so the boundary is compile-enforced.

304 +

305 +## Topology

306 +

307 +New topology producers MUST use `src/go/pkg/topology/v1`, not legacy topology

308 +payloads. New producers MUST NOT import the non-v1 `src/go/pkg/topology`

309 +payload model.

310 +

311 +Rules:

312 +

313 +- build topology from normalized collector state, not directly from raw API

314 + payloads;

315 +- publish immutable snapshots for Function readers;

316 +- MUST NOT mutate a published topology value;

317 +- use `src/go/plugin/go.d/collector/cato_networks/topology.go` as the concrete

318 + construction reference for actors, links, detail tables, and telemetry fields;

319 +- MUST validate topology payloads in tests with both

320 + `topologyv1.ValidateDecodedData` and

321 + `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`; see

322 + `src/go/plugin/go.d/collector/cato_networks/topology_test.go`

323 + `validateCatoTopologyV1Data` for the full marshal/decode/schema check shape;

324 +- follow `.agents/skills/project-create-topology/SKILL.md` for actor/link/table

325 + design.

326 +

327 +## Repository Wiring

328 +

329 +For a new collector `<name>`:

330 +

331 +1. Add the collector package under `src/go/plugin/go.d/collector/<name>/`.

332 +2. Import it in `src/go/plugin/go.d/collector/init.go`.

333 +3. Add the default stock config:

334 + `src/go/plugin/go.d/config/go.d/<name>.conf`.

335 +4. Add the module toggle in `src/go/plugin/go.d/config/go.d.conf`.

336 +5. Add or update `src/go/plugin/go.d/README.md`.

337 +6. Add health alerts under `src/health/health.d/<name>.conf` only when alerts

338 + are useful and backed by emitted chart contexts.

339 +7. If adding or changing service-discovery rules under

340 + `src/go/plugin/go.d/config/go.d/sd/` or `sdext`, update generated

341 + service-discovery documentation through the integrations lifecycle recipe.

342 +8. Generate `integrations/<slug>.md` and the README symlink from

343 + `metadata.yaml`.

344 + Single-integration collector directories normally use the symlinked README.

345 + Multi-integration plugin directories may keep a hand-authored umbrella

346 + README; follow `.agents/skills/integrations-lifecycle/consistency.md`.

347 +

348 +Use `.agents/skills/integrations-lifecycle/recipes/add-go-collector.md` for the

349 +integration-generation commands and taxonomy pipeline details.

350 +

351 +The PR description or design note MUST enumerate the relevant collector

352 +consistency artifacts and justify every artifact that did not need a matching

353 +change. Most of this is not CI-enforced; it must be reviewer-visible.

354 +

355 +## Tests

356 +

357 +Tests SHOULD be table-driven with `map[string]struct{}` when cases share setup and

358 +assertion shape.

359 +

360 +Recommended test coverage:

361 +

362 +- config JSON/YAML serialization with `collecttest.TestConfigurationSerialize`;

363 +- config validation, including required credentials and unsafe URLs;

364 +- `Init`, cheap `Check`, `Collect`, `Cleanup`, `MetricStore`;

365 +- hard failure, partial failure, context cancellation, and recovery behavior;

366 +- chart-template schema validation with `collecttest.AssertChartTemplateSchema`

367 + and chart-template compile validation through the chartengine path used by

368 + nearby V2 collectors;

369 +- post-collect chart coverage with `collecttest.AssertChartCoverage`;

370 +- state-set values for every known state and unknown fallback;

371 +- host-scope routing when scopes/vnodes are used;

372 +- Function handler tests with fake deps when Functions exist;

373 +- topology schema validation when topology exists;

374 +- fixture validity and attribution when fixtures come from public third-party

375 + projects.

376 +

377 +Do not let tests depend on real credentials or live services unless the test is

378 +explicitly an integration test gated outside the default unit-test path.

379

380 ## Validate Locally

381

50 -From `src/go`:

382 +From `src/go`, run the narrow collector tests:

383 +

384 +```bash

385 +go test -count=1 ./plugin/go.d/collector/<name>/...

386 +```

387 +

388 +Verify that go.d can load the module:

389 +

390 +```bash

391 +timeout 15s go run ./cmd/godplugin -m <name> -d

392 +```

393 +

394 +Success means the module is registered, a job starts, and the command keeps

395 +running until the timeout stops it. Treat `unknown module`, `no jobs started`,

396 +config-load errors, or an immediate exit before the timeout as failures. Use

397 +`-c <config-dir>` when the test config lives outside the normal go.d config

398 +search path.

399 +

400 +When the collector uses concurrency or Functions, also run:

401 +

402 +```bash

403 +go test -race -count=1 ./plugin/go.d/collector/<name>/...

404 +```

405 +

406 +When integration metadata, generated pages, taxonomy, or health alerts change,

407 +run the relevant integrations pipeline checks from

408 +`.agents/skills/integrations-lifecycle/`.

409 +

410 +Do not claim full-project validation from a narrow collector command. State

411 +exactly what was run.

412

52 -- `go test ./plugin/go.d/collector/foo/...`

53 -- `go test ./plugin/go.d/collector/...`

54 -- `go test ./cmd/godplugin`

413 +## Anti-Patterns

414

56 -If you run the binary manually, CLI uses legacy flag naming: `-m/--modules`.

415 +- New collector using `Collect() map[string]int64`.

416 +- Full live collection from `Check()`.

417 +- Public config knobs for internal implementation details.

418 +- Custom selector or retry framework when existing package/framework behavior is

419 + enough.

420 +- Collector-local singleton, adapter, or glue code that substitutes for a

421 + missing shared framework capability.

422 +- Per-cycle warning/error logs for recoverable partial failures.

423 +- Metric charts for collector internals when logs are enough.

424 +- Mutable names used as chart or vnode identity.

425 +- Function package holding `*Collector`.

426 +- New topology producer using legacy topology payloads.

427 +- Hand-written `README.md` when the integration page should be generated.

src/go/plugin/go.d/docs/migrate-v1-to-v2.md new

+410

@@ -0,0 +1,410 @@

1 +# Migrating go.d Collectors From V1 To V2

2 +

3 +Requirement language follows the root `AGENTS.md` definitions.

4 +

5 +This guide is for migrating an existing go.d collector from framework V1 to

6 +framework V2. It is not the starting point for a new collector; use

7 +`src/go/plugin/go.d/docs/how-to-write-a-collector.md` for new work.

8 +

9 +V1 collectors are public integrations. A migration MUST preserve their existing

10 +user-visible contracts unless the user explicitly approves a breaking change.

11 +

12 +## Core Rule

13 +

14 +A V1-to-V2 migration is compatibility work first. The clean end state is a V2

15 +collector that behaves like the old collector from the user's point of view,

16 +with the old public contracts preserved and the internal collection path moved

17 +to `collectorapi.CollectorV2` and `metrix.CollectorStore`.

18 +

19 +Do not combine a compatibility migration with new enrichment, new topology,

20 +new host scopes, config expansion, chart redesign, or framework changes unless

21 +that work is required for the migration itself. If the migration reveals useful

22 +new work, split it into a later batch.

23 +

24 +## Before Writing Code

25 +

26 +Create a compatibility manifest before implementation. The manifest can live in

27 +the active TODO or SOW. It MUST cover:

28 +

29 +1. Module identity.

30 + - collector directory;

31 + - module name;

32 + - chart-template `context_namespace` and any group context namespaces;

33 + - `go.d.conf` toggle;

34 + - stock job config path;

35 + - service-discovery rules, if any, including

36 + `src/go/plugin/go.d/config/go.d/sd/` and

37 + `src/go/plugin/go.d/discovery/sdext/` references.

38 +2. Registration and lifecycle.

39 + - current `collectorapi.Register` entry;

40 + - `Defaults`;

41 + - `Init`, `Check`, `Collect`, and `Cleanup` behavior;

42 + - any `Once`, reconnect, cache, or retry behavior.

43 +3. Config contract.

44 + - YAML and JSON keys;

45 + - defaults;

46 + - validation;

47 + - `config_schema.json`;

48 + - stock `.conf`;

49 + - DYNCFG behavior.

50 +4. Metric/chart contract.

51 + - chart IDs;

52 + - contexts;

53 + - dimension IDs and names;

54 + - algorithms;

55 + - units;

56 + - title, family, type, priority, multiplier, divisor, hidden, and float

57 + flags;

58 + - labels;

59 + - chart variables (`Vars`) -- STOP if present; see `Chart Variables`

60 + before implementation;

61 + - dynamic chart/instance generation rules;

62 + - chart lifecycle and obsoletion timing.

63 +5. Integration artifacts.

64 + - `metadata.yaml`;

65 + - `taxonomy.yaml`;

66 + - health alerts;

67 + - generated integration page and README symlink;

68 + - `COLLECTORS.md` / plugin README entries when affected;

69 + - service-discovery docs such as `SERVICE-DISCOVERY.md`, when affected;

70 + - secrets docs such as `SECRETS.md`, when affected.

71 +6. Tests and fixtures.

72 + - existing tests to preserve;

73 + - missing contract tests to add before or during migration;

74 + - real fixture coverage.

75 +

76 +## What Must Change

77 +

78 +A V2 migration MUST replace the V1 collection path:

79 +

80 +- registration uses `CreateV2`;

81 +- the collector implements `collectorapi.CollectorV2`;

82 +- `New()` creates `metrix.NewCollectorStore()`;

83 +- the collector stores `metrix.CollectorStore`;

84 +- `Configuration()` preserves existing config return behavior;

85 +- `VirtualNode()` preserves existing vnode behavior when the V1 collector has

86 + one;

87 +- `MetricStore()` returns the store;

88 +- `ChartTemplateYAML()` returns embedded `charts.yaml`;

89 +- if the compatibility manifest contains V1 chart `Vars`, the migration stops

90 + until the `Chart Variables` decision path is resolved;

91 +- `Collect(ctx)` returns `error` and writes observations to `metrix`;

92 +- the completed migration removes the V1 `Collect() map[string]int64` output

93 + path and any runtime bridge from V1 maps into V2 `metrix`.

94 +- the completed migration removes `Charts()` and runtime `collectorapi.Charts`

95 + mutation from the production collection path.

96 +

97 +Use `src/go/plugin/framework/collectorapi/collector.go` as the source of truth

98 +for the interface.

99 +

100 +## Parity During Development

101 +

102 +Temporary V1 logic can be useful while developing the migration. For example,

103 +tests can compare V1 map output against V2 `metrix` observations after mapping

104 +both sides through the V1 chart manifest and any V2 chart-identity tooling that

105 +exists for the migration.

106 +

107 +Do not invent an unreviewed local snapshot format when the comparison needs

108 +compiled chart identity that the framework does not expose. If the migration

109 +needs reusable chart-identity or alert-variable comparison helpers, treat that

110 +as a framework/test-helper prerequisite and follow

111 +`src/go/plugin/framework/docs/changing-framework-code.md` before relying on it.

112 +

113 +That parity bridge is a development tool only. It MUST NOT remain in the final

114 +migrated collector runtime path. A finished migration that runs as

115 +V1-to-bridge-to-V2 is not a clean end state.

116 +

117 +Runtime path means any code reachable from `Init(ctx)`, `Check(ctx)`,

118 +`Collect(ctx)`, or `Cleanup(ctx)` during normal execution. V1 collection logic

119 +MUST NOT remain reachable from that runtime path, regardless of function names

120 +or return types.

121 +

122 +If parity helpers are useful long term, keep only `_test.go` helpers or

123 +fixtures under `testdata/`. Before finishing, audit imports and prove no

124 +non-test file imports the old V1 path or parity bridge.

125 +

126 +From `src/go`, run an import audit for the migrated collector:

127 +

128 +```bash

129 +go list -deps -test=false ./plugin/go.d/collector/<collector>/... |

130 + rg 'pkg/stm|plugin/go\.d/pkg/oldmetrix'

131 +rg -n 'Collect\(.*map\[string\]int64|map\[string\]int64|collectorapi\.Charts|func .*Charts\(' \

132 + plugin/go.d/collector/<collector> -g '*.go'

133 +```

134 +

135 +The dependency command MUST return no runtime dependencies on old V1-only

136 +helpers such as `stm`, `oldmetrix`, or any collector-local parity bridge

137 +package. The source grep MUST NOT find a remaining runtime V1 map output,

138 +runtime `collectorapi.Charts` mutation, or `Charts()` path. If the old path is

139 +hard to identify, delete the temporary bridge and build/test the collector; the

140 +final runtime must still compile without it.

141 +

142 +## What Must Stay Stable

143 +

144 +Unless the user approves a breaking change, the migration MUST preserve:

145 +

146 +- module name and job identity;

147 +- config field names and defaults;

148 +- chart contexts;

149 +- chart IDs;

150 +- dimension IDs and names;

151 +- dimension algorithms;

152 +- chart title, family, type, priority, units, multiplier, divisor, hidden, and

153 + float flags;

154 +- chart variable semantics used by health alerts;

155 +- health alert lookups;

156 +- metadata metric descriptions and units;

157 +- source metadata content that drives generated integration docs;

158 +- taxonomy coverage and CI behavior;

159 +- service-discovery behavior;

160 +- vnode behavior;

161 +- user-facing lifecycle behavior.

162 +

163 +If an existing collector has an accidental bug or inconsistent artifact, record

164 +it separately. Fix it in the migration only when preserving the bug would make

165 +the V2 collector incorrect or untestable; otherwise split the fix into its own

166 +tracked batch in the active TODO/SOW with owner-approved disposition. Do not

167 +close a migration with vague deferred items.

168 +

169 +## Implementation Shape

170 +

171 +Prefer the same file ownership as new V2 collectors:

172 +

173 +```text

174 +collector.go # registration, New, public lifecycle, MetricStore, ChartTemplateYAML

175 +init.go # Init helper methods when setup is non-trivial

176 +config.go # Config, defaults, validation

177 +collect.go # Collect orchestration

178 +collect_<area>.go # separate upstream operations when there are several

179 +metrix.go # typed instruments built once

180 +write_metrics.go # observations into metrix

181 +charts.yaml # V2 chart template

182 +*_test.go # focused, table-driven tests

183 +```

184 +

185 +Keep public lifecycle methods in `collector.go`. Helper methods can move into

186 +focused files when that makes ownership clearer.

187 +

188 +## Lifecycle Rules

189 +

190 +- `Init(ctx)` MUST perform setup and validation only. It MUST NOT collect the

191 + full metric set just to initialize state.

192 +- `Check(ctx)` SHOULD be a cheap probe when the upstream API supports one. If

193 + V1 used full collection for autodetection, preserve the user-visible result

194 + while making the V2 path as light as the source allows.

195 +- `Collect(ctx)` MUST write observations to `metrix` and return an error only

196 + when the cycle should abort. Fail-soft partial collection must be deliberate,

197 + tested, and logged without per-cycle spam.

198 +- `Cleanup(ctx)` MUST preserve existing cleanup behavior and release any V2

199 + Function or client resources added by the migration.

200 +

201 +## Metrics And Charts

202 +

203 +Use typed `metrix` instruments and `charts.yaml`.

204 +

205 +- Prefer creating instruments once when the metric surface is stable. Some

206 + collectors intentionally build instruments in the collection path when the

207 + surface is dynamic; if you keep that pattern, record why it is still the clean

208 + V2 shape for that collector.

209 +- Use `StateSet` for fixed one-active-state values.

210 +- Use `Counter.ObserveTotal()` when the upstream value is a source counter.

211 +- Put multipliers, divisors, hidden flags, float formatting, `instances`, and

212 + `label_promotion` in `charts.yaml`, not ad hoc runtime chart code.

213 +- Preserve V1 dimension algorithms exactly unless the old algorithm was wrong

214 + and the user approves the change.

215 +

216 +When adding labels during migration, verify they are bounded and do not alter

217 +chart identity unexpectedly. Labels that improve filtering are acceptable only

218 +when they do not break existing chart/dimension contracts.

219 +

220 +### Chart Variables

221 +

222 +V1 `collectorapi.Chart.Vars` have no direct `charts.yaml` / `charttpl` support

223 +today. Some shipped health alerts depend on those variables.

224 +

225 +If the V1 collector uses `Vars`, the migration MUST NOT silently drop them.

226 +Choose one of these paths before implementation:

227 +

228 +- add clean framework support by following

229 + `src/go/plugin/framework/docs/changing-framework-code.md`;

230 +- preserve the alert semantics through an approved equivalent design;

231 +- get explicit user approval for a breaking alert change and update health,

232 + metadata, generated docs, and release notes accordingly.

233 +

234 +Until one of those paths is approved, migrating a collector that uses chart

235 +variables is blocked. Known V1 go.d collectors using chart variables at the

236 +time of writing include `postgres`, `cockroachdb`, `hdfs`, `puppet`, `scaleio`,

237 +`whoisquery`, and `zookeeper`.

238 +

239 +### Obsoletion Timing

240 +

241 +V1 collectors often obsolete dynamic charts immediately with `MarkRemove()` /

242 +`MarkNotCreated()`. V2 chart templates expire unseen chart instances through

243 +`lifecycle.expire_after_cycles` and related chartengine policy. Exact immediate

244 +V1 timing is not always reproducible in V2.

245 +

246 +For every V1 dynamic chart, record the old obsoletion timing and choose the V2

247 +lifecycle policy deliberately. If the timing changes, document the behavioral

248 +change and get approval when it affects alerts or user-visible chart lifetime.

249 +

250 +### Dynamic IDs And Contexts

251 +

252 +V1 collectors often build chart IDs with `fmt.Sprintf`. V2 templates derive

253 +contexts from `context_namespace`, group context namespaces, and chart context

254 +leaves. V2 autogen also has `engine.autogen.max_type_id_len` behavior.

255 +

256 +The migration MUST prove that generated chart IDs, contexts, and dimensions

257 +match the old public contract, or explicitly record and approve any difference.

258 +

259 +## Config Rules

260 +

261 +Migrations MUST keep existing YAML and JSON field names. Do not rename config

262 +keys to match new code style.

263 +

264 +Do not add public config options as part of a migration unless they are required

265 +to preserve existing behavior. A proposed config option MUST name the concrete

266 +operator decision it enables; "operators may want to tune it" is not enough.

267 +Internal tuning SHOULD use constants. New user choices belong in a later

268 +feature batch with schema, stock config, metadata, and docs updated together.

269 +

270 +`autodetection_retry`, `update_every`, and `vnode` are job/runtime fields in

271 +many existing collectors. Preserve the migrated collector's current YAML/JSON

272 +behavior and keep `config.go`, `config_schema.json`, stock config, and metadata

273 +consistent. Do not copy another collector's schema treatment for these fields

274 +without checking current framework expectations.

275 +

276 +## Host Scopes, Functions, And Topology

277 +

278 +Host scopes, Functions, and topology are product design choices, not automatic

279 +migration side effects.

280 +

281 +- If the V1 collector already has vnode behavior, preserve it.

282 +- If adding host scopes would be useful but is not required for compatibility,

283 + split it into a later product decision.

284 +- If the collector exposes Functions, isolate Function code in a dedicated

285 + `<name>func/` package behind a narrow `Deps` interface. That interface MUST

286 + NOT expose, return, or embed `*Collector`.

287 +- New topology producers MUST use `src/go/pkg/topology/v1` and validate against

288 + `src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json`.

289 +

290 +## Tests

291 +

292 +Migration tests MUST prove compatibility and V2 behavior.

293 +

294 +Current shared helpers can prove schema validation, template compilation, and

295 +fixture chart coverage. They do not prove full V1-to-V2 chart identity parity,

296 +and they do not prove alert variables unless the migration records the variable

297 +source explicitly.

298 +

299 +The non-negotiable minimum is alert-observable parity. The migration MUST prove

300 +that health alert `on:` contexts, referenced dimensions, referenced variables,

301 +and dimension algorithms/units used by alerts still resolve after migration.

302 +This is provable today with chart-template checks plus manual alert-variable

303 +review.

304 +

305 +Exhaustive compiled chart-identity parity is broader: title, family, type,

306 +priority, multiplier, divisor, hidden, float flags, label promotion, and

307 +lifecycle policy. If existing exported helpers cannot observe those fields,

308 +either add a framework `collecttest` helper under

309 +`src/go/plugin/framework/docs/changing-framework-code.md`, or record a manual

310 +comparison recipe with exact fields and evidence. Do not claim exhaustive chart

311 +identity parity without one of those paths.

312 +

313 +At minimum:

314 +

315 +- config YAML/JSON serialization compatibility;

316 +- `Init`, `Check`, `Collect`, and `Cleanup` lifecycle coverage;

317 +- metric-store cycle behavior, including `BeginCycle`, successful commit, and

318 + abort on expected hard collection errors;

319 +- alert-observable parity for chart contexts, dimension IDs/names, algorithms,

320 + units, and variables used by health alerts;

321 +- chart template schema/decode/validate/compile coverage;

322 +- chart coverage for fixture data expected to materialize all dimensions;

323 +- health alert compatibility when alerts exist: each alert `on:` context must

324 + exist in the compiled template, and every variable referenced by alert

325 + calculations must still be provided by a dimension, variable-equivalent

326 + design, or approved alert change;

327 +- generated integration artifact consistency when metadata/taxonomy changes;

328 +- taxonomy coverage checks when chart contexts change;

329 +- host-scope tests if scopes/vnodes are preserved or introduced.

330 +

331 +`collecttest.AssertChartCoverage` is not a replacement for chart-identity

332 +parity. It verifies that emitted series and the template agree; it can still

333 +pass when both writer and template were renamed consistently.

334 +

335 +Exhaustive chart-identity parity is required before claiming full compatibility.

336 +If the migration lacks a shared helper or recorded manual comparison recipe for

337 +the compiled fields listed above, claim only the narrower compatibility that was

338 +actually proven.

339 +

340 +Until a shared alert-variable helper exists, use a manual grep/review pass for

341 +alert variables:

342 +

343 +```bash

344 +rg -n "\\$[A-Za-z_][A-Za-z0-9_]*" src/health/health.d/<collector>.conf

345 +rg -n "Vars:" src/go/plugin/go.d/collector/<collector> -g '*.go'

346 +```

347 +

348 +Every referenced variable must still be supplied by a dimension,

349 +variable-equivalent design, or approved alert change.

350 +

351 +Prefer table-driven tests using `map[string]struct{}` keyed by case name when

352 +cases share setup and assertion shape.

353 +

354 +## Validation

355 +

356 +Run the narrowest commands that prove the changed contract. Typical migration

357 +validation includes:

358 +

359 +```bash

360 +cd src/go

361 +go test -count=1 ./plugin/go.d/collector/<name>/...

362 +go test -race -count=1 ./plugin/go.d/collector/<name>/...

363 +timeout 15s go run ./cmd/godplugin -m <name> -d

364 +```

365 +

366 +For the load-verification command, success means the module is registered, a

367 +job starts, and the command keeps running until the timeout stops it. Treat

368 +`unknown module`, `no jobs started`, config-load errors, or an immediate exit

369 +before the timeout as failures. Use `-c <config-dir>` when the migrated test

370 +config lives outside the normal go.d config search path.

371 +

372 +Also run framework or integration checks when the migration touches those

373 +contracts:

374 +

375 +- chart template/framework changes:

376 + `go test -count=1 ./plugin/framework/charttpl ./plugin/framework/chartengine`

377 +- `metrix` changes:

378 + `go test -count=1 ./pkg/metrix/...`

379 +- runtime/framework changes:

380 + follow `src/go/plugin/framework/docs/changing-framework-code.md`

381 +- metadata/taxonomy/generated docs:

382 + follow `.agents/skills/integrations-lifecycle/consistency.md`

383 +

384 +Record exactly what ran. Full validation MUST NOT be claimed from a narrow

385 +command.

386 +

387 +## Commit Shape

388 +

389 +Prefer small coherent commits:

390 +

391 +1. Add missing compatibility tests, if needed.

392 +2. Move registration and collection path to V2.

393 +3. Convert charts to `charts.yaml` while preserving chart identity.

394 +4. Update synchronized integration artifacts.

395 +5. Add follow-up enrichment only in a separate batch.

396 +

397 +If the migration requires a framework change, stop and follow

398 +`src/go/plugin/framework/docs/changing-framework-code.md` before implementing

399 +collector-local glue.

400 +

401 +## Anti-Patterns

402 +

403 +- Rewriting chart IDs, contexts, or dimensions only because the V2 template

404 + makes a new name easier.

405 +- Adding labels, host scopes, topology, or Functions in the same commit as the

406 + compatibility migration without a product decision.

407 +- Shipping a V1-to-bridge-to-V2 runtime path after the V2 store is in place.

408 +- Hiding framework gaps in collector-local helpers.

409 +- Treating generated integration pages or README symlinks as authoring sources.

410 +- Claiming compatibility without a manifest and tests.

src/go/plugin/go.d/pkg/README.md

+49 -15

@@ -1,15 +1,49 @@

1 -# Helper Packages

2 -

3 -- if you need to run an external command, please use [`ndexec`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg/ndexec).

4 -- if you need IP ranges consider to

5 - use [`iprange`](/src/go/plugin/go.d/pkg/iprange).

6 -- if you parse an application log files, then [`log`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg/logs) is

7 - handy.

8 -- if you need filtering

9 - check [`matcher`](/src/go/pkg/matcher).

10 -- if you collect metrics from an HTTP endpoint use [`web`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg/web).

11 -- if you collect metrics from a prometheus endpoint,

12 - then [`prometheus`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg/prometheus)

13 - and [`web`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg/web) is what you need.

14 -- [`tlscfg`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg/tlscfg) provides TLS support.

15 -- [`stm`](https://github.com/netdata/netdata/tree/master/src/go/plugin/go.d/pkg/stm) helps you to convert any struct to a `map[string]int64`.

1 +# go.d Helper Packages

2 +

3 +This directory contains go.d-specific reusable helpers. go.d collectors also use

4 +shared Go helpers from `src/go/pkg/*` and logging helpers from `src/go/logger`.

5 +

6 +For new go.d collectors, start with

7 +`src/go/plugin/go.d/docs/how-to-write-a-collector.md`. For when and why to use

8 +helper packages, read `src/go/plugin/go.d/docs/helper-packages.md`.

9 +

10 +## V2 Metrics And Charts

11 +

12 +New go.d collectors MUST use framework V2:

13 +

14 +- `src/go/pkg/metrix` for metric instruments and `CollectorStore`.

15 +- `src/go/plugin/framework/charttpl` for `charts.yaml` templates.

16 +- `src/go/plugin/framework/chartengine` for chart-template runtime behavior.

17 +- `src/go/plugin/framework/collectorapi` for `CollectorV2` registration and

18 + lifecycle contracts.

19 +

20 +If an existing V1 collector is being migrated, use

21 +`src/go/plugin/go.d/docs/migrate-v1-to-v2.md`.

22 +

23 +## Common Collector Helpers

24 +

25 +- `src/go/plugin/go.d/pkg/ndexec` runs external commands.

26 +- `src/go/plugin/go.d/pkg/iprange` parses and checks IP ranges.

27 +- `src/go/plugin/go.d/pkg/logs` helps parse application log files.

28 +- `src/go/pkg/matcher` provides selector/matcher implementations.

29 +- `src/go/pkg/confopt` provides duration and tri-state config option types.

30 +- `src/go/pkg/web` provides HTTP client configuration helpers.

31 +- `src/go/pkg/prometheus` parses Prometheus endpoints; use it with

32 + `src/go/pkg/web`.

33 +- `src/go/pkg/tlscfg` provides TLS support.

34 +- `src/go/plugin/go.d/pkg/sqlquery` provides reusable SQL row/query helpers.

35 +- `src/go/plugin/go.d/pkg/socket` provides TCP/UDP/Unix line-protocol clients.

36 +- `src/go/plugin/go.d/pkg/cloudauth` provides shared cloud authentication

37 + config/credential helpers.

38 +- `src/go/plugin/go.d/pkg/pinger` provides shared ping probing and

39 + latency/jitter calculations.

40 +

41 +## Legacy V1 Helpers

42 +

43 +- `src/go/pkg/stm` converts structs into `map[string]int64`. This is V1-shaped

44 + and MUST NOT be used as the metric path for new V2 collectors. It MAY be

45 + useful while maintaining legacy V1 collectors or building temporary migration

46 + parity tests that are removed from the final runtime path.

47 +- `src/go/plugin/go.d/pkg/oldmetrix` provides V1 metric vector helper types

48 + used by existing V1 collectors. New V2 collectors SHOULD use

49 + `src/go/pkg/metrix` instead.

src/go/plugin/ibm.d/AGENTS.md

+1 -1

@@ -132,7 +132,7 @@ The flag implicitly enables dump mode and exits once every job has produced at l

132 ## Contributing Guidelines

133

134 1. Review [`framework/README.md`](framework/README.md) for IBM.D framework details.

135 -2. Follow [General collector best practices](../BEST-PRACTICES.md).

135 +2. Follow the Go-area rules in [`../../AGENTS.md`](../../AGENTS.md).

136 3. **Never edit auto-generated files** – see [Auto-Generated Files](#auto-generated-files) section.

137 4. Always regenerate code after modifying `contexts.yaml`, `config.go`, or `module.yaml`.

138 5. Run `gofmt` on generated Go files before committing.

src/go/plugin/ibm.d/framework/README.md

+1 -1

@@ -148,6 +148,6 @@ go generate ./...

148

149 - [`../README.md`](../README.md) – project overview, build instructions, and directory map

150 - [`../AGENTS.md`](../AGENTS.md) – authoring checklist and best practices for AI assistants

151 -- [`../BEST-PRACTICES.md`](../BEST-PRACTICES.md) – in-depth guidance on go.d/ibm.d collector development

151 +- [`../../../AGENTS.md`](../../../AGENTS.md) – Go-area rules and routing

152

153 Contributions are welcome! Keep documentation, schemas, metadata, and health alerts synchronized to guarantee a smooth user experience.