| 1 | # Recipe: add a new go.d collector integration |
| 2 | |
| 3 | This recipe assumes you are adding a brand-new go.d module |
| 4 | called `<name>`. For modifying an existing collector, see |
| 5 | `update-collector.md`. |
| 6 | |
| 7 | ## 0. Read first |
| 8 | |
| 9 | - `<repo>/.agents/skills/project-writing-collectors/SKILL.md` |
| 10 | -- the broader "how to write a collector" context (NIDL |
| 11 | contexts, dashboard shaping, plugin landscape). |
| 12 | - `<repo>/src/go/plugin/go.d/docs/how-to-write-a-collector.md` |
| 13 | -- the canonical framework V2 code/layout guide for new go.d collectors. |
| 14 | - `../SKILL.md` -- this skill's overview. |
| 15 | - `../schema-reference.md` -- the `collector.json` schema |
| 16 | fields you will be filling in. |
| 17 | |
| 18 | ## 1. Create the module skeleton |
| 19 | |
| 20 | New go.d collectors MUST use framework V2. Code layout details live in |
| 21 | `src/go/plugin/go.d/docs/how-to-write-a-collector.md`; this recipe covers the |
| 22 | integration artifact side. A normal V2 collector directory includes: |
| 23 | |
| 24 | ``` |
| 25 | src/go/plugin/go.d/collector/<name>/ |
| 26 | |-- collector.go # Register/CreateV2/New/public lifecycle |
| 27 | |-- config.go # Config struct |
| 28 | |-- collect.go # Collect orchestration |
| 29 | |-- metrix.go # Typed metrix instruments |
| 30 | |-- write_metrics.go # Metric writes |
| 31 | |-- charts.yaml # V2 chart template |
| 32 | |-- config_schema.json # DYNCFG schema |
| 33 | |-- metadata.yaml # Integration metadata (this skill's territory) |
| 34 | |-- taxonomy.yaml # Dashboard TOC placement |
| 35 | |-- README.md # Will become a symlink to integrations/<slug>.md once gen runs |
| 36 | |-- testdata/ # Fixtures |
| 37 | `-- ...other .go files |
| 38 | ``` |
| 39 | |
| 40 | Plus stock conf: |
| 41 | |
| 42 | ``` |
| 43 | src/go/plugin/go.d/config/go.d/<name>.conf |
| 44 | ``` |
| 45 | |
| 46 | Plus alerts (if any): |
| 47 | |
| 48 | ``` |
| 49 | src/health/health.d/<name>.conf |
| 50 | ``` |
| 51 | |
| 52 | ## 2. Author `metadata.yaml` |
| 53 | |
| 54 | Use an existing rich collector as a template: |
| 55 | `src/go/plugin/go.d/collector/postgres/metadata.yaml`. |
| 56 | |
| 57 | Required top-level fields per `collector.json`: |
| 58 | |
| 59 | ```yaml |
| 60 | plugin_name: go.d.plugin |
| 61 | modules: |
| 62 | - meta: |
| 63 | plugin_name: go.d.plugin |
| 64 | module_name: <name> |
| 65 | monitored_instance: |
| 66 | name: "<Display Name>" |
| 67 | link: "https://upstream-site.example/" |
| 68 | categories: |
| 69 | - data-collection.<category> # see categories.yaml for valid ids |
| 70 | icon_filename: "<name>.svg" |
| 71 | keywords: [<keywords>] |
| 72 | related_resources: |
| 73 | integrations: |
| 74 | list: [] |
| 75 | info_provided_to_referring_integrations: |
| 76 | description: "" |
| 77 | overview: |
| 78 | data_collection: |
| 79 | metrics_description: | |
| 80 | First sentence: Monitor <thing> or collect <data> from <thing>. |
| 81 | Add more detail after that only if it is useful on the full page. |
| 82 | method_description: | |
| 83 | One paragraph: how we collect it. |
| 84 | supported_platforms: |
| 85 | include: [] |
| 86 | exclude: [] |
| 87 | multi_instance: true |
| 88 | additional_permissions: |
| 89 | description: "" |
| 90 | default_behavior: |
| 91 | auto_detection: |
| 92 | description: "" |
| 93 | limits: |
| 94 | description: "" |
| 95 | performance_impact: |
| 96 | description: "" |
| 97 | setup: |
| 98 | prerequisites: |
| 99 | list: [] |
| 100 | configuration: |
| 101 | file: |
| 102 | name: "go.d/<name>.conf" |
| 103 | options: |
| 104 | description: "" |
| 105 | folding: |
| 106 | title: "Config options" |
| 107 | enabled: true |
| 108 | list: [] |
| 109 | examples: |
| 110 | folding: |
| 111 | title: "Config" |
| 112 | enabled: true |
| 113 | list: |
| 114 | - name: "Basic" |
| 115 | description: "Basic configuration." |
| 116 | config: | |
| 117 | jobs: |
| 118 | - name: local |
| 119 | url: http://localhost:1234 |
| 120 | troubleshooting: |
| 121 | problems: |
| 122 | list: [] |
| 123 | alerts: [] |
| 124 | metrics: |
| 125 | folding: |
| 126 | title: "Metrics" |
| 127 | enabled: false |
| 128 | description: "" |
| 129 | availability: [] |
| 130 | scopes: |
| 131 | - name: global # will be auto-rewritten to "<Display Name> instance" |
| 132 | description: "" |
| 133 | labels: [] |
| 134 | metrics: |
| 135 | - name: <name>.<context> |
| 136 | description: <Chart title> |
| 137 | unit: <unit> |
| 138 | chart_type: line # one of: line, area, stacked, heatmap |
| 139 | dimensions: |
| 140 | - name: <dim> |
| 141 | ``` |
| 142 | |
| 143 | The first sentence of `metrics_description` is also used as the |
| 144 | description in generated catalog-style pages such as |
| 145 | `src/collectors/COLLECTORS.md`. It SHOULD stay product-facing and stable: |
| 146 | start with an action phrase, describe the integration, and MUST NOT |
| 147 | describe configuration variables, defaults, limits, or setup steps. |
| 148 | Put those details in the setup, default-behavior, examples, or |
| 149 | troubleshooting fields. |
| 150 | |
| 151 | Hit every REQUIRED field. The validator is strict (fatal on |
| 152 | warnings). Refer to `../schema-reference.md` for the |
| 153 | exhaustive field list. |
| 154 | |
| 155 | ## 3. Make sure `categories.yaml` has your category |
| 156 | |
| 157 | If your `monitored_instance.categories` references a category |
| 158 | that doesn't exist in `integrations/categories.yaml`, the |
| 159 | validator will warn (fatal). Either pick an existing category |
| 160 | or add a new one under the appropriate parent (typically |
| 161 | `data-collection`). |
| 162 | |
| 163 | ## 4. Taxonomy, stock `.conf`, `config_schema.json`, alerts, README |
| 164 | |
| 165 | These files are the rest of the collector consistency rule and MUST stay |
| 166 | synchronized with the collector code: |
| 167 | |
| 168 | - `src/go/plugin/go.d/collector/<name>/taxonomy.yaml` -- |
| 169 | dashboard TOC placement for chart contexts. Static collectors |
| 170 | use ordered `items:` trees; plain strings in structural `items:` |
| 171 | own chart contexts. Dynamic collectors use `type: selector` with |
| 172 | `context_prefix:` or `collect_plugin:` and matching |
| 173 | `metadata.yaml.metrics.dynamic_*` declarations. Display widgets |
| 174 | use `type: context` with `contexts:` and `chart_library`; those |
| 175 | referenced contexts MUST also be owned by structural items. |
| 176 | Pick `--section-id` from |
| 177 | `integrations/taxonomy/sections.yaml`; `section_id` is a stable |
| 178 | registry ID, not a path to invent in the collector file. |
| 179 | Seed the initial explicit context list with: |
| 180 | ```bash |
| 181 | python3 integrations/gen_taxonomy_seed.py src/go/plugin/go.d/collector/<name>/metadata.yaml --module-name <name> --section-id <section.id> --placement-id <name> --icon <icon> |
| 182 | ``` |
| 183 | For a rich recent example with groups, context ownership, and |
| 184 | generated integration docs, read |
| 185 | `src/go/plugin/go.d/collector/cato_networks/taxonomy.yaml`. |
| 186 | - `src/go/plugin/go.d/config/go.d/<name>.conf` -- the stock |
| 187 | config users will see at |
| 188 | `/etc/netdata/go.d/<name>.conf`. It SHOULD stay minimal but |
| 189 | representative. Show every common option with a comment. |
| 190 | - `src/go/plugin/go.d/collector/<name>/config_schema.json` -- |
| 191 | the DYNCFG schema. Each option in the stock `.conf` SHOULD |
| 192 | have a corresponding entry here, with the same default. |
| 193 | - `src/health/health.d/<name>.conf` -- alerts on the metrics |
| 194 | declared in `metadata.yaml`. Each alert in this file SHOULD |
| 195 | have a matching entry under `metadata.yaml.modules[0].alerts[]`. |
| 196 | - `src/go/plugin/go.d/collector/<name>/README.md` -- this is the |
| 197 | USER-FACING documentation. After step 5, this file will be |
| 198 | REPLACED with a symlink to |
| 199 | `integrations/<slug>.md`. You MUST NOT hand-write the |
| 200 | README; the generator does. Stub it as empty initially. |
| 201 | |
| 202 | ## 5. Run the pipeline locally |
| 203 | |
| 204 | From the repo root: |
| 205 | |
| 206 | ```bash |
| 207 | ./integrations/pip.sh # once |
| 208 | python3 integrations/gen_integrations.py |
| 209 | python3 integrations/gen_taxonomy.py --check-only |
| 210 | python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD |
| 211 | python3 -m unittest integrations.tests.test_taxonomy |
| 212 | python3 integrations/gen_docs_integrations.py -c go.d.plugin/<name> |
| 213 | python3 integrations/gen_doc_collector_page.py |
| 214 | python3 integrations/gen_doc_secrets_page.py |
| 215 | # If service-discovery rules or sdext metadata changed: |
| 216 | python3 integrations/gen_doc_service_discovery_page.py |
| 217 | ``` |
| 218 | |
| 219 | Expected outputs: |
| 220 | |
| 221 | - `integrations/integrations.js` and `integrations/integrations.json` |
| 222 | regenerated (gitignored, MUST NOT be committed). |
| 223 | - Collector taxonomy validated. If `gen_taxonomy.py` fails, fix |
| 224 | `taxonomy.yaml` or the matching `metadata.yaml.metrics.dynamic_*` |
| 225 | declaration before continuing. |
| 226 | - Touched-collector taxonomy coverage validated in the same `--pr-diff` mode |
| 227 | used by CI. If your local base branch is not `master`, adjust the diff range |
| 228 | to the PR base. |
| 229 | - `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md` |
| 230 | CREATED. Inspect: it SHOULD contain the `<!--startmeta` |
| 231 | banner with your `sidebar_label` and `learn_rel_path`, then |
| 232 | the rendered overview / setup / metrics / alerts / |
| 233 | troubleshooting sections. |
| 234 | - `src/go/plugin/go.d/collector/<name>/README.md` becomes a |
| 235 | symlink to `integrations/<slug>.md` (because there is |
| 236 | exactly one integration in this directory). |
| 237 | - `src/collectors/COLLECTORS.md` updated to include your new |
| 238 | collector in its category section. |
| 239 | - If service-discovery rules or `sdext` metadata changed, |
| 240 | `src/collectors/SERVICE-DISCOVERY.md` updated. |
| 241 | |
| 242 | If `gen_integrations.py` exits non-zero, read the warning |
| 243 | output -- a schema validation failed. Fix `metadata.yaml` and |
| 244 | re-run. |
| 245 | |
| 246 | ## 6. Verify locally |
| 247 | |
| 248 | - Open the generated `integrations/<slug>.md` and make sure |
| 249 | every section reads correctly. |
| 250 | - Open `src/collectors/COLLECTORS.md` and find your collector |
| 251 | in the table. |
| 252 | - Run `python3 integrations/check_collector_taxonomy.py --pr-diff master...HEAD` |
| 253 | before opening the PR. If your local base branch is not `master`, adjust the |
| 254 | diff range to the PR base. |
| 255 | - From `src/go`, run `timeout 15s go run ./cmd/godplugin -m <name> -d` to |
| 256 | confirm go.d can load the module after the `init.go`, `go.d.conf`, stock |
| 257 | config, and README wiring changes. Success means the module is registered, a |
| 258 | job starts, and the command keeps running until timeout stops it; `unknown |
| 259 | module`, `no jobs started`, config-load errors, or immediate exit are |
| 260 | failures. Use `-c <config-dir>` when testing a non-standard config path. |
| 261 | - If service-discovery rules or `sdext` metadata changed, run |
| 262 | `python3 integrations/gen_doc_service_discovery_page.py` and commit |
| 263 | `src/collectors/SERVICE-DISCOVERY.md`. |
| 264 | - Run |
| 265 | `git status --porcelain | rg '^(\?\?|!!| M|M |A |AM) integrations/(integrations\.(js|json)|taxonomy\.json)$' || true` |
| 266 | and make sure it prints no output; these generated runtime catalogs are |
| 267 | gitignored and MUST NOT be committed. |
| 268 | - Run `git diff` and confirm the only changes are in: |
| 269 | - `src/go/plugin/go.d/collector/<name>/...` (your new module |
| 270 | files). |
| 271 | - `src/go/plugin/go.d/collector/init.go` (registration import). |
| 272 | - `src/go/plugin/go.d/config/go.d.conf` (module toggle). |
| 273 | - `src/go/plugin/go.d/config/go.d/<name>.conf` (stock config). |
| 274 | - `src/go/plugin/go.d/README.md` (collector list). |
| 275 | - `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md` |
| 276 | (the generated integration page). |
| 277 | - `src/go/plugin/go.d/collector/<name>/README.md` (now a |
| 278 | symlink). |
| 279 | - `src/collectors/COLLECTORS.md` (umbrella page updated). |
| 280 | - `src/health/health.d/<name>.conf` (alerts file). |
| 281 | - `src/collectors/SERVICE-DISCOVERY.md` if service-discovery changed. |
| 282 | - Possibly `integrations/categories.yaml` if you added a |
| 283 | category. |
| 284 | - NOT `integrations/integrations.js` or |
| 285 | `integrations/integrations.json` (gitignored). |
| 286 | - NOT `integrations/taxonomy.json` (gitignored). |
| 287 | |
| 288 | ## 7. Commit and push |
| 289 | |
| 290 | Single PR, single commit (or a few logical commits) covering |
| 291 | the collector consistency rule plus the generated integration |
| 292 | page and umbrella update. Reviewers will check that affected |
| 293 | artifacts were updated together. |
| 294 | |
| 295 | ## 8. CI |
| 296 | |
| 297 | - `check-markdown.yml` will run on the PR. It runs the same |
| 298 | pipeline scripts, validates taxonomy, and validates Learn ingest. If your |
| 299 | committed integration page diverges from CI's regen, the |
| 300 | workflow fails -- fix locally and re-push. |
| 301 | - After merge, `generate-integrations.yml` triggers on master. |
| 302 | Since you already committed the regen, this SHOULD NOT |
| 303 | produce changes. If it does, the auto-PR |
| 304 | (`Regenerate integrations docs`) catches the drift -- merge |
| 305 | it. |
| 306 | |
| 307 | ## 9. Surface arrival timing |
| 308 | |
| 309 | - `src/collectors/COLLECTORS.md` is live in the repo |
| 310 | immediately after merge. |
| 311 | - The cloud-frontend dashboard's Integrations page rebuilds |
| 312 | on its own schedule (when the cloud-frontend CI re-runs |
| 313 | `gen_integrations.py` against master). Coordinate with |
| 314 | the dashboard team if you need to know the exact next |
| 315 | build. |
| 316 | - The Learn site's per-integration page lands within a few |
| 317 | hours -- Learn's `ingest.yml` workflow runs every 3 hours |
| 318 | (see the `learn-site-structure` skill for details). |
| 319 | |
| 320 | ## Common mistakes |
| 321 | |
| 322 | - **Forgetting one collector-consistency artifact.** The most common |
| 323 | cause of review feedback. Use `git status` after step 5 to |
| 324 | confirm every affected source/generated artifact is staged. |
| 325 | - **Hand-editing `integrations/<slug>.md` after generation.** |
| 326 | Never. It is regenerated each time. Edit `metadata.yaml` |
| 327 | and re-run. |
| 328 | - **Skipping `gen_doc_collector_page.py`.** This forgets to |
| 329 | update `src/collectors/COLLECTORS.md`, leaving your |
| 330 | collector invisible in the umbrella table even though the |
| 331 | per-integration page exists. |
| 332 | - **Categories typo.** A category id that doesn't match |
| 333 | `categories.yaml` causes validation to fail (warnings are |
| 334 | fatal). The renderer would silently fall back to |
| 335 | `data-collection.applications` if your only declared |
| 336 | category is bogus -- that fallback is itself the symptom |
| 337 | of a typo, not the desired outcome. |
| 338 | - **Slug collision.** If `clean_string(meta.name)` produces |
| 339 | the same slug as an existing collector in the same |
| 340 | directory, one overwrites the other silently. Pick a |
| 341 | unique enough display name. |