| 1 | --- |
| 2 | name: integrations-lifecycle |
| 3 | description: Authoritative reference for Netdata's integrations pipeline -- how `metadata.yaml` drives per-integration pages, collector `taxonomy.yaml` drives dashboard TOC placement, the `COLLECTORS.md`/`SECRETS.md`/`SERVICE-DISCOVERY.md` umbrellas, the `integrations.js` and `integrations/taxonomy.json` artifacts consumed by downstream systems, and per-integration `.md` files committed to the repo. Use when adding/modifying any integration (collector, exporter, agent or cloud notification, authentication, secretstore, service-discovery, log type, deploy method); editing `metadata.yaml` or `taxonomy.yaml`; checking whether `integrations/*.md` should be hand-edited; reading generator scripts under `integrations/`, schemas under `integrations/schemas/`, taxonomy registries under `integrations/taxonomy/`, templates under `integrations/templates/`, the workflows `generate-integrations.yml` or `check-markdown.yml`; ibm.d modules where `metadata.yaml` is generated from `contexts.yaml`; the collector-consistency rule (metadata.yaml + taxonomy.yaml + config_schema.json + stock conf + alerts + README move together). |
| 4 | --- |
| 5 | |
| 6 | # integrations-lifecycle |
| 7 | |
| 8 | This skill is the **single place** to learn how Netdata's |
| 9 | integrations pipeline works end to end. It documents: |
| 10 | |
| 11 | - the generator pipeline rooted in |
| 12 | `integrations/gen_integrations.py`; |
| 13 | - the collector taxonomy pipeline rooted in |
| 14 | `integrations/gen_taxonomy.py`; |
| 15 | - the JSON-Schema contracts every `metadata.yaml` and |
| 16 | `taxonomy.yaml` is validated against; |
| 17 | - every artifact the pipeline produces (gitignored runtime files |
| 18 | AND committed `.md` documentation); |
| 19 | - the `<!--startmeta` banner conventions and DO-NOT-EDIT rules; |
| 20 | - the two CI workflows that auto-PR or gate the regenerated docs |
| 21 | (`generate-integrations.yml`, `check-markdown.yml`); |
| 22 | - the secondary ibm.d generation chain |
| 23 | (`contexts.yaml` -> `metadata.yaml`); |
| 24 | - the contract by which downstream dashboard code consumes |
| 25 | `integrations.js` and, when opted in, `integrations/taxonomy.json`; |
| 26 | - the collector-consistency rule (`taxonomy.yaml` moves with |
| 27 | metadata and docs) |
| 28 | and what is and is NOT enforced by tooling; |
| 29 | - every surprising/dead/edge-case behavior an assistant or |
| 30 | maintainer is likely to hit. |
| 31 | |
| 32 | After reading SKILL.md plus the per-domain guides linked below, |
| 33 | an assistant should never need to ask "how does metadata.yaml |
| 34 | work?", "are these `integrations/*.md` files generated?", "what |
| 35 | fields does the schema support?", "what runs in CI?", "where |
| 36 | does the in-app integrations page get its data?". |
| 37 | |
| 38 | ## Key concepts (read first) |
| 39 | |
| 40 | 1. **`metadata.yaml` is the single source of truth.** Every |
| 41 | per-integration page on every surface (Learn site, in-app |
| 42 | dashboard, the umbrella `COLLECTORS.md` / `SECRETS.md` / |
| 43 | `SERVICE-DISCOVERY.md` pages) is rendered from |
| 44 | `metadata.yaml` by the pipeline. Edit `metadata.yaml`, run |
| 45 | the pipeline, commit the regenerated artifacts. |
| 46 | |
| 47 | `src/collectors/COLLECTORS.md` is the source page for |
| 48 | Learn's "Monitor anything with Netdata" page. It is generated |
| 49 | from `integrations/integrations.js` by |
| 50 | `integrations/gen_doc_collector_page.py`; never hand-edit its |
| 51 | integration tables. To update that page, change the source |
| 52 | metadata/categories or the generator, then run |
| 53 | `gen_integrations.py` and `gen_doc_collector_page.py`. |
| 54 | |
| 55 | Treat short descriptions as public product copy. Catalog |
| 56 | descriptions must say what the integration is and what it |
| 57 | monitors, enriches, exports, authenticates, or discovers. |
| 58 | For collector-like metadata, the first sentence of |
| 59 | `overview.data_collection.metrics_description` is the catalog |
| 60 | sentence used by generated pages such as `COLLECTORS.md`. |
| 61 | Start that sentence with a user-facing action phrase such as |
| 62 | `Monitor...`, `Collect...`, `Enrich network flows with...`, or |
| 63 | `Annotate network flows with...`. |
| 64 | Do not use the catalog description for variables, defaults, |
| 65 | option names, setup instructions, limits, or troubleshooting. |
| 66 | Put those details in setup, default-behavior, examples, or |
| 67 | troubleshooting fields. See `description-authoring.md`. |
| 68 | |
| 69 | 2. **`integrations/*.md` files are GENERATED. DO NOT EDIT.** |
| 70 | Every per-integration `.md` opens with a |
| 71 | `<!--startmeta ... endmeta-->` block that ends with |
| 72 | `message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED |
| 73 | BY THE COLLECTOR'S/EXPORTER'S/...'S metadata.yaml FILE"`. |
| 74 | See `artifacts-and-banners.md` for the full banner spec. |
| 75 | Edit the source `metadata.yaml`, regenerate, commit. |
| 76 | |
| 77 | 3. **The CI workflow auto-opens a "Regenerate integrations |
| 78 | docs" PR.** After a `metadata.yaml` change merges to |
| 79 | `master`, `.github/workflows/generate-integrations.yml` |
| 80 | regenerates every per-integration `.md` and the umbrella |
| 81 | pages and opens a PR for a maintainer to merge. You CAN |
| 82 | regenerate locally and include the changes in the same PR; |
| 83 | that is preferred to avoid two PRs per change. |
| 84 | |
| 85 | 4. **The collector consistency rule.** Anything that touches a |
| 86 | collector's runtime behavior MUST land in one PR with |
| 87 | matching changes to: |
| 88 | - `metadata.yaml` (the integration page driver), |
| 89 | - `taxonomy.yaml` (dashboard TOC placement for chart contexts), |
| 90 | - `config_schema.json` (the dashboard's DYNCFG editor), |
| 91 | - the stock `.conf` (what `/etc/netdata/...` ships), |
| 92 | - `health.d/*.conf` (the alert definitions), |
| 93 | - `README.md` (which is a symlink to the generated |
| 94 | `integrations/<slug>.md` for single-integration plugins). |
| 95 | See `consistency.md` for what is and is NOT automatically |
| 96 | enforced. |
| 97 | |
| 98 | 5. **ibm.d is different.** ibm.d module `metadata.yaml`, |
| 99 | `README.md`, and `config_schema.json` are GENERATED from |
| 100 | `contexts.yaml` + `config.go` + `module.yaml` via |
| 101 | `go generate`. NEVER hand-edit them. See `ibm-d.md`. |
| 102 | |
| 103 | 6. **The dashboard consumes generated integration artifacts.** |
| 104 | The cloud-frontend at |
| 105 | `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/` runs |
| 106 | `gen_integrations.py` in its own CI to copy |
| 107 | `integrations.js` into its source tree. The historical contract is |
| 108 | that `.js` file's exact shape: |
| 109 | `export const categories = [...]; export const integrations |
| 110 | = [...]`. Collector taxonomy is emitted separately as |
| 111 | `integrations/taxonomy.json` by `gen_taxonomy.py`; downstream |
| 112 | consumers opt in to that JSON contract. See `in-app-contract.md`. |
| 113 | |
| 114 | ## Table of contents |
| 115 | |
| 116 | | Guide | Purpose | |
| 117 | |---|---| |
| 118 | | `pipeline.md` | The 4-stage pipeline graph, every script, every artifact, the CI workflows. | |
| 119 | | `schema-reference.md` | Per-field reference for JSON Schemas under `integrations/schemas/`, including collector taxonomy schemas. | |
| 120 | | `description-authoring.md` | Product-copy rules for `metadata.yaml` descriptions and the Monitor Anything table text. | |
| 121 | | `per-type-matrix.md` | One-row-per-integration-type quick lookup: source paths, validator, render keys, output location. | |
| 122 | | `artifacts-and-banners.md` | Every committed and gitignored artifact; banner conventions; symlink rules. | |
| 123 | | `ibm-d.md` | The `contexts.yaml` -> `metadata.yaml` chain for ibm.d modules. | |
| 124 | | `consistency.md` | The collector consistency rule and what tooling enforces. | |
| 125 | | `in-app-contract.md` | How the cloud-frontend dashboard consumes `integrations.js`. | |
| 126 | | `gotchas.md` | Every surprise, dead-code reference, hardcoded marketing anchor, custom Jinja delimiter. | |
| 127 | | `recipes/INDEX.md` | Step-by-step recipes for adding/updating each integration type. | |
| 128 | | `how-tos/INDEX.md` | Live catalog: every analysis question gets a how-to entry. | |
| 129 | |
| 130 | ## Live how-to rule (mandatory) |
| 131 | |
| 132 | If an assistant is asked a concrete question about the |
| 133 | integrations pipeline that is NOT already documented under |
| 134 | `how-tos/` or one of the per-domain guides above, AND answering |
| 135 | it requires non-trivial analysis (reading multiple scripts, |
| 136 | running the pipeline, cross-referencing schemas), the assistant |
| 137 | MUST author a new how-to under `how-tos/<slug>.md` and add a |
| 138 | one-line entry to `how-tos/INDEX.md` BEFORE completing the task. |
| 139 | This rule is durable. Skipping it means the next assistant |
| 140 | repeats the analysis from scratch. |
| 141 | |
| 142 | ## Path discipline |
| 143 | |
| 144 | This skill follows |
| 145 | `<repo>/.agents/sow/specs/sensitive-data-discipline.md`: |
| 146 | |
| 147 | - Files in this repo: repo-relative |
| 148 | (`integrations/gen_integrations.py`, |
| 149 | `<repo>/integrations/...`, `src/...`). |
| 150 | - Files in sibling Netdata-org repos: |
| 151 | `${NETDATA_REPOS_DIR}/<repo-name>/...` (env-key from `.env`). |
| 152 | - No literal home-directory or workstation-root paths anywhere (use the env-keyed placeholder above instead). |
| 153 | |
| 154 | ## Sources of truth referenced by this skill |
| 155 | |
| 156 | - `<repo>/integrations/` -- generators, schemas, templates, |
| 157 | shared metadata files (`categories.yaml`, `deploy.yaml`). |
| 158 | - `<repo>/integrations/schemas/*.json` -- all 12 schemas. |
| 159 | - `<repo>/integrations/templates/` -- all Jinja templates. |
| 160 | - `<repo>/.github/workflows/generate-integrations.yml` and |
| 161 | `<repo>/.github/workflows/check-markdown.yml` -- the CI. |
| 162 | - `<repo>/.github/data/distros.yml` -- platform table fed into |
| 163 | deploy rendering. |
| 164 | - `<repo>/src/go/plugin/ibm.d/` -- the secondary generator |
| 165 | chain (`docgen/main.go`, `metricgen/main.go`). |
| 166 | - `<repo>/AGENTS.md` -- the "Collector Consistency |
| 167 | Requirements" policy text. |
| 168 | |
| 169 | ## Related skills |
| 170 | |
| 171 | - `project-writing-collectors` -- the broader collector |
| 172 | authoring context (NIDL contexts, dashboard shaping, plugin |
| 173 | landscape). Read FIRST when authoring a brand-new collector; |
| 174 | read THIS skill when working with the integration metadata |
| 175 | side. |
| 176 | - `learn-site-structure` -- how the per-integration `.md` files |
| 177 | ultimately get published on `learn.netdata.cloud`. The |
| 178 | Learn-side mapping is driven by `<repo>/docs/.map/map.yaml`; |
| 179 | for integration pages, the relevant `<!--startmeta` block |
| 180 | inside each generated `.md` is what Learn's ingest reads. |