| 1 | # Artifacts and banners |
| 2 | |
| 3 | Every file the integrations pipeline reads or writes, with |
| 4 | banner conventions and edit rules. |
| 5 | |
| 6 | ## Tracked vs runtime artifacts |
| 7 | |
| 8 | | Path | Producer | Tracked in git? | Auto-regenerated by | |
| 9 | |---|---|---|---| |
| 10 | | `integrations/integrations.js` | `gen_integrations.py:1311-1325` | NO -- gitignored (`.gitignore:162`) | Every CI run; `rm`-ed by the workflow before opening the PR | |
| 11 | | `integrations/integrations.json` | `gen_integrations.py:1311-1325` (clean variant) | NO -- gitignored (`.gitignore:163`) | as above | |
| 12 | | `<plugin-dir>/integrations/<slug>.md` (per-integration) | `gen_docs_integrations.py` | **YES (committed)** | `generate-integrations.yml` auto-PR; verified in `check-markdown.yml` | |
| 13 | | `<plugin-dir>/README.md` (symlink) | `gen_docs_integrations.py:make_symlinks` (`:527-544`) | **YES** (the symlink itself) | as above | |
| 14 | | `src/health/notifications/<dir>/README.md` (DIRECT, not a symlink) | `gen_docs_integrations.py:agent_notification branch` (`:488-496`) | **YES** | as above | |
| 15 | | `src/collectors/COLLECTORS.md` | `gen_doc_collector_page.py` | **YES** | as above | |
| 16 | | `src/collectors/SECRETS.md` | `gen_doc_secrets_page.py` | **YES** | as above | |
| 17 | | `src/collectors/SERVICE-DISCOVERY.md` | `gen_doc_service_discovery_page.py` | **YES** | **NOT** in `generate-integrations.yml` -- manual or cmake `render-docs` only -- see `gotchas.md` | |
| 18 | | `src/go/plugin/ibm.d/modules/<m>/metadata.yaml` | ibm.d `docgen` | **YES** | manual `go generate ./...` -- never CI | |
| 19 | | `src/go/plugin/ibm.d/modules/<m>/README.md` | ibm.d `docgen` | **YES** | as above | |
| 20 | | `src/go/plugin/ibm.d/modules/<m>/config_schema.json` | ibm.d `docgen` | **YES** | as above | |
| 21 | | `src/go/plugin/ibm.d/modules/<m>/contexts/zz_generated_contexts.go` | ibm.d `metricgen` | **YES** | as above | |
| 22 | | Hand-written `metadata.yaml` (non-ibm.d), `config_schema.json`, stock `.conf`, `health.d/<...>.conf`, hand-written `README.md` | collector author | **YES** | none -- author edits + commits manually | |
| 23 | | `<collector-dir>/taxonomy.yaml` | collector author / `gen_taxonomy_seed.py` starter output | **YES** | validated by `check-markdown.yml`; not auto-authored | |
| 24 | | `integrations/taxonomy/sections.yaml` | taxonomy framework author | **YES** | validated by `gen_taxonomy.py` | |
| 25 | | `integrations/taxonomy/icons.yaml` | taxonomy framework author | **YES** | validated by `gen_taxonomy.py` | |
| 26 | | `integrations/taxonomy.json` | `gen_taxonomy.py` | NO -- gitignored (`.gitignore:164`) | generated locally/CI; downstream cloud-frontend contract | |
| 27 | |
| 28 | ## Banner conventions per file kind |
| 29 | |
| 30 | ### `integrations/integrations.js` |
| 31 | |
| 32 | Top of file (`integrations/templates/integrations.js:1-2`): |
| 33 | |
| 34 | ```js |
| 35 | // DO NOT EDIT THIS FILE DIRECTLY |
| 36 | // It gets generated by integrations/gen_integrations.py in the Netdata repo |
| 37 | ``` |
| 38 | |
| 39 | Body shape: |
| 40 | |
| 41 | ```js |
| 42 | export const categories = [/* JSON array of category trees */]; |
| 43 | export const integrations = [/* JSON array of integration objects */]; |
| 44 | ``` |
| 45 | |
| 46 | The body uses the rendered-with-folding/`{% details %}` markers |
| 47 | (the `clean=False` variant). Consumed by the cloud-frontend |
| 48 | dashboard which has its own renderer that interprets these |
| 49 | markers. |
| 50 | |
| 51 | ### `integrations/integrations.json` |
| 52 | |
| 53 | Pure JSON, no banner. Top-level shape: |
| 54 | |
| 55 | ```json |
| 56 | { "categories": [...], "integrations": [...] } |
| 57 | ``` |
| 58 | |
| 59 | Uses the `clean=True` variant (markers stripped). Consumed by |
| 60 | `scripts/checkIntegrations.js` in the dashboard repo (validation |
| 61 | fetch from `https://raw.githubusercontent.com/netdata/netdata/master/integrations/integrations.json`) |
| 62 | and by anything that needs GitHub-renderable markdown. |
| 63 | |
| 64 | ### Per-integration `.md` (collector, exporter, secretstore, etc.) |
| 65 | |
| 66 | Every per-integration `.md` opens with a `<!--startmeta` |
| 67 | HTML-comment block. This block is consumed by the Learn ingest |
| 68 | pipeline (`<repo>/docs/.map/map.yaml` row matching is by |
| 69 | `custom_edit_url`). |
| 70 | |
| 71 | Schema: |
| 72 | |
| 73 | ```markdown |
| 74 | <!--startmeta |
| 75 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/<plugin-dir>/integrations/<slug>.md" |
| 76 | meta_yaml: "https://github.com/netdata/netdata/edit/master/<plugin-dir>/metadata.yaml" |
| 77 | sidebar_label: "<display name>" |
| 78 | learn_status: "Published" |
| 79 | learn_rel_path: "<Learn category path>" |
| 80 | keywords: ['k1', 'k2', ...] # only when metadata has keywords |
| 81 | most_popular: true # only when metadata sets it |
| 82 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" |
| 83 | endmeta--> |
| 84 | ``` |
| 85 | |
| 86 | The exact wording of `message:` varies by type. From |
| 87 | `gen_docs_integrations.py`: |
| 88 | |
| 89 | | Integration type | `message:` text | |
| 90 | |---|---| |
| 91 | | `collector` | `DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE` | |
| 92 | | `exporter` | `... GENERATED BY THE EXPORTER'S metadata.yaml FILE` | |
| 93 | | `agent_notification`, `cloud_notification` | `... GENERATED BY THE NOTIFICATION'S metadata.yaml FILE` | |
| 94 | | `logs` | `... GENERATED BY THE LOGS' metadata.yaml FILE` | |
| 95 | | `authentication` | `... GENERATED BY THE AUTHENTICATION'S metadata.yaml FILE` | |
| 96 | | `secretstore` | `... GENERATED BY THE SECRETSTORE'S metadata.yaml FILE` | |
| 97 | | `service_discovery` | `... GENERATED BY THE SERVICE DISCOVERY DISCOVERER'S metadata.yaml FILE` | |
| 98 | |
| 99 | After the banner, the script injects a community-or-Netdata |
| 100 | badge image before the first `##` (`:422-437`). Badge color |
| 101 | depends on `meta.community` boolean. |
| 102 | |
| 103 | ### `learn_rel_path` mapping |
| 104 | |
| 105 | The first category in `meta.monitored_instance.categories` is |
| 106 | remapped to a Learn navigation path (`gen_docs_integrations.py` |
| 107 | per-type branches): |
| 108 | |
| 109 | | Integration type | Mapping | |
| 110 | |---|---| |
| 111 | | `collector` | `data-collection.<x>` -> `Collecting Metrics/Collectors/<x>` | |
| 112 | | `exporter` | hardcoded `Exporting Metrics/Connectors` | |
| 113 | | `agent_notification` | `<x>` with `notifications` -> `Alerts & Notifications/Notifications` | |
| 114 | | `cloud_notification` | same as above | |
| 115 | | `logs` | `<x>` with `logs` -> `Logs` | |
| 116 | | `authentication` | `<x>` with `authentication` -> `Netdata Cloud/Authentication & Authorization/Cloud Authentication & Authorization Integrations` | |
| 117 | | `secretstore` | hardcoded `Collecting Metrics/Secrets Management/Secret Stores` | |
| 118 | | `service_discovery` | hardcoded `Collecting Metrics/Service Discovery` | |
| 119 | |
| 120 | ### `<plugin-dir>/README.md` (symlink case) |
| 121 | |
| 122 | When the symlink rule fires (single-integration directory), the |
| 123 | README is a symlink to `integrations/<slug>.md`. The symlink |
| 124 | target's banner is the per-integration banner above, with two |
| 125 | adjustments: |
| 126 | - `custom_edit_url` points at the README path, not the |
| 127 | integration file path -- so editing through Learn opens the |
| 128 | README. |
| 129 | - The body has `{element}/{symlinks[element]}` replaced with |
| 130 | `{element}/README.md` (`:542-544`) so internal anchor refs |
| 131 | don't break. |
| 132 | |
| 133 | ### `src/health/notifications/<dir>/README.md` (direct case) |
| 134 | |
| 135 | Same banner as above. Written DIRECTLY, not as a symlink. Keep |
| 136 | in mind for the collector consistency rule -- the README.md is |
| 137 | the generated artifact. |
| 138 | |
| 139 | ### `src/collectors/COLLECTORS.md`, `SECRETS.md`, `SERVICE-DISCOVERY.md` |
| 140 | |
| 141 | **No generated-file banner.** `COLLECTORS.md` currently opens |
| 142 | with `<!-- markdownlint-disable-file -->` and then its marketing |
| 143 | header (`# Monitor anything with Netdata`). `SECRETS.md` and |
| 144 | `SERVICE-DISCOVERY.md` open with their marketing headers |
| 145 | (`# Secrets Management`, `# Service Discovery`). This is a |
| 146 | known oversight -- there is no DO-NOT-EDIT marker on these |
| 147 | files even though they are generated. Editing them directly |
| 148 | will be silently overwritten on the next CI run (for |
| 149 | COLLECTORS.md and SECRETS.md). For SERVICE-DISCOVERY.md the |
| 150 | absence of CI wiring means manual edits stick until someone |
| 151 | runs the script. |
| 152 | |
| 153 | `COLLECTORS.md` is the Learn page titled "Monitor anything with |
| 154 | Netdata". It is generated by `integrations/gen_doc_collector_page.py` |
| 155 | from `integrations/integrations.js`. The generator groups normal |
| 156 | collector entries by children of `data-collection` and also treats |
| 157 | the top-level `flows` category as a first-class section named |
| 158 | `Network Flows`; without that special case, NetFlow / IPFIX / sFlow |
| 159 | and flow enrichment integrations either appear in the wrong generic |
| 160 | section or are easy to miss. |
| 161 | |
| 162 | To update the page, do not edit the generated table rows in |
| 163 | `src/collectors/COLLECTORS.md`. Edit the source |
| 164 | `metadata.yaml` / `integrations/categories.yaml` or the |
| 165 | generator, then run: |
| 166 | |
| 167 | ```bash |
| 168 | python3 integrations/gen_integrations.py |
| 169 | python3 integrations/gen_doc_collector_page.py |
| 170 | ``` |
| 171 | |
| 172 | ### ibm.d-generated files |
| 173 | |
| 174 | | File | Banner | |
| 175 | |---|---| |
| 176 | | `metadata.yaml` | First line: `# Generated metadata.yaml for <module> module` (template at `src/go/plugin/ibm.d/docgen/main.go:562`) | |
| 177 | | `zz_generated_contexts.go` | Filename prefix `zz_generated_`. Package doc at `<module>/contexts/doc.go:2-3` declares `DO NOT EDIT zz_generated_contexts.go - it is auto-generated from contexts.yaml`. | |
| 178 | | `README.md` | Banner depends on `docgen`'s readme template. Includes module info, metric tables, config tables. | |
| 179 | | `config_schema.json` | Pure JSON, no banner. | |
| 180 | |
| 181 | See `ibm-d.md` for the full chain. |
| 182 | |
| 183 | ## Maintainer rules |
| 184 | |
| 185 | 1. **Per-integration `.md` files: NEVER edit by hand.** Edit |
| 186 | the source `metadata.yaml`, regenerate, commit. |
| 187 | 2. **Symlinked `README.md` files: NEVER edit by hand.** Same |
| 188 | reason -- they point at the generated `.md`. |
| 189 | 3. **`src/health/notifications/<dir>/README.md`: NEVER edit by |
| 190 | hand.** Direct generated file, looks identical to a normal |
| 191 | README -- the `<!--startmeta` banner is the giveaway. |
| 192 | 4. **`src/collectors/COLLECTORS.md` / `SECRETS.md`: NEVER edit |
| 193 | the dynamic tables.** Edit the source `metadata.yaml`. The |
| 194 | static prose ("# Monitor anything with Netdata", the |
| 195 | secret-store onboarding text) is baked into the generator |
| 196 | scripts -- to change static prose, edit |
| 197 | `gen_doc_collector_page.py` / `gen_doc_secrets_page.py` and |
| 198 | commit. |
| 199 | 5. **`src/collectors/SERVICE-DISCOVERY.md`: same rule.** Caveat: |
| 200 | the script is not in CI, so manual regen is required after |
| 201 | editing source metadata until that gap is fixed. |
| 202 | 6. **ibm.d generated files: NEVER edit.** Edit `contexts.yaml`, |
| 203 | `config.go`, or `module.yaml` instead and re-run |
| 204 | `go generate`. |