master
md 9.83 KB

Artifacts and banners

Every file the integrations pipeline reads or writes, with banner conventions and edit rules.

Tracked vs runtime artifacts

Path Producer Tracked in git? Auto-regenerated by
integrations/integrations.js gen_integrations.py:1311-1325 NO -- gitignored (.gitignore:162) Every CI run; rm-ed by the workflow before opening the PR
integrations/integrations.json gen_integrations.py:1311-1325 (clean variant) NO -- gitignored (.gitignore:163) as above
<plugin-dir>/integrations/<slug>.md (per-integration) gen_docs_integrations.py YES (committed) generate-integrations.yml auto-PR; verified in check-markdown.yml
<plugin-dir>/README.md (symlink) gen_docs_integrations.py:make_symlinks (:527-544) YES (the symlink itself) as above
src/health/notifications/<dir>/README.md (DIRECT, not a symlink) gen_docs_integrations.py:agent_notification branch (:488-496) YES as above
src/collectors/COLLECTORS.md gen_doc_collector_page.py YES as above
src/collectors/SECRETS.md gen_doc_secrets_page.py YES as above
src/collectors/SERVICE-DISCOVERY.md gen_doc_service_discovery_page.py YES NOT in generate-integrations.yml -- manual or cmake render-docs only -- see gotchas.md
src/go/plugin/ibm.d/modules/<m>/metadata.yaml ibm.d docgen YES manual go generate ./... -- never CI
src/go/plugin/ibm.d/modules/<m>/README.md ibm.d docgen YES as above
src/go/plugin/ibm.d/modules/<m>/config_schema.json ibm.d docgen YES as above
src/go/plugin/ibm.d/modules/<m>/contexts/zz_generated_contexts.go ibm.d metricgen YES as above
Hand-written metadata.yaml (non-ibm.d), config_schema.json, stock .conf, health.d/<...>.conf, hand-written README.md collector author YES none -- author edits + commits manually
<collector-dir>/taxonomy.yaml collector author / gen_taxonomy_seed.py starter output YES validated by check-markdown.yml; not auto-authored
integrations/taxonomy/sections.yaml taxonomy framework author YES validated by gen_taxonomy.py
integrations/taxonomy/icons.yaml taxonomy framework author YES validated by gen_taxonomy.py
integrations/taxonomy.json gen_taxonomy.py NO -- gitignored (.gitignore:164) generated locally/CI; downstream cloud-frontend contract

integrations/integrations.js

Top of file (integrations/templates/integrations.js:1-2):

// DO NOT EDIT THIS FILE DIRECTLY
// It gets generated by integrations/gen_integrations.py in the Netdata repo

Body shape:

export const categories = [/* JSON array of category trees */];
export const integrations = [/* JSON array of integration objects */];

The body uses the rendered-with-folding/{% details %} markers (the clean=False variant). Consumed by the cloud-frontend dashboard which has its own renderer that interprets these markers.

integrations/integrations.json

Pure JSON, no banner. Top-level shape:

{ "categories": [...], "integrations": [...] }

Uses the clean=True variant (markers stripped). Consumed by scripts/checkIntegrations.js in the dashboard repo (validation fetch from https://raw.githubusercontent.com/netdata/netdata/master/integrations/integrations.json) and by anything that needs GitHub-renderable markdown.

Per-integration .md (collector, exporter, secretstore, etc.)

Every per-integration .md opens with a <!--startmeta HTML-comment block. This block is consumed by the Learn ingest pipeline (<repo>/docs/.map/map.yaml row matching is by custom_edit_url).

Schema:

<!--startmeta
custom_edit_url: "https://github.com/netdata/netdata/edit/master/<plugin-dir>/integrations/<slug>.md"
meta_yaml: "https://github.com/netdata/netdata/edit/master/<plugin-dir>/metadata.yaml"
sidebar_label: "<display name>"
learn_status: "Published"
learn_rel_path: "<Learn category path>"
keywords: ['k1', 'k2', ...]                                    # only when metadata has keywords
most_popular: true                                             # only when metadata sets it
message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE"
endmeta-->

The exact wording of message: varies by type. From gen_docs_integrations.py:

Integration type message: text
collector DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE
exporter ... GENERATED BY THE EXPORTER'S metadata.yaml FILE
agent_notification, cloud_notification ... GENERATED BY THE NOTIFICATION'S metadata.yaml FILE
logs ... GENERATED BY THE LOGS' metadata.yaml FILE
authentication ... GENERATED BY THE AUTHENTICATION'S metadata.yaml FILE
secretstore ... GENERATED BY THE SECRETSTORE'S metadata.yaml FILE
service_discovery ... GENERATED BY THE SERVICE DISCOVERY DISCOVERER'S metadata.yaml FILE

After the banner, the script injects a community-or-Netdata badge image before the first ## (:422-437). Badge color depends on meta.community boolean.

learn_rel_path mapping

The first category in meta.monitored_instance.categories is remapped to a Learn navigation path (gen_docs_integrations.py per-type branches):

Integration type Mapping
collector data-collection.<x> -> Collecting Metrics/Collectors/<x>
exporter hardcoded Exporting Metrics/Connectors
agent_notification <x> with notifications -> Alerts & Notifications/Notifications
cloud_notification same as above
logs <x> with logs -> Logs
authentication <x> with authentication -> Netdata Cloud/Authentication & Authorization/Cloud Authentication & Authorization Integrations
secretstore hardcoded Collecting Metrics/Secrets Management/Secret Stores
service_discovery hardcoded Collecting Metrics/Service Discovery

When the symlink rule fires (single-integration directory), the README is a symlink to integrations/<slug>.md. The symlink target's banner is the per-integration banner above, with two adjustments: - custom_edit_url points at the README path, not the integration file path -- so editing through Learn opens the README. - The body has {element}/{symlinks[element]} replaced with {element}/README.md (:542-544) so internal anchor refs don't break.

src/health/notifications/<dir>/README.md (direct case)

Same banner as above. Written DIRECTLY, not as a symlink. Keep in mind for the collector consistency rule -- the README.md is the generated artifact.

src/collectors/COLLECTORS.md, SECRETS.md, SERVICE-DISCOVERY.md

No generated-file banner. COLLECTORS.md currently opens with <!-- markdownlint-disable-file --> and then its marketing header (# Monitor anything with Netdata). SECRETS.md and SERVICE-DISCOVERY.md open with their marketing headers (# Secrets Management, # Service Discovery). This is a known oversight -- there is no DO-NOT-EDIT marker on these files even though they are generated. Editing them directly will be silently overwritten on the next CI run (for COLLECTORS.md and SECRETS.md). For SERVICE-DISCOVERY.md the absence of CI wiring means manual edits stick until someone runs the script.

COLLECTORS.md is the Learn page titled "Monitor anything with Netdata". It is generated by integrations/gen_doc_collector_page.py from integrations/integrations.js. The generator groups normal collector entries by children of data-collection and also treats the top-level flows category as a first-class section named Network Flows; without that special case, NetFlow / IPFIX / sFlow and flow enrichment integrations either appear in the wrong generic section or are easy to miss.

To update the page, do not edit the generated table rows in src/collectors/COLLECTORS.md. Edit the source metadata.yaml / integrations/categories.yaml or the generator, then run:

python3 integrations/gen_integrations.py
python3 integrations/gen_doc_collector_page.py

ibm.d-generated files

File Banner
metadata.yaml First line: # Generated metadata.yaml for <module> module (template at src/go/plugin/ibm.d/docgen/main.go:562)
zz_generated_contexts.go Filename prefix zz_generated_. Package doc at <module>/contexts/doc.go:2-3 declares DO NOT EDIT zz_generated_contexts.go - it is auto-generated from contexts.yaml.
README.md Banner depends on docgen's readme template. Includes module info, metric tables, config tables.
config_schema.json Pure JSON, no banner.

See ibm-d.md for the full chain.

Maintainer rules

  1. Per-integration .md files: NEVER edit by hand. Edit the source metadata.yaml, regenerate, commit.
  2. Symlinked README.md files: NEVER edit by hand. Same reason -- they point at the generated .md.
  3. src/health/notifications/<dir>/README.md: NEVER edit by hand. Direct generated file, looks identical to a normal README -- the <!--startmeta banner is the giveaway.
  4. src/collectors/COLLECTORS.md / SECRETS.md: NEVER edit the dynamic tables. Edit the source metadata.yaml. The static prose ("# Monitor anything with Netdata", the secret-store onboarding text) is baked into the generator scripts -- to change static prose, edit gen_doc_collector_page.py / gen_doc_secrets_page.py and commit.
  5. src/collectors/SERVICE-DISCOVERY.md: same rule. Caveat: the script is not in CI, so manual regen is required after editing source metadata until that gap is fixed.
  6. ibm.d generated files: NEVER edit. Edit contexts.yaml, config.go, or module.yaml instead and re-run go generate.