Render flow integration content sections (#22454)
Costa Tsaousis committed
May 8, 2026 at 18:44 UTC
2cbbe087a03d56073b67f7533dfbcbf00f71d1b4
4 files changed
+160
-8
.agents/skills/integrations-lifecycle/how-tos/adding-new-integration-type.md
+16
-6
@@ -29,7 +29,7 @@ Examples that would NOT qualify (use an existing type instead):
29
30
Clone the closest existing schema and trim/adjust:
31
32
-- For data-bearing types (something that produces records / metrics / events): start from `collector.json`. Trim fields that don't apply (e.g., flows have no `metrics` or `alerts` sections in the traditional sense).
32
+- For data-bearing types (something that produces records / metrics / events): start from `collector.json`. If the new type keeps collector-style sections such as `metrics`, `alerts`, or `functions`, those sections still need to be rendered to markdown before publication, even when they describe non-metric data or say that no alerts exist.
33
- For thin types (a config target with overview + setup + troubleshooting): start from `logs.json` or `exporter.json`.
34
35
The schema field set drives validation. Anything not in the schema is rejected as an unknown property when `additionalProperties: false` is set on the relevant object.
@@ -43,6 +43,14 @@ There are typically 3-5 places that need updating:
43
- The render-keys table — which sections (overview, setup, troubleshooting, alerts, metrics, ...) the type renders.
44
- The categorisation in `integrations.js` — making sure the new type ends up in the flat `integrations` array with a correct `integration_type` field.
45
46
+Render keys are a downstream contract, not only a docs-output choice. Any
47
+section name that website or cloud-frontend treats as markdown content must
48
+be emitted as a string in `integrations.json` and `integrations.js`. If a new
49
+type reuses `collector.json`, default to the collector render keys unless
50
+there is evidence a key is invalid for that type. Publishing raw YAML
51
+objects or arrays under keys such as `metrics` or `alerts` breaks website
52
+Hugo templates and cloud-frontend integration tabs/checkers.
53
+
54
### 3. Pipeline rendering (`integrations/gen_docs_integrations.py`)
55
56
Add a branch for the new type in the main loop. The branch typically:
@@ -120,7 +128,7 @@ Three downstream repos may need touching, in roughly decreasing likelihood:
128
129
- **`netdata/website`** (`~/src/netdata/website`): the daily `update-integrations.yml` workflow renders marketing cards from `integrations.json`. Cards usually appear automatically. But pages that reference the type explicitly (FAQ entries, solution pages) may need rewriting if the new type changes the story (e.g., "we don't do flows" → "we do flows natively").
130
- **`netdata/learn`** (`~/src/netdata/learn`): no PR usually needed. Learn ingest reads `map.yaml` + the `<!--startmeta-->` markers in the generated `.md` files. Sidebar regenerates automatically. ~3 hours after netdata-repo merge.
123
-- **`netdata/dashboard/cloud-frontend`** (`~/src/dashboard/cloud-frontend`): no PR usually needed. The catalog rendering in `src/domains/integrations/` is fully data-driven from `integrations.js`. New `integration_type` values appear automatically. Special UI (a dedicated tab elsewhere in the dashboard) is a separate concern.
131
+- **`netdata/dashboard/cloud-frontend`** (`~/src/dashboard/cloud-frontend`): cards/categories are mostly data-driven, but content tabs and validation scripts still assume rendered markdown strings for standard section keys. Before merging a new type, inspect the generated `integrations.js` shape against `src/domains/integrations/components/content/integration/tabs.js`, `src/components/markdown/useRenderableTree.js`, and `scripts/checkIntegrations.js`. Special UI (a dedicated tab elsewhere in the dashboard) is a separate concern.
132
133
## Verification checklist
134
@@ -128,10 +136,12 @@ After all the changes:
136
137
1. `python3 integrations/gen_integrations.py` — exits 0, regenerates `integrations.js` and `integrations.json`. The new type appears in the flat list, in the new top-level category, with the correct count.
138
2. `python3 integrations/gen_docs_integrations.py` — exits 0, generates per-integration `.md` files. Each has `<!--startmeta-->` with the correct `learn_rel_path`.
131
-3. Manual inspection of one generated `.md` — frontmatter complete (`custom_edit_url`, `meta_yaml`, `sidebar_label`, `learn_status: Published`, `learn_rel_path`, `keywords`, `message: "DO NOT EDIT..."`).
132
-4. `grep` the categories tree in `integrations.json` — the new top-level node exists with the new entries.
133
-5. Open `learn.netdata.cloud` after the next ingest cycle (~3 hours after merge) — confirm the new section renders in the sidebar.
134
-6. Open the in-app integrations catalog — confirm the new top-level filter appears in the sidebar with the correct entries.
139
+3. Type-check the generated artifacts — every standard content key consumed as markdown downstream (`overview`, `setup`, `troubleshooting`, `alerts`, `metrics`, `functions`, `related_resources`, and type-specific equivalents) is a string, not an object or array.
140
+4. Manual inspection of one generated `.md` — frontmatter complete (`custom_edit_url`, `meta_yaml`, `sidebar_label`, `learn_status: Published`, `learn_rel_path`, `keywords`, `message: "DO NOT EDIT..."`).
141
+5. `grep` the categories tree in `integrations.json` — the new top-level node exists with the new entries.
142
+6. Run or locally emulate the website integrations update/build when the new type introduces a new top-level category or a new section shape. The website PR must build with the Hugo version pinned in `netlify.toml`.
143
+7. Open `learn.netdata.cloud` after the next ingest cycle (~3 hours after merge) — confirm the new section renders in the sidebar.
144
+8. Open the in-app integrations catalog — confirm the new top-level filter appears in the sidebar with the correct entries and content tabs are not blank.
145
146
## What to commit, and where
147
.agents/skills/integrations-lifecycle/in-app-contract.md
+18
-2
@@ -72,6 +72,16 @@ export const integrations = [
72
];
73
```
74
75
+All public content sections consumed by downstream renderers must be
76
+markdown strings in the generated artifacts, even when the source
77
+`metadata.yaml` stores them as structured YAML objects or arrays.
78
+Examples: collector-like `metrics`, `alerts`, `functions`, `overview`,
79
+`setup`, `troubleshooting`, and `related_resources` must pass through
80
+the renderer before reaching `integrations.js` / `integrations.json`.
81
+Leaving raw objects or arrays in these fields breaks the website Hugo
82
+renderer and produces blank tabs or link-check failures in
83
+cloud-frontend.
84
+
85
The dashboard's renderer interprets the `{% details %}` /
86
`{% /details %}` markers embedded in the rendered text (the
87
`clean=False` variant is the one written into the `.js` file).
@@ -128,10 +138,16 @@ In practice this means:
138
keys) is a contract. Avoid breaking changes; coordinate
139
with the cloud-frontend team if a key must be renamed or
140
removed.
131
-2. **Custom Jinja markers in metadata** (`{% details %}`,
141
+2. **Render structured metadata before publication**. A new
142
+ integration type that reuses collector-style sections must
143
+ include every structured content key in its render-key list.
144
+ Do not publish raw `metrics` objects, `alerts` arrays, or
145
+ similar YAML structures under the public markdown section
146
+ names.
147
+3. **Custom Jinja markers in metadata** (`{% details %}`,
148
`{% relatedResource %}`, `{% if %}`) are part of the
149
contract. The dashboard's renderer interprets them. Test
150
any new marker against both surfaces before relying on it.
135
-3. **Do not commit `integrations.js` to this repo**. It is
151
+4. **Do not commit `integrations.js` to this repo**. It is
152
gitignored on purpose; the dashboard pulls fresh on each
153
build.
.agents/sow/done/SOW-0014-20260506-netflow-sflow-ipfix-documentation-guide.md
+123
@@ -8,6 +8,14 @@ Reopened 2026-05-07 after the netlify deploy preview for learn PR #2852 surfaced
8
9
Reopened 2026-05-08 after PR #22449 review and CI reported additional issues after the SOW had been marked completed and moved to `done/`. The open items are tracked in `## Regression - 2026-05-08` and include automated review threads, `yamllint`, `check-documentation`, Codacy triage, a code-only review subagent requested by the user, and a new user-requested local Learn preview skill/workflow.
10
11
+Reopened 2026-05-08 after the merged integration artifacts broke downstream
12
+publishing contracts for the website and in-app integrations. The flow
13
+integration schema delegates to collector metadata, but the flow renderer did
14
+not render the collector-style `metrics` and `alerts` sections, leaving raw
15
+objects/arrays in `integrations.json` and `integrations.js`.
16
+The regression was repaired and revalidated on 2026-05-08; see
17
+`## Regression - 2026-05-08 - Flow Integration Section Rendering`.
18
+
19
## Requirements
20
21
### Purpose
@@ -2876,3 +2884,118 @@ Repairs:
2884
- **SOW lifecycle**: SOW 14 repair is complete; status is `completed`, and
2885
the file is moved back to `.agents/sow/done/` in the same commit as the
2886
repair.
2887
+
2888
+## Regression - 2026-05-08 - Flow Integration Section Rendering
2889
+
2890
+### What broke
2891
+
2892
+The merged network-flow integration artifacts expose `metrics` as a JSON object
2893
+and `alerts` as a JSON array for `integration_type: flows` entries. Downstream
2894
+surfaces expect rendered markdown strings for integration content sections.
2895
+
2896
+Evidence:
2897
+
2898
+- `integrations/schemas/flows.json` delegates to `collector.json`, so flow
2899
+ metadata legitimately contains collector-style `metrics` and `alerts`.
2900
+- `integrations/gen_integrations.py` `FLOWS_RENDER_KEYS` rendered only
2901
+ `overview`, `related_resources`, `setup`, and `troubleshooting`, leaving
2902
+ `metrics` and `alerts` in their source YAML shape.
2903
+- The website PR generated from the merged artifacts failed production-pinned
2904
+ Hugo `0.140.0` because `themes/tailwind/layouts/partials/integration-tabs.html`
2905
+ calls `markdownify` on `.integration.metrics`.
2906
+- The in-app integrations renderer in cloud-frontend adds tabs for truthy
2907
+ `metrics` and `alerts`, while its Markdoc wrapper parses only string input.
2908
+ The result would be blank Metrics and Alerts tabs after the next data sync.
2909
+- The cloud-frontend integration link checker calls `.match()` on markdown
2910
+ fields, so raw objects/arrays in `metrics`/`alerts` can break that check too.
2911
+
2912
+### Why previous validation missed it
2913
+
2914
+The earlier closeout validated the Netdata integrations generator and local
2915
+Learn ingest/build, but did not validate the website Hugo renderer or the
2916
+cloud-frontend integrations consumer against the newly introduced
2917
+`integration_type: flows` artifacts. `gen_integrations.py` itself accepted the
2918
+raw structured fields because they are schema-valid before rendering.
2919
+
2920
+### Repair plan
2921
+
2922
+Render flow `metrics` and `alerts` through the same markdown templates used by
2923
+collector integrations. This keeps the source metadata schema unchanged and
2924
+restores the downstream contract: content sections in `integrations.json` and
2925
+`integrations.js` are markdown strings.
2926
+
2927
+### Validation plan
2928
+
2929
+- Run `python3 integrations/gen_integrations.py`.
2930
+- Verify all `flows` entries in `integrations/integrations.json` have string
2931
+ `metrics` and string `alerts`.
2932
+- Run `python3 integrations/gen_docs_integrations.py`.
2933
+- Run `python3 integrations/gen_doc_collector_page.py`.
2934
+- Rebuild the website PR artifacts with the repaired generated
2935
+ `integrations.json` and production-pinned Hugo `0.140.0`.
2936
+- Verify cloud-frontend's current renderer and link checker receive strings
2937
+ for `flows` `metrics` and `alerts`.
2938
+
2939
+### Artifact updates needed
2940
+
2941
+- **Code**: update `integrations/gen_integrations.py` flow render keys.
2942
+- **Generated artifacts**: regenerate `integrations/integrations.json`,
2943
+ `integrations/integrations.js`, per-integration markdown, and
2944
+ `src/collectors/COLLECTORS.md` if the generator changes them.
2945
+- **Runtime project skills**: update `integrations-lifecycle` if the durable
2946
+ downstream contract was not already documented clearly enough.
2947
+- **Specs**: no product behavior change is expected; this repairs generated
2948
+ publishing artifacts.
2949
+- **End-user/operator docs**: no content change is expected beyond generated
2950
+ artifacts.
2951
+
2952
+### Repair completed
2953
+
2954
+`integrations/gen_integrations.py` now renders the flow `alerts`, `metrics`,
2955
+and `functions` sections through the standard templates, matching the
2956
+collector-like schema that `flows.json` delegates to.
2957
+
2958
+### Validation evidence
2959
+
2960
+- `python3 integrations/gen_integrations.py` passed.
2961
+- `python3 integrations/gen_docs_integrations.py` passed.
2962
+- `python3 integrations/gen_doc_collector_page.py` passed.
2963
+- Flow artifact type check passed: every `integration_type: flows` entry in
2964
+ `integrations/integrations.json` has string `metrics`, string `alerts`, and
2965
+ string `functions`.
2966
+- Non-deploy markdown-field type check passed: no non-deploy integration emits
2967
+ raw object or array values for `overview`, `setup`, `troubleshooting`,
2968
+ `alerts`, `metrics`, `functions`, or `related_resources`.
2969
+- Website validation passed in a temporary copy of PR #1212 using the repaired
2970
+ `integrations.json` and production-pinned Hugo `0.140.0`: `hugo --gc
2971
+ --minify` built 3141 pages successfully.
2972
+- Cloud-frontend compatibility check passed against the repaired
2973
+ `integrations.json`: all flow markdown fields inspected by
2974
+ `scripts/checkIntegrations.js` were strings, and `getMarkdownUrls()` extracted
2975
+ 52 markdown URLs without throwing.
2976
+- `git diff --check` passed.
2977
+- `.agents/sow/audit.sh` exited 2 because of a pre-existing unrelated
2978
+ sensitive-pattern warning in
2979
+ `.agents/skills/mirror-netdata-repos/SKILL.md`; SOW status/directory
2980
+ consistency passed and SOW 14 reports `completed` in `.agents/sow/done/`.
2981
+
2982
+### Artifact maintenance gate
2983
+
2984
+- **AGENTS.md**: no update needed; this does not change repo-wide workflow.
2985
+- **Runtime project skills**: updated `integrations-lifecycle` with the
2986
+ downstream markdown-string contract and the new-integration-type validation
2987
+ checklist.
2988
+- **Specs**: no update needed; product behavior and public data semantics did
2989
+ not change.
2990
+- **End-user/operator docs**: no hand-authored user docs changed; this repairs
2991
+ generated publishing artifacts.
2992
+- **End-user/operator skills**: no update needed; no operator workflow changed.
2993
+- **SOW lifecycle**: SOW 14 reopened as a regression, status returned to
2994
+ `completed`, and the file is moved back to `.agents/sow/done/` in the same
2995
+ commit as the repair.
2996
+
2997
+### Follow-up mapping
2998
+
2999
+No deferred follow-up remains for this regression. The website integration PR
3000
+must be regenerated after this repair reaches `netdata/master`; that is the
3001
+normal downstream propagation path rather than a separate source-code TODO.
integrations/gen_integrations.py
+3
@@ -83,6 +83,9 @@ COLLECTOR_RENDER_KEYS = [
83
]
84
85
FLOWS_RENDER_KEYS = [
86
+ 'alerts',
87
+ 'metrics',
88
+ 'functions',
89
'overview',
90
'related_resources',
91
'setup',