@cryptotaxi247 / netdata-1 / commits / bc4e83248

feat(integrations): add collector taxonomy framework POC (#22489)

Ilya Mashchenko committed May 15, 2026 at 11:43 UTC bc4e83248ccf1d0dab11215a722deaa6fc2dd641

41 files changed +6740 -273

.agents/skills/integrations-lifecycle/SKILL.md

+19 -13

@@ -1,6 +1,6 @@

1 ---

2 name: integrations-lifecycle

3 -description: Authoritative reference for Netdata's integrations pipeline -- how `metadata.yaml` drives per-integration pages, the `COLLECTORS.md`/`SECRETS.md`/`SERVICE-DISCOVERY.md` umbrellas, the `integrations.js` artifact consumed by the cloud-frontend, and per-integration `.md` files committed to the repo. Use when adding/modifying any integration (collector, exporter, agent or cloud notification, authentication, secretstore, service-discovery, log type, deploy method); editing `metadata.yaml`; checking whether `integrations/*.md` should be hand-edited; reading the four generator scripts under `integrations/`, schemas under `integrations/schemas/`, templates under `integrations/templates/`, the workflows `generate-integrations.yml` or `check-markdown.yml`; ibm.d modules where `metadata.yaml` is generated from `contexts.yaml`; the 5-file consistency rule (metadata.yaml + config_schema.json + stock conf + alerts + README move together).

3 +description: Authoritative reference for Netdata's integrations pipeline -- how `metadata.yaml` drives per-integration pages, collector `taxonomy.yaml` drives dashboard TOC placement, the `COLLECTORS.md`/`SECRETS.md`/`SERVICE-DISCOVERY.md` umbrellas, the `integrations.js` and `integrations/taxonomy.json` artifacts consumed by downstream systems, and per-integration `.md` files committed to the repo. Use when adding/modifying any integration (collector, exporter, agent or cloud notification, authentication, secretstore, service-discovery, log type, deploy method); editing `metadata.yaml` or `taxonomy.yaml`; checking whether `integrations/*.md` should be hand-edited; reading generator scripts under `integrations/`, schemas under `integrations/schemas/`, taxonomy registries under `integrations/taxonomy/`, templates under `integrations/templates/`, the workflows `generate-integrations.yml` or `check-markdown.yml`; ibm.d modules where `metadata.yaml` is generated from `contexts.yaml`; the collector-consistency rule (metadata.yaml + taxonomy.yaml + config_schema.json + stock conf + alerts + README move together).

4 ---

6 # integrations-lifecycle

@@ -8,10 +8,12 @@ description: Authoritative reference for Netdata's integrations pipeline -- how

8 This skill is the **single place** to learn how Netdata's

9 integrations pipeline works end to end. It documents:

11 -- the four-stage generator pipeline rooted in

11 +- the generator pipeline rooted in

12 `integrations/gen_integrations.py`;

13 -- the 12 JSON-Schema contracts every `metadata.yaml` is validated

14 - against;

13 +- the collector taxonomy pipeline rooted in

14 + `integrations/gen_taxonomy.py`;

15 +- the JSON-Schema contracts every `metadata.yaml` and

16 + `taxonomy.yaml` is validated against;

17 - every artifact the pipeline produces (gitignored runtime files

18 AND committed `.md` documentation);

19 - the `<!--startmeta` banner conventions and DO-NOT-EDIT rules;

@@ -19,9 +21,10 @@ integrations pipeline works end to end. It documents:

21 (`generate-integrations.yml`, `check-markdown.yml`);

22 - the secondary ibm.d generation chain

23 (`contexts.yaml` -> `metadata.yaml`);

22 -- the contract by which the cloud-frontend dashboard consumes

23 - `integrations.js`;

24 -- the collector-consistency rule (5 files must move together)

24 +- the contract by which downstream dashboard code consumes

25 + `integrations.js` and, when opted in, `integrations/taxonomy.json`;

26 +- the collector-consistency rule (`taxonomy.yaml` moves with

27 + metadata and docs)

28 and what is and is NOT enforced by tooling;

29 - every surprising/dead/edge-case behavior an assistant or

30 maintainer is likely to hit.

@@ -79,10 +82,11 @@ does the in-app integrations page get its data?".

82 regenerate locally and include the changes in the same PR;

83 that is preferred to avoid two PRs per change.

82 -4. **The five-file consistency rule.** Anything that touches a

85 +4. **The collector consistency rule.** Anything that touches a

86 collector's runtime behavior MUST land in one PR with

87 matching changes to:

88 - `metadata.yaml` (the integration page driver),

89 + - `taxonomy.yaml` (dashboard TOC placement for chart contexts),

90 - `config_schema.json` (the dashboard's DYNCFG editor),

91 - the stock `.conf` (what `/etc/netdata/...` ships),

92 - `health.d/*.conf` (the alert definitions),

@@ -96,26 +100,28 @@ does the in-app integrations page get its data?".

100 `contexts.yaml` + `config.go` + `module.yaml` via

101 `go generate`. NEVER hand-edit them. See `ibm-d.md`.

102

99 -6. **The dashboard consumes `integrations/integrations.js`.**

103 +6. **The dashboard consumes generated integration artifacts.**

104 The cloud-frontend at

105 `${NETDATA_REPOS_DIR}/dashboard/cloud-frontend/` runs

106 `gen_integrations.py` in its own CI to copy

103 - `integrations.js` into its source tree. The contract is

107 + `integrations.js` into its source tree. The historical contract is

108 that `.js` file's exact shape:

109 `export const categories = [...]; export const integrations

106 - = [...]`. See `in-app-contract.md`.

110 + = [...]`. Collector taxonomy is emitted separately as

111 + `integrations/taxonomy.json` by `gen_taxonomy.py`; downstream

112 + consumers opt in to that JSON contract. See `in-app-contract.md`.

113

114 ## Table of contents

115

116 | Guide | Purpose |

117 |---|---|

118 | `pipeline.md` | The 4-stage pipeline graph, every script, every artifact, the CI workflows. |

113 -| `schema-reference.md` | Exhaustive per-field reference for all 12 JSON Schemas under `integrations/schemas/`. |

119 +| `schema-reference.md` | Per-field reference for JSON Schemas under `integrations/schemas/`, including collector taxonomy schemas. |

120 | `description-authoring.md` | Product-copy rules for `metadata.yaml` descriptions and the Monitor Anything table text. |

121 | `per-type-matrix.md` | One-row-per-integration-type quick lookup: source paths, validator, render keys, output location. |

122 | `artifacts-and-banners.md` | Every committed and gitignored artifact; banner conventions; symlink rules. |

123 | `ibm-d.md` | The `contexts.yaml` -> `metadata.yaml` chain for ibm.d modules. |

118 -| `consistency.md` | The 5-file consistency rule and what tooling enforces (mostly nothing). |

124 +| `consistency.md` | The collector consistency rule and what tooling enforces. |

125 | `in-app-contract.md` | How the cloud-frontend dashboard consumes `integrations.js`. |

126 | `gotchas.md` | Every surprise, dead-code reference, hardcoded marketing anchor, custom Jinja delimiter. |

127 | `recipes/INDEX.md` | Step-by-step recipes for adding/updating each integration type. |

.agents/skills/integrations-lifecycle/artifacts-and-banners.md

+5 -1

@@ -20,6 +20,10 @@ banner conventions and edit rules.

28 ## Banner conventions per file kind

@@ -129,7 +133,7 @@ adjustments:

133 ### `src/health/notifications/<dir>/README.md` (direct case)

134

135 Same banner as above. Written DIRECTLY, not as a symlink. Keep

132 -in mind for the five-file consistency rule -- the README.md is

136 +in mind for the collector consistency rule -- the README.md is

137 the generated artifact.

138

139 ### `src/collectors/COLLECTORS.md`, `SECRETS.md`, `SERVICE-DISCOVERY.md`

.agents/skills/integrations-lifecycle/consistency.md

+45 -15

@@ -1,26 +1,31 @@

1 -# Five-file consistency rule

1 +# Collector consistency rule

3 `<repo>/AGENTS.md` declares ("Collector Consistency

4 Requirements") that any change touching a collector MUST land

5 -in one PR with matching changes to all five of:

5 +in one PR with matching changes to all relevant collector artifacts:

7 1. **The code** -- the collector implementation files.

8 2. **`metadata.yaml`** -- the integration page driver.

9 -3. **`config_schema.json`** -- the dashboard's DYNCFG editor.

10 -4. **The stock `.conf`** -- what `/etc/netdata/<plugin>/...`

9 +3. **`taxonomy.yaml`** -- dashboard table-of-contents placement

10 + for the collector's chart contexts.

11 +4. **`config_schema.json`** -- the dashboard's DYNCFG editor.

12 +5. **The stock `.conf`** -- what `/etc/netdata/<plugin>/...`

13 ships.

12 -5. **`health.d/*.conf`** -- alert definitions for the

14 +6. **`health.d/*.conf`** -- alert definitions for the

15 collector's metrics.

14 -6. **`README.md`** -- comprehensive end-user documentation

16 +7. **`README.md`** -- comprehensive end-user documentation

17 (often a symlink into the generated

18 `integrations/<slug>.md`; see `artifacts-and-banners.md`).

18 -(`AGENTS.md` lists 6 items; the "5-file" shorthand merges

19 -"the code" into the implicit driver.)

20 +The old "5-file" shorthand is stale. Treat the list above as

21 +the durable review checklist; a given PR may legitimately not

22 +touch every file, but it must explain why an affected artifact

23 +does not need a matching edit.

25 The rule covers obvious cases (units change in code -> update

26 metadata.yaml; new config option -> update schema, stock conf,

23 -and docs; new metric -> update metadata.yaml and README.md)

27 +and docs; new metric -> update metadata.yaml, taxonomy.yaml,

28 +and README.md)

29 and subtle ones (renaming a metric label affects the alert

30 definition that refers to it; changing a default affects the

31 stock conf example and the documented default value).

@@ -33,6 +38,19 @@ stock conf example and the documented default value).

38 against its JSON Schema only. It does NOT cross-check

39 against `config_schema.json`, the stock `.conf`, or

40 `health.d/*.conf`.

41 +- **`gen_taxonomy.py`** validates committed collector

42 + `taxonomy.yaml` files, cross-references literal owned contexts

43 + and widget references against `metadata.yaml`, requires declared

44 + dynamic selectors, and emits the gitignored

45 + `integrations/taxonomy.json` artifact.

46 +- **`check_collector_taxonomy.py`** is wired into

47 + `check-markdown.yml` for pull requests. It fails when a PR

48 + touches a collector `taxonomy.yaml`, adds/removes it, or edits

49 + a `metadata.yaml` metrics block without a matching

50 + `taxonomy.yaml`. Non-metrics-only `metadata.yaml` edits such as

51 + setup prose, overview text, categories, and troubleshooting do not

52 + trigger touched-collector taxonomy coverage by themselves, although

53 + the global taxonomy validation still runs.

54 - **`integrations/check_collector_metadata.py`** is broken

55 (see `gotchas.md` and `validators.md` for details). Its

56 imports refer to symbols that no longer exist in

@@ -44,7 +62,7 @@ stock conf example and the documented default value).

62 metric names exist in the collector" check.

63 - **`check-markdown.yml`** only validates that generated

64 markdown links resolve through Learn ingest -- not that

47 - metadata.yaml is in sync with the other four files.

65 + metadata.yaml is in sync with config/schema/stock-conf/README.

67 The one exception is **ibm.d modules**: their `metadata.yaml`,

68 `README.md`, and `config_schema.json` are GENERATED by the

@@ -64,7 +82,16 @@ When reviewing a PR that touches a collector, verify:

82 `metadata.yaml`. If a metric is renamed, both files must

83 change.

67 -2. **Config changes propagate to all four config-related

85 +2. **Chart-context changes have matching `taxonomy.yaml`

86 + changes.** Structural `items:` entries that own contexts and

87 + widget `contexts:` references must name real contexts in the

88 + collector's `metadata.yaml`. Dynamic contexts must use

89 + `type: selector` or selector objects with `context_prefix:` or

90 + `collect_plugin:` and the corresponding

91 + `metrics.dynamic_context_prefixes:` or

92 + `metrics.dynamic_collect_plugins:` declaration.

93 +

94 +3. **Config changes propagate to all four config-related

95 files.**

96 - The Go struct field (in `config.go`).

97 - `config_schema.json` -- the field appears with the

@@ -74,12 +101,12 @@ When reviewing a PR that touches a collector, verify:

101 - `metadata.yaml` -- the option appears under

102 `setup.configuration.options.list`.

103

77 -3. **Alert changes have matching `metadata.yaml.alerts`

104 +4. **Alert changes have matching `metadata.yaml.alerts`

105 entries.** If `health.d/<plugin>.conf` adds, removes, or

106 renames an alert, `metadata.yaml.modules.<m>.alerts[]` must

107 reflect the change.

108

82 -4. **README.md handling.** If the plugin directory has a

109 +5. **README.md handling.** If the plugin directory has a

110 single integration, the README is a symlink to the

111 generated `integrations/<slug>.md` -- the symlink target

112 already updates when `metadata.yaml` updates. If the

@@ -88,13 +115,13 @@ When reviewing a PR that touches a collector, verify:

115 `agent_notification` is a special case: the README itself

116 is the generated artifact (no `integrations/` subdir).

117

91 -5. **`integrations/<slug>.md` regenerated.** The author

118 +6. **`integrations/<slug>.md` regenerated.** The author

119 should have run the pipeline locally and committed the

120 updated `.md` file. `check-markdown.yml` will re-run the

121 pipeline in CI; if the author's commit and CI's regen

122 diverge, the PR fails.

123

97 -6. **Umbrella pages.** If the diff added or removed a

124 +7. **Umbrella pages.** If the diff added or removed a

125 collector, `src/collectors/COLLECTORS.md` should reflect

126 it. Same for `SECRETS.md` (secretstore changes) and

127 `SERVICE-DISCOVERY.md` (service-discovery changes -- but

@@ -138,3 +165,6 @@ For now, the consistency rule is a review-time policy.

165 - "I changed a default in the stock `.conf` only." -> Update

166 `config_schema.json` `default`, `metadata.yaml.setup.configuration.options.list[].default_value`,

167 and the README in lockstep.

168 +- "I added a chart context but skipped `taxonomy.yaml` because

169 + the dashboard will discover it." -> No. Add the context to a

170 + placement or use a declared dynamic selector.

.agents/skills/integrations-lifecycle/gotchas.md

+57

@@ -5,6 +5,63 @@ anchor, custom Jinja delimiter, undocumented behavior, and

5 edge case the integrations pipeline carries today. Read this

6 before assuming the code does the obvious thing.

8 +## Taxonomy authoring gotchas

9 +

10 +### `grid.items` and `view_switch` branches do not accept string shorthand

11 +

12 +- Wrong shape: `type: grid` with `items: ["mysql.queries"]`.

13 +- Failure: schema validation rejects the string because grid children

14 + are display-only objects.

15 +- Correct shape: use `type: context` with `contexts:` and

16 + `chart_library:` inside the grid; own the context elsewhere in a

17 + structural item.

18 +

19 +### Dynamic selectors need explicit metadata opt-in

20 +

21 +- Wrong shape: `context_prefix: [snmp.device_prof_]` without

22 + `metrics.dynamic_context_prefixes:` in the sibling `metadata.yaml`.

23 +- Failure: TAX031 fatal.

24 +- Correct shape: declare the safe namespace in metadata, for example

25 + `dynamic_context_prefixes: [{prefix: snmp., reason: ...}]`.

26 +

27 +### Taxonomy fields use snake_case, not legacy FE camelCase

28 +

29 +- Wrong shape: `chartLibrary`, `groupByLabel`, `tableSortBy`.

30 +- Failure: closed-schema `additionalProperties` rejection.

31 +- Correct shape: `chart_library`, `group_by_label`, `table_sort_by`.

32 +

33 +### Structural containers need stable `id:` values

34 +

35 +- Wrong shape: `type: group` with only `title:` and `items:`.

36 +- Failure: schema validation rejects the missing `id`.

37 +- Correct shape: choose a stable kebab-case `id` that does not change

38 + when the display `title` is renamed.

39 +

40 +### `single_node:` is a sparse override, `view_switch` is replacement

41 +

42 +- Wrong shape: putting `multi_node:` next to ordinary placement or item

43 + fields.

44 +- Failure: TAX022 fatal unless `multi_node` is inside

45 + `type: view_switch`.

46 +- Correct shape: use `single_node:` only for small same-kind field

47 + deltas; use `type: view_switch` when the whole body differs.

48 +

49 +### `unresolved:` is only for staged literal widget references

50 +

51 +- Wrong shape: a bare unknown literal context in widget `contexts:`.

52 +- Failure: TAX003 fatal.

53 +- Correct shape: either own/reference a real metadata context, use a

54 + selector object, or use `{context, unresolved: {reason, owner,

55 + expires}}` when the missing context is an intentional staged rollout.

56 + `expires` must be `YYYY-MM-DD`.

57 +

58 +### Removed structural-only shapes stay rejected

59 +

60 +- Wrong shape: top-level or placement `contexts:` / `subsections:`, or

61 + old list-merge fields ending in `_extend`.

62 +- Failure: TAX001/TAX023 fatal.

63 +- Correct shape: use recursive `items:` with the v1 item kinds.

64 +

65 ## Dead / broken code in the pipeline

67 ### `integrations/check_collector_metadata.py` is broken

.agents/skills/integrations-lifecycle/how-tos/INDEX.md

@@ -27,6 +27,7 @@ violation.

27 | Adding a new top-level `integration_type` (peer of collector, logs, exporter, etc.) | [adding-new-integration-type](adding-new-integration-type.md) | 8-step recipe: schema, pipeline, templates, categories.yaml, map.yaml, source metadata, downstream repos. Covers what to clone from existing types, the hardcoded-vs-derived `learn_rel_path` distinction, and the `integration_placeholder` mechanism. |

28 | Auditing `metadata.yaml` links to Learn | [auditing-metadata-learn-links](auditing-metadata-learn-links.md) | Commands and repair rules for finding absolute Learn URL drift, validating fragments, and checking source-relative metadata links. |

29 | Keeping Network Flows on the Learn "Monitor anything" page | [monitor-anything-network-flows](monitor-anything-network-flows.md) | Explains that `src/collectors/COLLECTORS.md` is generated by `gen_doc_collector_page.py`, why top-level `flows` must be treated as a section, and how to validate the generated `Network Flows` section. |

30 +| Understanding collector taxonomy generation | [taxonomy-yaml-to-taxonomy-json](taxonomy-yaml-to-taxonomy-json.md) | General flow from collector `metadata.yaml` + `taxonomy.yaml` through validation to generated `integrations/taxonomy.json`, with references to the executable source of truth. |

32 Description authoring rules live in `../description-authoring.md`

33 because they apply to all metadata authors, not only to one

.agents/skills/integrations-lifecycle/how-tos/taxonomy-yaml-to-taxonomy-json.md new

+207

@@ -0,0 +1,207 @@

1 +# How collector taxonomy becomes `integrations/taxonomy.json`

2 +

3 +Question answered: what is the general flow from collector

4 +`metadata.yaml` and `taxonomy.yaml` to the generated dashboard

5 +taxonomy artifact consumed by downstream frontend code?

6 +

7 +## Short version

8 +

9 +`metadata.yaml` is the metric-context source of truth. Collector

10 +`taxonomy.yaml` files organize those contexts into the dashboard table

11 +of contents. `integrations/gen_taxonomy.py` validates both sides

12 +against the taxonomy registries and schemas, then emits the gitignored

13 +`integrations/taxonomy.json` cross-repo contract.

14 +

15 +The implementation details can evolve, but the durable model is:

16 +

17 +1. metadata declares what metric contexts exist;

18 +2. taxonomy declares where those contexts belong and which widgets

19 + reference them;

20 +3. the generator proves the references are valid;

21 +4. the generated JSON carries the normalized section tree, placements,

22 + recursive items, and context snapshots.

23 +

24 +## Inputs

25 +

26 +The taxonomy pipeline reads four source classes:

27 +

28 +- Collector `metadata.yaml` files. The generator loads collector

29 + modules through the shared integrations loader and extracts metric

30 + contexts from `metrics.scopes[].metrics[].name`; see

31 + `integrations/gen_taxonomy.py:269-276`.

32 +- Collector `taxonomy.yaml` files. These live next to collector

33 + metadata and use the closed v1 authoring schema

34 + `integrations/schemas/taxonomy_collector.json`.

35 +- `integrations/taxonomy/sections.yaml`. This registry owns stable

36 + `section_id` targets and parentage for the generated TOC section

37 + tree; schema: `integrations/schemas/taxonomy_sections.json`.

38 +- `integrations/taxonomy/icons.yaml`. This registry limits the icon IDs

39 + sections and placements may reference.

40 +

41 +The field-level contract is documented in

42 +`../schema-reference.md`. The contributor workflow is documented in

43 +`../recipes/add-go-collector.md` and

44 +`../recipes/update-collector.md`.

45 +

46 +## Metadata indexing

47 +

48 +The generator first builds metadata indexes from all known collector

49 +metadata:

50 +

51 +- `by_path_module`: matches a `taxonomy.yaml` file to its sibling

52 + `metadata.yaml` module by path, `plugin_name`, and `module_name`.

53 +- `all_contexts`: sorted global list of known metric contexts, used for

54 + prefix resolution.

55 +- `contexts_by_plugin`: contexts grouped by plugin name, used for

56 + `collect_plugin` selectors.

57 +- dynamic selector guardrails from

58 + `metrics.dynamic_context_prefixes` and

59 + `metrics.dynamic_collect_plugins`.

60 +

61 +The relevant implementation is `integrations/gen_taxonomy.py:286-315`.

62 +

63 +This is why `metadata.yaml` is the metric source of truth: a literal

64 +context in taxonomy authoring is valid only if the sibling metadata

65 +module declares it. A taxonomy file can organize and reference metric

66 +contexts; it cannot invent static metric contexts.

67 +

68 +## Taxonomy authoring validation

69 +

70 +Each collector `taxonomy.yaml` is loaded and validated against the

71 +closed authoring schema before semantic validation. The schema rejects

72 +old or ambiguous shapes such as placement-level `contexts:`,

73 +`section_path:`, and string shorthand in display-only positions.

74 +

75 +After schema validation, the generator checks:

76 +

77 +- `section_id` exists in `sections.yaml`;

78 +- icon IDs exist in `icons.yaml`;

79 +- literal owned contexts exist in the sibling metadata;

80 +- literal widget references exist in metadata unless they carry the

81 + explicit `unresolved` escape hatch;

82 +- dynamic selectors are declared by metadata guardrails;

83 +- display widgets reference contexts but do not own them;

84 +- every literal widget reference is owned somewhere else unless it is

85 + deliberately unresolved.

86 +

87 +The matching and semantic validation start in

88 +`integrations/gen_taxonomy.py:745-790`. Selector and literal-reference

89 +validation live around `integrations/gen_taxonomy.py:438-526`.

90 +

91 +## Ownership model

92 +

93 +The generated artifact separates ownership from display references:

94 +

95 +- Structural strings and `type: owned_context` own literal contexts.

96 +- Structural `type: selector` owns the contexts matched by

97 + `context_prefix` or `collect_plugin`.

98 +- Containers such as `group`, `flatten`, `grid`, `first_available`, and

99 + `view_switch` aggregate context snapshots from their children.

100 +- `type: context` display widgets reference contexts through

101 + `contexts:` but do not own them.

102 +

103 +Generated items and placements therefore carry:

104 +

105 +- `resolved_contexts`: contexts owned by that node after child and

106 + selector aggregation.

107 +- `referenced_contexts`: contexts referenced by display widgets.

108 +- `unresolved_references`: staged widget references that intentionally

109 + do not resolve yet, with `reason`, `owner`, `expires`, and

110 + `item_path`.

111 +

112 +The recursive emission logic is in `integrations/gen_taxonomy.py:551-719`.

113 +The FE-facing meaning of the generated fields is documented in

114 +`../in-app-contract.md`.

115 +

116 +## Output artifact

117 +

118 +The generated artifact is `integrations/taxonomy.json`. It is validated

119 +against `integrations/schemas/taxonomy_output.json` and is intentionally

120 +gitignored.

121 +

122 +Top-level shape:

123 +

124 +```json

125 +{

126 + "taxonomy_schema_version": 1,

127 + "source": {},

128 + "sections": [],

129 + "placements": [],

130 + "opted_out_collectors": []

131 +}

132 +```

133 +

134 +Important output concepts:

135 +

136 +- `sections[]` is the resolved global section registry.

137 +- `placements[]` is the ordered list of collector-owned TOC placements.

138 +- `placements[].items[]` is the normalized recursive item tree.

139 +- `collector_ids` links a placement back to the integration IDs produced

140 + from metadata.

141 +- `section_id` is the stable registry handle; `section_path` is the

142 + resolved path for consumers.

143 +

144 +Assembly, deterministic placement sorting, and output schema validation

145 +are handled in `integrations/gen_taxonomy.py:847-883`. Writing is handled

146 +by the generator CLI in `integrations/gen_taxonomy.py:890-920`.

147 +

148 +## CI flow

149 +

150 +Pull requests run the taxonomy checker from

151 +`.github/workflows/check-markdown.yml`. The checker:

152 +

153 +- validates all committed taxonomy sources by building the artifact;

154 +- enforces taxonomy coverage when a PR changes a collector

155 + `taxonomy.yaml`, adds/removes it, or edits metric-bearing parts of

156 + `metadata.yaml`;

157 +- runs the taxonomy unit tests.

158 +

159 +See `.github/workflows/check-markdown.yml:45-58` and

160 +`integrations/check_collector_taxonomy.py`.

161 +

162 +The master regeneration workflow runs `integrations/gen_taxonomy.py` as

163 +part of the integrations regeneration job; see

164 +`.github/workflows/generate-integrations.yml:59-68`. The generated

165 +`taxonomy.json` is still a runtime/downstream contract artifact, not a

166 +committed source file.

167 +

168 +## Worked mental model

169 +

170 +For a static collector such as MySQL:

171 +

172 +1. `metadata.yaml` declares `mysql.queries`.

173 +2. `mysql/taxonomy.yaml` owns `mysql.queries` in a structural item.

174 +3. A summary grid widget may also reference `mysql.queries`.

175 +4. The generated placement includes `mysql.queries` in

176 + `resolved_contexts` because it is owned, and in

177 + `referenced_contexts` where the widget uses it.

178 +

179 +For a dynamic collector such as SNMP:

180 +

181 +1. `metadata.yaml` declares a dynamic namespace such as `snmp.`.

182 +2. `snmp/taxonomy.yaml` may use a narrower selector like

183 + `snmp.device_prof_` under that declared namespace.

184 +3. Selector items own the matched context snapshot; selector references

185 + inside widgets reference dynamic contexts without claiming ownership.

186 +4. The generated JSON preserves selector objects so downstream frontend

187 + code can resolve runtime dynamic contexts cleanly.

188 +

189 +## How I figured this out

190 +

191 +Files read:

192 +

193 +- `integrations/gen_taxonomy.py`

194 +- `integrations/check_collector_taxonomy.py`

195 +- `integrations/schemas/taxonomy_collector.json`

196 +- `integrations/schemas/taxonomy_output.json`

197 +- `.github/workflows/check-markdown.yml`

198 +- `.github/workflows/generate-integrations.yml`

199 +- `../schema-reference.md`

200 +- `../in-app-contract.md`

201 +

202 +Commands used during the original analysis:

203 +

204 +```bash

205 +rg -n "def module_contexts|def build_metadata_indexes|def process_taxonomy_file|def emit_item|def build_taxonomy" integrations/gen_taxonomy.py

206 +rg -n "gen_taxonomy|check_collector_taxonomy|taxonomy.json|taxonomy.yaml" .github/workflows integrations/README.md .agents/sow/specs/taxonomy.md

207 +```

.agents/skills/integrations-lifecycle/ibm-d.md

+4 -3

@@ -156,10 +156,11 @@ consistency between:

156 - the user-facing documentation (`README.md`).

157

158 It is the closest thing this repo has to enforcement of the

159 -five-file consistency rule for the integration-page side

159 +collector consistency rule for the integration-page side

160 (metadata + README + config_schema), but it does NOT cover

161 -the stock `.conf` or `health.d/<...>.conf` -- those still

162 -need manual sync.

161 +taxonomy.yaml, the stock `.conf`, or `health.d/<...>.conf` --

162 +those still need manual sync unless a module-specific generator

163 +adds coverage.

164

165 ## Risks and gotchas

166

.agents/skills/integrations-lifecycle/in-app-contract.md

+66 -5

@@ -2,7 +2,9 @@

3 The Netdata cloud-frontend dashboard (the React app that powers

4 `app.netdata.cloud`) renders the Integrations page from the

5 -`integrations.js` artifact this repo produces. This guide

5 +`integrations.js` artifact this repo produces. Collector dashboard

6 +taxonomy is published separately as `integrations/taxonomy.json`.

7 +This guide

8 documents the contract between the two repositories so

9 maintainers know what is and is NOT in scope when working on

10 integrations-lifecycle changes.

@@ -17,14 +19,19 @@ SCOPE for this skill; only the artifact contract matters.

19 **This repo produces:** `integrations/integrations.js` (and

20 `integrations/integrations.json`) on every CI run of

21 `generate-integrations.yml` (or local run of

20 -`gen_integrations.py`). Both files are gitignored in this

21 -repo.

22 +`gen_integrations.py`). It also produces

23 +`integrations/taxonomy.json` from `gen_taxonomy.py`. All three

24 +files are gitignored in this repo.

26 **The cloud-frontend repo consumes:**

27 `integrations/integrations.js` -- specifically, it copies the

28 file into `src/domains/integrations/data/integrations.js` in

29 its own source tree.

31 +`integrations/taxonomy.json` is a new opt-in downstream contract.

32 +This repo validates and emits it, but cloud-frontend consumption is

33 +owned by the dashboard team and may land independently.

34 +

35 ## How the consumption works

37 Confirmed at

@@ -48,6 +55,11 @@ A third script, `scripts/checkLinks.js`, validates that links

55 in `src/domains/integrations/data/integrations.js` and

56 `src/domains/integrations/utils/integrations.js` resolve.

58 +At the time this skill was updated, cloud-frontend had not yet

59 +consumed `taxonomy.json`; its CI needs an explicit follow-up change

60 +to run `python3 integrations/gen_taxonomy.py` and copy the JSON

61 +artifact if/when the dashboard switches chart TOC ownership.

62 +

63 ## The artifact shape

65 `integrations/integrations.js`:

@@ -72,6 +84,45 @@ export const integrations = [

84 ];

85 ```

87 +`integrations/taxonomy.json`:

88 +

89 +```json

90 +{

91 + "taxonomy_schema_version": 1,

92 + "source": {

93 + "netdata_commit": "...",

94 + "generated_at": "..."

95 + },

96 + "sections": [],

97 + "placements": [],

98 + "opted_out_collectors": []

99 +}

100 +```

101 +

102 +Each taxonomy placement keeps the ordered recursive `items:` tree and

103 +snapshot fields generated from current metadata for CI/review

104 +diffing. `resolved_contexts` contains owned contexts;

105 +`referenced_contexts` contains display/widget references, and

106 +`unresolved_references` carries explicit unresolved-reference escape

107 +hatches for downstream consumers. The schema lives at

108 +`integrations/schemas/taxonomy_output.json`.

109 +

110 +FE adapters must discriminate these v1 taxonomy node kinds:

111 +

112 +- `owned_context` -- structural leaf that owns one literal context.

113 +- `group` -- structural container.

114 +- `flatten` -- structural container whose children flatten into the

115 + parent menu level.

116 +- `selector` -- structural dynamic owner from `context_prefix` or

117 + `collect_plugin`.

118 +- `context` -- display widget that references contexts.

119 +- `grid` -- display container with positioned child widgets.

120 +- `first_available` -- ordered display alternatives.

121 +- `view_switch` -- whole-body replacement for multi-node vs

122 + single-node rendering.

123 +- string shorthand appears only in authoring; generated output

124 + normalizes it to `owned_context`.

125 +

126 All public content sections consumed by downstream renderers must be

127 markdown strings in the generated artifacts, even when the source

128 `metadata.yaml` stores them as structured YAML objects or arrays.

@@ -118,6 +169,10 @@ In practice this means:

169 removed top-level field) WILL break the dashboard on the

170 next sync. There is no shape-versioning today; both repos

171 assume the JS export shape is stable.

172 +- A breaking change to `taxonomy.json` must bump

173 + `taxonomy_schema_version` and coordinate with downstream

174 + consumers. Additive fields are acceptable only when old

175 + consumers can ignore them safely.

176

177 ## What is OUT of scope for integrations-lifecycle

178

@@ -138,13 +193,19 @@ In practice this means:

193 keys) is a contract. Avoid breaking changes; coordinate

194 with the cloud-frontend team if a key must be renamed or

195 removed.

141 -2. **Render structured metadata before publication**. A new

196 +2. **Treat `taxonomy.json` as a versioned published artifact

197 + once consumed.** Keep v1 authoring closed: `section_id:`,

198 + ordered `items:`, explicit item `type` values, selector keys

199 + (`context_prefix:`, `context_prefix_exclude:`,

200 + `collect_plugin:`), widget `contexts:`, and sparse

201 + `single_node:` overrides.

202 +3. **Render structured metadata before publication**. A new

203 integration type that reuses collector-style sections must

204 include every structured content key in its render-key list.

205 Do not publish raw `metrics` objects, `alerts` arrays, or

206 similar YAML structures under the public markdown section

207 names.

147 -3. **Custom Jinja markers in metadata** (`{% details %}`,

208 +4. **Custom Jinja markers in metadata** (`{% details %}`,

209 `{% relatedResource %}`, `{% if %}`) are part of the

210 contract. The dashboard's renderer interprets them. Test

211 any new marker against both surfaces before relying on it.

.agents/skills/integrations-lifecycle/per-type-matrix.md

+10 -1

@@ -20,6 +20,15 @@ consume it.

20 | `distros` | `.github/data/distros.yml` | `distros.json` (declared but **NOT enforced** -- see `gotchas.md`) | n/a -- consumed only by `render_deploy` | n/a | n/a | n/a | feeds `deploy.platform_info` table |

21 | `shared` | n/a -- referenced by other schemas via `./shared.json#/$defs/...` | n/a | n/a | n/a | n/a | n/a | building block (instance, full_setup, troubleshooting, _folding) |

23 +## Collector taxonomy artifact matrix

24 +

26 +|---|---|---|---|---|---|

31 +

32 ## Slug rules summary

34 The slug used in the output filename comes from `clean_string`

@@ -110,6 +119,6 @@ file DIRECTLY to `<dir>/README.md` (`:488-496`). No

119 `integrations/` subdirectory, no symlink. So

120 `src/health/notifications/email/README.md` IS the generated

121 artifact, not a symlink. Keep this in mind when checking the

113 -five-file consistency rule (the README.md you would normally

122 +collector consistency rule (the README.md you would normally

123 not edit is the same physical file as the generated

124 integration page).

.agents/skills/integrations-lifecycle/pipeline.md

+82 -4

@@ -5,7 +5,7 @@ script, every input, every output, every CI workflow. All path

5 citations are repo-relative; line citations refer to the file at

6 HEAD of `master` at the time this skill was last updated.

8 -## The four-stage pipeline

8 +## The integration documentation pipeline

10 ```

11 [ YAML sources ]

@@ -19,6 +19,18 @@ HEAD of `master` at the time this skill was last updated.

19 +------------------------+

20 | integrations.js (gitignored)

21 | integrations.json (gitignored)

22 ++------------------------+

23 +

24 +[ collector metadata.yaml + taxonomy.yaml + taxonomy registries ]

25 + |

26 + v

27 ++------------------------+

28 +| gen_taxonomy.py | (taxonomy validator, resolver)

29 ++------------------------+

30 + |

31 + v

32 ++------------------------+

33 +| taxonomy.json (gitignored)

34 +------------------------+

35 |

36 v

@@ -185,6 +197,69 @@ fresh on every run; in CI, the workflow `rm`s them after the

197 downstream scripts read them so they are NOT included in the

198 auto-PR.

199

200 +## Parallel taxonomy stage -- `gen_taxonomy.py`

201 +

202 +Repo path: `integrations/gen_taxonomy.py`.

203 +

204 +### Inputs

205 +

206 +- Collector `metadata.yaml` files from the same collector source

207 + roots used by `gen_integrations.py`.

208 +- Sibling collector `taxonomy.yaml` files discovered as

209 + `<collector>/taxonomy.yaml`.

210 +- `integrations/taxonomy/sections.yaml` -- the stable section

211 + registry. Collector files reference only `section_id:`, never

212 + `section_path:`.

213 +- `integrations/taxonomy/icons.yaml` -- allowed icon ids.

214 +- `integrations/schemas/taxonomy_collector.json`,

215 + `taxonomy_sections.json`, and `taxonomy_output.json`.

216 +

217 +### Validation behavior

218 +

219 +The generator validates closed v1 authoring schemas, checks that

220 +literal owned contexts and widget references resolve to real contexts

221 +in the owning collector's `metadata.yaml`, and requires dynamic

222 +selectors to be declared by the owning collector:

223 +

224 +- `context_prefix:` requires

225 + `metrics.dynamic_context_prefixes: [{prefix, reason}]`; taxonomy may

226 + use a narrower prefix under the declared namespace.

227 +- `collect_plugin:` requires

228 + `metrics.dynamic_collect_plugins: [{plugin, reason}]`.

229 +

230 +Findings render as plain text locally and as GitHub Actions

231 +annotations in CI. Fatal findings fail the run.

232 +

233 +### Outputs

234 +

235 +`integrations/taxonomy.json` is written by default and validated

236 +against `integrations/schemas/taxonomy_output.json`. The file is

237 +gitignored and removed by `generate-integrations.yml` cleanup, just

238 +like `integrations/integrations.js` and `integrations/integrations.json`.

239 +

240 +Run validation only:

241 +

242 +```bash

243 +python3 integrations/gen_taxonomy.py --check-only

244 +```

245 +

246 +Seed a collector taxonomy from existing metadata contexts:

247 +

248 +```bash

249 +python3 integrations/gen_taxonomy_seed.py src/go/plugin/go.d/collector/apache/metadata.yaml --module-name apache --section-id applications.apache --placement-id apache --icon apache

250 +```

251 +

252 +The seed helper emits a flat `items:` tree. For collectors with richer

253 +dashboard layout needs, convert that flat list into explicit

254 +`owned_context`, `group`, `flatten`, `selector`, `context`, `grid`,

255 +`first_available`, or `view_switch` items before opening the PR.

256 +

257 +Pull-request coverage is checked by:

258 +

259 +```bash

260 +python3 integrations/check_collector_taxonomy.py --pr-diff origin/master...HEAD

261 +```

262 +

263 ### Commands a maintainer runs locally

264

265 ```bash

@@ -412,12 +487,14 @@ scripts directly during active development.

487 ## End-to-end: a single PR's flow

488

489 1. Developer edits `src/go/plugin/go.d/collector/foo/metadata.yaml`

415 - (and the four other consistency-rule files: `config_schema.json`,

416 - stock conf, `health.d/foo.conf`, `README.md`).

490 + (and any other affected consistency-rule files:

491 + `taxonomy.yaml`, `config_schema.json`, stock conf,

492 + `health.d/foo.conf`, `README.md`).

493 2. Developer runs locally:

494 ```bash

495 ./integrations/pip.sh

496 python3 integrations/gen_integrations.py

497 + python3 integrations/gen_taxonomy.py --check-only

498 python3 integrations/gen_docs_integrations.py -c go.d/foo

499 python3 integrations/gen_doc_collector_page.py

500 python3 integrations/gen_doc_secrets_page.py

@@ -429,7 +506,8 @@ scripts directly during active development.

506 4. PR is opened. `check-markdown.yml` runs, regenerates the

507 same files in CI, and validates Learn ingest. If the dev's

508 committed files differ from CI's regen, the PR fails.

432 -5. Reviewer checks the five-file consistency.

509 +5. Reviewer checks collector consistency, including taxonomy

510 + coverage for changed chart contexts.

511 6. PR merges. `generate-integrations.yml` triggers on master,

512 regenerates everything, and opens an `integrations-regen`

513 PR if anything is now stale (typically nothing, because the

.agents/skills/integrations-lifecycle/recipes/INDEX.md

+1 -1

@@ -39,7 +39,7 @@ python3 integrations/gen_docs_integrations.py -c go.d/<your-module>

39 1. Read `pipeline.md` for the end-to-end flow.

40 2. Read `schema-reference.md` for the exact field your

41 `metadata.yaml` change needs.

42 -3. Read `consistency.md` for the five-file rule.

42 +3. Read `consistency.md` for the collector consistency rule.

43 4. Read `gotchas.md` BEFORE assuming the pipeline does the

44 obvious thing.

45 5. If you encountered a question that this catalog doesn't

.agents/skills/integrations-lifecycle/recipes/add-go-collector.md

+36 -11

@@ -151,10 +151,28 @@ validator will warn (fatal). Either pick an existing category

151 or add a new one under the appropriate parent (typically

152 `data-collection`).

153

154 -## 4. Stock `.conf`, `config_schema.json`, alerts, README

155 -

156 -These three are the rest of the five-file consistency rule:

157 -

154 +## 4. Taxonomy, stock `.conf`, `config_schema.json`, alerts, README

155 +

156 +These files are the rest of the collector consistency rule:

157 +

158 +- `src/go/plugin/go.d/collector/<name>/taxonomy.yaml` --

159 + dashboard TOC placement for chart contexts. Static collectors

160 + use ordered `items:` trees; plain strings in structural `items:`

161 + own chart contexts. Dynamic collectors use `type: selector` with

162 + `context_prefix:` or `collect_plugin:` and matching

163 + `metadata.yaml.metrics.dynamic_*` declarations. Display widgets

164 + use `type: context` with `contexts:` and `chart_library`; those

165 + referenced contexts must also be owned by structural items.

166 + Pick `--section-id` from

167 + `integrations/taxonomy/sections.yaml`; `section_id` is a stable

168 + registry ID, not a path to invent in the collector file.

169 + Seed the initial explicit context list with:

170 + ```bash

171 + python3 integrations/gen_taxonomy_seed.py src/go/plugin/go.d/collector/<name>/metadata.yaml --module-name <name> --section-id <section.id> --placement-id <name> --icon <icon>

172 + ```

173 + For a rich example with summary grids, table widgets, nested

174 + groups, and ownership leaves, read

175 + `src/go/plugin/go.d/collector/mysql/taxonomy.yaml`.

176 - `src/go/plugin/go.d/config/go.d/<name>.conf` -- the stock

177 config users will see at

178 `/etc/netdata/go.d/<name>.conf`. Keep it minimal but

@@ -178,6 +196,7 @@ From the repo root:

196 ```bash

197 ./integrations/pip.sh # once

198 python3 integrations/gen_integrations.py

199 +python3 integrations/gen_taxonomy.py --check-only

200 python3 integrations/gen_docs_integrations.py -c go.d/<name>

201 python3 integrations/gen_doc_collector_page.py

202 python3 integrations/gen_doc_secrets_page.py

@@ -187,6 +206,9 @@ Expected outputs:

206

207 - `integrations/integrations.js` and `integrations/integrations.json`

208 regenerated (gitignored, do NOT commit them).

209 +- Collector taxonomy validated. If `gen_taxonomy.py` fails, fix

210 + `taxonomy.yaml` or the matching `metadata.yaml.metrics.dynamic_*`

211 + declaration before continuing.

212 - `src/go/plugin/go.d/collector/<name>/integrations/<slug>.md`

213 CREATED. Inspect: it should contain the `<!--startmeta`

214 banner with your `sidebar_label` and `learn_rel_path`, then

@@ -208,6 +230,9 @@ re-run.

230 every section reads correctly.

231 - Open `src/collectors/COLLECTORS.md` and find your collector

232 in the table.

233 +- Run `python3 integrations/check_collector_taxonomy.py` before

234 + opening the PR. In CI this also runs with `--pr-diff` to enforce

235 + touched-collector taxonomy coverage.

236 - Run `git diff` and confirm the only changes are in:

237 - `src/go/plugin/go.d/collector/<name>/...` (your new module

238 files).

@@ -221,18 +246,19 @@ re-run.

246 category.

247 - NOT `integrations/integrations.js` or

248 `integrations.json` (gitignored).

249 + - NOT `integrations/taxonomy.json` (gitignored).

250

251 ## 7. Commit and push

252

253 Single PR, single commit (or a few logical commits) covering

228 -the five-file rule plus the generated integration page and

229 -umbrella update. Reviewers will check that all five files

230 -were updated together.

254 +the collector consistency rule plus the generated integration

255 +page and umbrella update. Reviewers will check that affected

256 +artifacts were updated together.

257

258 ## 8. CI

259

260 - `check-markdown.yml` will run on the PR. It runs the same

235 - pipeline scripts and validates Learn ingest. If your

261 + pipeline scripts, validates taxonomy, and validates Learn ingest. If your

262 committed integration page diverges from CI's regen, the

263 workflow fails -- fix locally and re-push.

264 - After merge, `generate-integrations.yml` triggers on master.

@@ -256,10 +282,9 @@ were updated together.

282

283 ## Common mistakes

284

259 -- **Forgetting one of the five files.** The most common

285 +- **Forgetting one collector-consistency artifact.** The most common

286 cause of review feedback. Use `git status` after step 5 to

261 - confirm all five (or six counting the umbrella) are

262 - staged.

287 + confirm every affected source/generated artifact is staged.

288 - **Hand-editing `integrations/<slug>.md` after generation.**

289 Never. It is regenerated each time. Edit `metadata.yaml`

290 and re-run.

.agents/skills/integrations-lifecycle/recipes/update-collector.md new

+106

@@ -0,0 +1,106 @@

1 +# Recipe: update an existing collector integration

2 +

3 +Use this when a collector's metrics, chart contexts, configuration,

4 +alerts, or generated docs change. The goal is to keep runtime behavior,

5 +metadata, taxonomy, docs, and CI validation in one coherent PR.

6 +

7 +## 0. Read first

8 +

9 +- `../SKILL.md` -- integrations lifecycle overview.

10 +- `../consistency.md` -- what the collector consistency rule requires

11 + and what CI enforces.

12 +- `../schema-reference.md` -- exact `metadata.yaml` and

13 + `taxonomy.yaml` fields.

14 +

15 +## 1. Identify what changed

16 +

17 +From the collector directory, list the changed surfaces:

18 +

19 +- runtime `.go` / script code;

20 +- `metadata.yaml` metric contexts, units, dimensions, setup, alerts;

21 +- `taxonomy.yaml` dashboard TOC placement;

22 +- `config_schema.json`;

23 +- stock `.conf`;

24 +- `health.d/*.conf`;

25 +- generated `integrations/<slug>.md` and `README.md` symlink.

26 +

27 +If chart contexts are added, removed, renamed, or moved between dynamic

28 +and static emission, update `taxonomy.yaml` in the same PR.

29 +

30 +## 2. Update `metadata.yaml`

31 +

32 +Keep `metrics.scopes[].metrics[].name` aligned with the collector's

33 +actual emitted chart contexts. Keep units and descriptions aligned with

34 +the code. If a collector emits runtime-only dynamic contexts, declare

35 +the guardrail in metadata:

36 +

37 +```yaml

38 +metrics:

39 + dynamic_context_prefixes:

40 + - prefix: snmp.

41 + reason: SNMP profiles emit device-specific contexts at runtime.

42 +```

43 +

44 +Use `dynamic_collect_plugins` only when a stable context-name prefix is

45 +not available.

46 +

47 +## 3. Update `taxonomy.yaml`

48 +

49 +Check whether the existing taxonomy still owns every static context

50 +exactly once:

51 +

52 +```bash

53 +python3 integrations/gen_taxonomy.py --check-only

54 +```

55 +

56 +Rules of thumb:

57 +

58 +- plain strings in structural `items:` own contexts;

59 +- `type: context` widgets reference contexts but do not own them;

60 +- every literal widget reference must be owned elsewhere or carry an

61 + explicit `unresolved` escape hatch;

62 +- dynamic collectors use `type: selector` with declared

63 + `context_prefix:` or `collect_plugin:`;

64 +- pick section IDs from `integrations/taxonomy/sections.yaml`.

65 +

66 +For a rich reference, compare against

67 +`src/go/plugin/go.d/collector/mysql/taxonomy.yaml`.

68 +

69 +## 4. Update the remaining collector artifacts

70 +

71 +Keep these synchronized when the corresponding behavior changes:

72 +

73 +- `config_schema.json` for dynamic configuration;

74 +- stock `.conf` for user-visible defaults;

75 +- `health.d/*.conf` and `metadata.yaml.modules[].alerts[]`;

76 +- generated docs via the integrations pipeline.

77 +

78 +Do not hand-edit generated `integrations/<slug>.md` files.

79 +

80 +## 5. Run local validation

81 +

82 +From the repo root:

83 +

84 +```bash

85 +python3 integrations/gen_integrations.py

86 +python3 integrations/gen_taxonomy.py --check-only

87 +python3 integrations/check_collector_taxonomy.py

88 +python3 -m unittest integrations.tests.test_taxonomy

89 +python3 integrations/gen_docs_integrations.py -c go.d.plugin/<module>

90 +```

91 +

92 +Use the repo-local `.venv/bin/python` when one exists for the current

93 +worktree.

94 +

95 +## 6. Before opening the PR

96 +

97 +Run:

98 +

99 +```bash

100 +git status --short

101 +```

102 +

103 +Commit source changes, generated docs, and taxonomy updates together.

104 +Do not commit gitignored runtime artifacts such as

105 +`integrations/integrations.js`, `integrations/integrations.json`, or

106 +`integrations/taxonomy.json`.

.agents/skills/integrations-lifecycle/schema-reference.md

+105 -1

@@ -1,6 +1,6 @@

1 # Schema reference

3 -Exhaustive per-field reference for all 12 JSON Schemas under

3 +Per-field reference for JSON Schemas under

4 `integrations/schemas/`. Each schema is JSON Schema Draft 7;

5 cross-refs use `./shared.json#/$defs/...` resolved by

6 `Registry(retrieve=retrieve_from_filesystem)`

@@ -162,6 +162,10 @@ where each module is one collector integration.

162 | `modules[].metrics.folding` | $ref `_folding` | yes | -- | learn (clean strips) | Folding for the entire metrics section. |

164 | `modules[].metrics.availability` | array<string> | yes | -- | metrics table column-set | Defines which "availability" columns the table will have. |

169 | `modules[].metrics.scopes[].name` | string | yes | -- | learn metrics table | Special: `global` is rewritten to `<instance> instance` at `gen_integrations.py:914-916`. |

171 | `modules[].metrics.scopes[].labels[].name` | string | yes | -- | learn | Label name. |

@@ -206,6 +210,106 @@ Required on `meta`: `plugin_name`, `module_name`,

210 `monitored_instance`, `keywords`, `related_resources`,

211 `info_provided_to_referring_integrations` (`collector.json:94-101`).

212

213 +## taxonomy_collector.json

214 +

215 +Sibling authoring file for collector dashboard placement:

216 +`<collector>/taxonomy.yaml`. The schema is intentionally closed

217 +(`additionalProperties: false` plus `x_*` extension keys on core

218 +nodes). `section_id:` is the only accepted section reference in v1;

219 +`section_path:` is rejected.

220 +

222 +|---|---|---|---|---|---|

223 +| `taxonomy_version` | integer | yes | `1` | taxonomy | Authoring schema version. |

224 +| `plugin_name` | string | yes | -- | taxonomy | Must match owning `metadata.yaml`. |

225 +| `module_name` | string | yes | -- | taxonomy | Must match owning `metadata.yaml` module. |

231 +| `placements[].title` | string | yes | -- | taxonomy | Multi-node canonical title. |

233 +| `placements[].families` | boolean / array<string> | no | -- | taxonomy | Preserved for the dashboard TOC consumer. |

250 +| `context.table_sort_by[]` | array<object> | no | `{id, desc}` | taxonomy | Table sort directives. |

253 +| `context.eliminate_zero_dimensions` | boolean | no | -- | taxonomy | Renderer hint to hide all-zero dimensions. |

254 +| `context.context_items[]` | array<object> | no | `{value, label}` | taxonomy | Per-widget context item labels for selector-like UI. |

256 +| `context.show_post_aggregations` | boolean | no | -- | taxonomy | FE post-aggregation display toggle. |

258 +| `context.sparkline` | boolean | no | -- | taxonomy | Render compact sparkline form when supported. |

261 +

262 +Item-kind matrix:

263 +

265 +|---|---|---|---|

275 +

276 +For a rich collector example with grids, table widgets, nested groups,

277 +and ownership leaves, read

278 +`src/go/plugin/go.d/collector/mysql/taxonomy.yaml`.

279 +

280 +Widget `contexts[]` entries may be:

281 +

282 +- a literal context string;

283 +- an unresolved literal reference object:

284 + `{context, unresolved: {reason, owner, expires}}`, where

285 + `expires` is `YYYY-MM-DD`;

286 +- a selector reference object with `context_prefix` or

287 + `collect_plugin`.

288 +

289 +Generated output adds `unresolved_references[]` to each placement and

290 +item that aggregates unresolved escape hatches with `context`,

291 +`reason`, `owner`, `expires`, and `item_path`.

292 +

293 +## taxonomy_sections.json

294 +

295 +Schema for `integrations/taxonomy/sections.yaml`. Sections have

296 +stable opaque `id` values and parentage through `parent_id`.

297 +Moving a section means changing `parent_id`, not editing collector

298 +`taxonomy.yaml` files.

299 +

300 +Required fields per section: `id`, `title`, `section_order`,

301 +`status`. Optional fields: `parent_id`, `short_name`, `icon`,

302 +`deprecation`, and `x_*` extensions.

303 +

304 +## taxonomy_output.json

305 +

306 +Schema for generated `integrations/taxonomy.json`. The artifact

307 +contains `taxonomy_schema_version`, `source`, normalized `sections`,

308 +normalized `placements`, and `opted_out_collectors`. Each placement

309 +preserves the ordered item tree and includes `resolved_contexts`

310 +(owned contexts), `referenced_contexts` (display references), and

311 +`unresolved_references` snapshots for CI/review diffing.

312 +

313 ## agent_notification.json

314

315 Single object OR array of objects (oneOf).

.agents/skills/project-writing-collectors/SKILL.md

+19 -15

@@ -179,13 +179,16 @@ A new or modified collector ships these in sync:

179

180 - the code

181 - `metadata.yaml` — drives integration pages, in-app help, alert references

182 +- `taxonomy.yaml` — places emitted chart contexts in the dashboard TOC

183 + with an ordered `items:` tree; structural strings/`owned_context`

184 + entries own contexts, widgets reference them

185 - `config_schema.json` — DYNCFG schema rendered by the dashboard

186 - stock `.conf` — safe, representative example

187 - `health.d/*.conf` — alert templates bound to chart `context`

188 - `README.md` — concise narrative

189 - if exposing a Function: response shape conforming to `src/plugins.d/FUNCTION_UI_SCHEMA.json`

190

188 -Treat them as one unit. Change a unit in code → update `metadata.yaml` in the same commit. Add a config knob → update schema, stock conf, and metadata together.

191 +Treat them as one unit. Change a unit in code → update `metadata.yaml` in the same commit. Add or rename a chart context → update `taxonomy.yaml` or a declared dynamic selector. Add a config knob → update schema, stock conf, and metadata together.

192

193 ### 2.9 Cross-plugin enrichment via netipc

194

@@ -282,20 +285,21 @@ A collector is *production-quality* when it satisfies all of:

285 5. Does the collection cycle allocate, log per iteration, or reconnect every cycle?

286 6. Do error logs answer *what operation, what target, what was expected vs observed*?

287 7. Are config knobs in `config_schema.json` and `metadata.yaml`? Does the stock `.conf` show a representative example?

285 -8. Are alerts present in `health.d/`?

286 -9. Is `README.md` updated? (Not the generated `integrations/<name>.md`.)

287 -10. For remote targets: is vnode wiring done?

288 -11. For SNMP: did I extend a profile rather than hardcode OIDs?

289 -12. For statsd / OTEL: did I document and ship the operator-side config (synthetic_charts file or OTEL mapping YAML)?

290 -13. For Prometheus scraping: are selectors correct? Are untyped metrics handled?

291 -14. For cross-plugin enrichment: am I using netipc?

292 -15. For Functions: does the response conform to one of the six shapes? Non-blocking with respect to the collection loop? Schema-validated?

293 -16. For ibm.d only: did I run `go generate` after touching `contexts.yaml`?

294 -17. For new go.d modules: are all four wiring steps done (init.go, go.d.conf, stock conf, README)?

295 -18. Tests: real fixtures or real instances? Would they catch the bug I just fixed?

296 -19. High-cardinality labels / instances: bounded by `max_*` + selectors? Aggregated "Other" bucket or upstream-supplied aggregation present where applicable?

297 -20. Entities that can go away: obsoleted when the collector knows they're gone? Anti-flip-flop window applied where churn is expected?

298 -21. Production-quality criteria above — would this collector survive hours of target outage without leaks or log floods?

288 +8. Does `taxonomy.yaml` cover every emitted chart context, or are dynamic contexts declared with `metrics.dynamic_context_prefixes` / `metrics.dynamic_collect_plugins`?

289 +9. Are alerts present in `health.d/`?

290 +10. Is `README.md` updated? (Not the generated `integrations/<name>.md`.)

291 +11. For remote targets: is vnode wiring done?

292 +12. For SNMP: did I extend a profile rather than hardcode OIDs?

293 +13. For statsd / OTEL: did I document and ship the operator-side config (synthetic_charts file or OTEL mapping YAML)?

294 +14. For Prometheus scraping: are selectors correct? Are untyped metrics handled?

295 +15. For cross-plugin enrichment: am I using netipc?

296 +16. For Functions: does the response conform to one of the six shapes? Non-blocking with respect to the collection loop? Schema-validated?

297 +17. For ibm.d only: did I run `go generate` after touching `contexts.yaml`?

298 +18. For new go.d modules: are all four wiring steps done (init.go, go.d.conf, stock conf, README)?

299 +19. Tests: real fixtures or real instances? Would they catch the bug I just fixed?

300 +20. High-cardinality labels / instances: bounded by `max_*` + selectors? Aggregated "Other" bucket or upstream-supplied aggregation present where applicable?

301 +21. Entities that can go away: obsoleted when the collector knows they're gone? Anti-flip-flop window applied where churn is expected?

302 +22. Production-quality criteria above — would this collector survive hours of target outage without leaks or log floods?

303

304 ## 5. Plugins and frameworks — what's available and where

305

.agents/sow/current/SOW-0016-20260510-collector-taxonomy-unification.md new

+728

@@ -0,0 +1,728 @@

1 +

2 +# SOW-0016 - Unify collector metric taxonomy with Cloud-Frontend dashboard TOC

3 +

4 +## Status

5 +

6 +Status: in-progress

7 +

8 +Sub-state: one-PR framework+POC scope locked by user on 2026-05-11 after the audit-phase start. SOW moved from `pending/` to `current/`; on 2026-05-14 the user superseded the "structural taxonomy only" v1 boundary and required v1 to cover full cloud-frontend TOC shapes, including grids, context/table widgets, ordered alternatives, and nested groups. The full-shape v1 contract was redesigned, adversarially reviewed, amended, and re-reviewed as READY TO IMPLEMENT. Implementation has resumed under the full-shape contract. Full collector taxonomy coverage, global all-collector fatality, production ibm.d sweep, and cloud-frontend consumption are follow-up work unless explicitly pulled into the POC by user decision. After each major implementation step, if an external Claude review would add value, provide a focused prompt with exact files and questions.

9 +

10 +Implementation progress 2026-05-11: framework, schemas, generator/checker/seed tooling, CI wiring, docs/spec/skills updates, and five go.d POC collector taxonomies were implemented locally and committed as a framework POC snapshot. 2026-05-14: structural-only implementation was paused, full-shape v1 was redesigned/reviewed, and the framework/POC files were updated locally to the ordered recursive `items:` contract.

11 +

12 +## Requirements

13 +

14 +### Purpose

15 +

16 +Eliminate cross-repo taxonomy drift between Netdata's public collector definitions and the private cloud-frontend dashboard TOC. Ownership moves next to collectors. Validation lives entirely in the public netdata repo. Cloud-frontend consumes a generated JSON artifact, exactly as it consumes `integrations.js` today.

17 +

18 +**Scope of this SOW (Netdata-only)**: framework schemas, generator, validators, CI gates, seed tooling, contributor docs/skills, and a small POC set of collector `taxonomy.yaml` files. The published artifact `integrations/taxonomy.json` is the contract surface. Full collector taxonomy coverage is deliberately out of the initial PR.

19 +

20 +**Out of scope**: cloud-frontend consumption work (consumer module, legacy taxonomy module removal, chart-spec extraction, `dynamicSections` removal, regex-catchall removal, rollback runbook). The FE team owns Phase B on their schedule; tracked as a downstream FE-team SOW.

21 +

22 +### User Request

23 +

24 +> The collector definitions and frontend taxonomy are disconnected. Currently:

25 +> - Collectors define metrics and contexts in `metadata.yaml`

26 +> - Cloud frontend separately defines taxonomy/TOC mappings in JS

27 +> - There is no validation that taxonomy entries reference valid metrics/contexts

28 +> - CI cannot validate consistency because:

29 +> - `netdata` repo is public

30 +> - `cloud-frontend` repo is private

31 +> - we cannot grant the public repo access to the private repo

32 +> This causes taxonomy drift and broken references.

33 +>

34 +> Move taxonomy ownership closer to collectors. Each collector should define its own taxonomy in YAML near the collector itself (either embedded in `metadata.yaml`, or stored in a dedicated taxonomy YAML file). Then Python scripts should aggregate all collector taxonomy YAML files, produce one normalized JSON artifact. Cloud frontend CI will later consume this JSON and generate JS code (out of scope).

35 +

36 +User refinements (2026-05-10):

37 +

38 +> we dont care about less churn but clean end state. Lets do as much as possible now, without phase1/2 - i mean about the framework (creating taxonomy.yaml for each collector is routine work).

39 +

40 +> about isSingleNode - this is important. Different view depends on the view, we need this in the taxonomy.

41 +

42 +User refinement (2026-05-11):

43 +

44 +> we will do everything in one PR (framework - w/o adding taxonomy for all collectors, can add a few as a POC).

45 +

46 +User refinement (2026-05-14):

47 +

48 +> There is no need to narrow the scope of v1, it should cover everything. Less churn is not our concern.

49 +

50 +User decision (2026-05-14):

51 +

52 +> Pause implementation, redesign the v1 full-TOC schema/contract first, then run adversarial review before implementation resumes.

53 +

54 +User decision (2026-05-14):

55 +

56 +> Cloud-frontend is not written in stone for this phase. SOW-0016 may define a clean Netdata-side taxonomy contract that requires downstream FE adapter/renderer changes, as long as those FE changes are clean and not compatibility hacks.

57 +

58 +### Assistant Understanding

59 +

60 +Facts (established from 3 independent Opus 4.7 analysis reports under `.local/audits/taxonomy-design/`):

61 +

62 +- `cloud-frontend/src/domains/charts/toc/taxonomy/` is ~15,160 LoC across 18 JS files. It is a dashboard-composition DSL, not a flat taxonomy: it mixes section structure, context references, regex catchalls, function-typed entries (`({ isSingleNode }) => ...`), grids of pre-configured chart widgets (`type: "grid"`), and chart-spec bodies (`type: "context"`). Only ~5–10% of the LoC is pure structural taxonomy.

63 +- The integrations marketplace axis (`integrations/categories.yaml` + `meta.monitored_instance.categories`) is **orthogonal** to the dashboard TOC: MySQL is `data-collection.databases` in the catalog and `Applications > MySQL` in the TOC. The two axes have different cardinality, different routing target (catalog page vs in-app dashboard), different consumers.

64 +- The bridge token between collectors and the TOC is the chart context name (e.g. `mysql.queries`). Contexts are already declared in `metadata.yaml` under `metrics.scopes[*].metrics[*].name` (`integrations/schemas/collector.json:264-394`). This is the cross-reference key.

65 +- The integrations pipeline (`integrations/gen_integrations.py`, 1469 LoC) discovers metadata.yaml files via `COLLECTOR_SOURCES` (`gen_integrations.py:27-37`), validates against JSON schemas (Draft7), and emits `integrations/integrations.{js,json}` plus per-integration markdown via Jinja templates. CI is `.github/workflows/generate-integrations.yml` and `check-markdown.yml`. Warnings become fatal in CI via `fail_on_warnings()` (`gen_integrations.py:155-174`).

66 +- ibm.d collectors generate `metadata.yaml` from `contexts.yaml` via `go generate`. Embedding taxonomy into `metadata.yaml` would force generator-on-generator complexity. ibm.d is the structural reason to use a sibling file.

67 +- All three independent analyses converged unanimously on: sibling-file design, cross-cutting parent registry, categories-vs-TOC are different axes, cross-reference validation as the load-bearing CI check, ICOn registry as a string-keyed allowlist with FE-side asset map.

68 +

69 +Inferences (not directly stated):

70 +

71 +- "Clean end state, no framework phasing" now applies to the framework PR only: the framework lands in one delivery with POC collector taxonomies. It does not imply a full initial taxonomy sweep for every collector.

72 +- The `isSingleNode` requirement implies that view-conditional rendering exists at the section/structure level, not just inside chart-spec bodies. The schema must support this as a first-class concept.

73 +- Superseded 2026-05-14: v1 is no longer limited to section/context taxonomy. The public contract must model full TOC shape where needed for parity: ordered items, structural groups, owned context leaves, flattening groups, selector leaves, grids, context/table widgets, first-available alternatives, and view-conditioned item bodies. `include_charts:` handles remain absent; the replacement is explicit typed item bodies in `taxonomy.yaml`.

74 +

75 +Unknowns (resolve before the implementation step that depends on them):

76 +

77 +- Whether function-typed entries appear ONLY inside chart-spec bodies, or also at section-structure level. If the latter, the schema needs richer condition expressions than `view: single_node | multi_node`.

78 +- Whether `families: true` semantics depend on Netdata's `family` chart attribute (which may be deprecating in some flows). Maintainer confirmation needed before locking the schema.

79 +- The exhaustive set of icon keys used across all 18 cloud-frontend taxonomy files (~175 from `icons.js`, but verify by grep).

80 +- Whether `virtualContexts` ever appear at section-structure level (vs only inside chart-spec bodies). If at structure level, schema needs a `virtual:` opt-in.

81 +- Whether ibm.d's `contexts.yaml` schema needs extension to carry canonical `section_id`, `priority`, and subsection/placement metadata for production codegen. This only blocks the initial PR if an ibm.d collector is selected as a POC.

82 +

83 +### Acceptance Criteria

84 +

85 +- Audit evidence required by a framework component exists before that component lands. There is no separate 10-output audit gate before implementation; unresolved design forks must not be hidden in code.

86 +- **Full-shape redesign gate (added 2026-05-14)**: satisfied. The ordered-`items:` v1 authoring/output contract was written, amended after adversarial review, and re-reviewed. Draft/review artifact: `.local/audits/taxonomy-design/full-shape-v1-redesign.md`.

87 +- `integrations/_common.py` extracted; existing `integrations/integrations.json` AND `integrations/integrations.js` are byte-identical before/after the refactor (verified by the `diff -u` baseline-copy procedure in the implementation plan, not by `git diff --exit-code`, because these outputs are gitignored ephemeral artifacts). **Gate-fatal.**

88 +- `integrations/gen_taxonomy.py` exists, runs in CI, schema-validates every committed `taxonomy.yaml`, cross-references against `metadata.yaml` contexts, and emits `integrations/taxonomy.json`. Verified by green CI on the single implementation PR.

89 +- `integrations/gen_taxonomy_seed.py` exists and seeds flat structural `items:` lists from `metadata.yaml`. Documented in the integrations contributor docs with the direct script command. **Initial PR hard requirement.**

90 +- `integrations/check_collector_taxonomy.py` exists as a fresh wrapper around `_common.py` and taxonomy validators (NOT a clone of the stale `check_collector_metadata.py`).

91 +- Output is **deterministic**: re-running `gen_taxonomy.py` 10× on identical input produces byte-identical `taxonomy.json`. Verified by golden test.

92 +- Cross-reference validator catches every TAX invariant in v1 (TAX001–TAX025, TAX028–TAX038; TAX026/TAX027/TAX040–TAX042 removed with `only_views:` drop and chart-recipe-manifest removal). Lint-code matrix in spec doc lists every code with severity, example, remediation, and superseded codes.

93 +- **Selector overlap detection**: across all three selector types (`contexts`, `context_prefix`, `collect_plugin`), pairwise intersection raises a fatal error.

94 +- **Closed core schema (Decision 13)**: misspelled field names (e.g. `single-node`, `include_chart`) fail the schema validator. Verified by negative tests.

95 +- Every collector whose `metadata.yaml` metrics block or `taxonomy.yaml` is touched in the implementation PR has matching taxonomy coverage in the same PR (fatal — Decision 12). Global all-collector coverage is informational/warning only in this SOW.

96 +- `taxonomy_optout: { reason: "..." }` is a top-level per-collector authoring object in `taxonomy_collector.json`; it is mutually exclusive with `placements`, cannot appear inside a placement, and requires a non-empty reason. `inline_dynamic_declarations` may appear alongside `taxonomy_optout` only for no-metadata plugins documented by audit 1.6. `taxonomy_output.json` carries opt-out collectors in a separate `opted_out_collectors` array, not as empty placements.

97 +- Every structural literal owner and widget literal reference resolves to a real declared context unless the exact widget reference carries the explicit unresolved-reference escape hatch (TAX003 fatal from day 1; TAX038 warns when an escape hatch becomes stale).

98 +- Every `context_prefix:` is declared in `metadata.yaml.metrics.dynamic_context_prefixes:` of the owning collector (or inline in `taxonomy.yaml` for plugins without `metadata.yaml`); TAX031 fatal.

99 +- Every `collect_plugin:` is declared in `metadata.yaml.metrics.dynamic_collect_plugins:` (or inline); TAX035 fatal.

100 +- Every `context_prefix_exclude:` is paired with a `context_prefix:` and contains valid prefix strings; otherwise TAX029 fatal.

101 +- Every `section_id` resolves against `integrations/taxonomy/sections.yaml`. `section_path:` authoring is rejected by the closed v1 schema. Verified by validator (fatal).

102 +- Every `icon` is in `integrations/taxonomy/icons.yaml` allowlist. Verified by validator (fatal).

103 +- ~~`include_charts:` validation~~ — **REPLACED 2026-05-14**: no `include_charts:` handle namespace in v1. Full-shape parity is represented directly by typed ordered `items:` entries such as `owned_context`, `group`, `flatten`, `selector`, `grid`, `context`, `first_available`, and `view_switch`.

104 +- Production ibm.d `taxonomy.yaml` codegen is not required in the initial framework+POC PR unless an ibm.d collector is selected as a POC. If included, `go generate ./src/go/plugin/ibm.d/modules/... && git diff --exit-code` must be clean for the touched module(s).

105 +- `integrations/taxonomy.json` matches `integrations/schemas/taxonomy_output.json` (self-validation in `gen_taxonomy.py`). Output schema version is `taxonomy_schema_version: 1`.

106 +- Output JSON includes `source: { netdata_commit, generated_at }` metadata (Decision 3 amendment).

107 +- Output JSON carries unresolved selectors, build-time-resolved owned-context snapshots, and display-reference snapshots per placement/item (Decision 2 amendment plus 2026-05-14 ownership/reference split).

108 +- Legacy diff tooling and full drift triage are follow-up work for the full collector migration, not acceptance criteria for the initial framework+POC PR.

109 +- Performance budget: full taxonomy validation completes in <5 seconds on the current fleet; synthetic 10K-context fixture completes in <10 seconds. Verified by CI timing.

110 +- `Finding` model emits valid GitHub Actions annotations (`::error file=PATH,line=N,title=TAXNNN::MESSAGE`), text, JSON sidecar, and optional SARIF.

111 +- Collector consistency policy includes `taxonomy.yaml` as the dashboard TOC placement artifact. Documented in `AGENTS.md` and `.agents/skills/integrations-lifecycle/`.

112 +- **Cloud-frontend Phase B is NOT a SOW-0016 acceptance criterion** (out of scope per 2026-05-11 user clarification). Netdata-side SOW closes when the single framework+POC PR merges with green validation. FE-team Phase B is tracked separately on their schedule.

113 +

114 +## Analysis

115 +

116 +Sources checked:

117 +

118 +- `/Users/ilyam/Projects/github/ilyam8/cloud-frontend/src/domains/charts/toc/taxonomy/` — 18 JS files, fully read by 3 independent agents.

119 +- `/Users/ilyam/Projects/github/ilyam8/netdata/src/go/plugin/go.d/collector/mysql/metadata.yaml` and ~5 other representative collectors (apache, postgres, nvidia_smi, snmp, db2/ibm.d).

120 +- `/Users/ilyam/Projects/github/ilyam8/netdata/integrations/gen_integrations.py:1-1469` — full pipeline read.

121 +- `/Users/ilyam/Projects/github/ilyam8/netdata/integrations/schemas/collector.json` — collector schema (627 lines).

122 +- `/Users/ilyam/Projects/github/ilyam8/netdata/integrations/categories.yaml` — catalog axis.

123 +- `/Users/ilyam/Projects/github/ilyam8/netdata/.github/workflows/generate-integrations.yml` and `check-markdown.yml`.

124 +- `.agents/skills/integrations-lifecycle/` — current pipeline knowledge.

125 +- `.agents/sow/specs/` — checked for prior taxonomy specs (none).

126 +- 3 independent Opus 4.7 agent reports under `.local/audits/taxonomy-design/`.

127 +

128 +Current state:

129 +

130 +- Cloud-frontend taxonomy is hand-maintained JS, no validation, no cross-reference to collector contexts. Drift is invisible until a chart fails to render.

131 +- `metadata.yaml` already declares the full set of contexts every collector emits (`metrics.scopes[*].metrics[*].name`). All cross-reference data needed by the new validator is already present in the public repo — no new data sources required.

132 +- `gen_integrations.py` is the obvious plug-in point. Discovery, schema validation, warning-fatal-in-CI patterns exist and can be reused.

133 +- `meta.monitored_instance.categories` is structurally separate from the TOC. No code in the cloud-frontend taxonomy references it.

134 +- ibm.d's `contexts.yaml` codegen pattern (`go generate` emits `metadata.yaml`) extends naturally to also emit `taxonomy.yaml`.

135 +

136 +Risks:

137 +

138 +- **Function-typed entries (`isSingleNode` and similar)**: simple scalar/list deltas remain handled by the curated-and-override pattern (`single_node:` sparse block). Full item-body switches are now in scope for v1 via the proposed `type: view_switch` item; review must confirm this covers every current `({ isSingleNode }) => ...` occurrence without reintroducing `only_views:`.

139 +- **Regex non-equivalence between JS and Python**: irrelevant since Decision 1 drops regex entirely. Risk eliminated.

140 +- **`families: true` may depend on a deprecating attribute**: medium risk. Mitigation: confirm semantics before finalizing the schema; if `family` is being phased out, schema gets `group_by_label: <label>` as a sibling/replacement field before the generator/checker ships.

141 +- **Catalog vs TOC contributor confusion**: low/medium risk. Mitigation: explicit guidance in `AGENTS.md` collector-consistency rule + integrations-lifecycle skill update; pre-commit lint flags suspicious `categories:` edits that look like TOC tweaks.

142 +- **Full migration size**: ~150 collector `taxonomy.yaml` files remain out of the initial PR. Mitigation: the initial PR proves the framework and POC shapes; full migration gets its own follow-up SOW/PR plan.

143 +- **Downstream FE consumption timing**: if the FE team consumes `taxonomy.json` later than the Netdata framework PR, the public repo temporarily publishes an unconsumed artifact. Acceptable — the framework is still useful for validation and later migration.

144 +- **Coverage-fatal flip timing**: global all-collector fatality is deliberately deferred. The initial PR enforces changed/touched collector coverage only, so unrelated collector PRs are not blocked by missing taxonomy files.

145 +- **Schema evolution**: future view axes (beyond `single_node | multi_node`) will require a schema bump. Mitigation: `taxonomy_version: 1` is pinned in every file; major bumps are explicit and gate-able. Core authoring fields use **closed schemas** (no broad `additionalProperties: true` for `single_node`, selector declarations, etc.); only namespaced extension keys (`x_*`) are permitted on core nodes (Decision 13).

146 +- **Virtual contexts at section-structure level (factually present, not absent)**: `cloud-frontend/.../taxonomy/systemStorage.js:3-31` defines non-empty `virtualContexts`, and one is consumed in the taxonomy structure at `systemStorage.js:103`. The SOW's previous "confirm none" framing of audit step 1.4 is wrong. Mitigation: rewrite audit 1.4 to classify every `virtualContexts` def/use and assign one of {frontend recipe handle, encoded in generated contract, explicit diff exception, deferred with new SOW}.

147 +- **`netdata.*` negative-lookahead selector cannot be expressed by the three-matcher set**: `cloud-frontend/.../taxonomy/netdata.js:79-83` uses `^netdata\.(?!(ebpf|statsd|apps|tcp_connects|tcp_connected|private_charts|machine_learning|training|metric_types|queue_ops|queue_size|plugin)).*`. A bare `context_prefix: ["netdata."]` would over-claim contexts that the cloud-frontend deliberately routes elsewhere. Mitigation: Decision 2 amended to require an explicit per-collector exclusion field (or static enumeration); regex remains forbidden.

148 +- **First-available chart alternatives are in scope after the 2026-05-14 decision**: `cloud-frontend/.../charts/toc/getMenu.js:77-82` and `:91-98` execute "if item is array, choose first available context" semantics. Used in Kubernetes (`kubernetes.js:139-155, 183-198, 273-288, 337-352, 405-421, 445-461, 485-500, 527-538`), Containers/VMs (`containersAndVms.js:311-324, 347-365, 574-593, 616-674, 702-712`), and Pulsar grid (`applications.js:3902-4001`). Mitigation: v1 redesign adds a typed `first_available` item whose alternatives are fully validated against metadata and preserved for FE runtime selection.

149 +- **`_collect_plugin` selector feasibility is unproven on the FE side**: Agent stores the label on RRDSETs (`src/database/rrdset-index-id.c:23-27`), but `getMenu.js:50-56` filters by chart id, not by labels. Mitigation: audit step 1.10 remains a non-blocking coordination note; Netdata can publish the selector contract, and FE may adapt or open a selector-replacement SOW.

150 +- **Cloud-frontend JSON consumption is plausible but unproven (mitigation revised 2026-05-11)**: current FE consumes `integrations.js` via `cloud-frontend/.github/workflows/sync-to-s3.yaml:47-67`, not a taxonomy JSON. After scope correction (FE out of scope of SOW-0016), this is the FE team's responsibility — they extend their `sync-to-s3.yaml` to also run `gen_taxonomy.py` and copy `integrations/taxonomy.json` into their tree, exactly as they already do for `integrations.js`. SOW-0016 publishes the artifact and the `taxonomy_output.json` schema; consumption is downstream. If the FE team finds the consumption infeasible (audit 1.10 coordination response), that's a downstream FE-SOW design problem, not a blocker on SOW-0016.

151 +- **FE consumption is non-trivial but downstream**: `applications.js` is 5,885 LoC, `kubernetes.js` 1,228, `systemStorage.js` 1,567, `systemHardware.js` 1,241, `containersAndVms.js` 1,165; `contexts.js` is 34,055 LoC. Existing FE tests cover overview-vs-single-node flavor selection (`getMenu.test.js:326-343`), menu ancestry (`:369-423`), regex sections (`:450-506`), grids (`:509-517`), and virtual contexts (`:520-528`). Mitigation: this SOW does not gate on FE refactor timing; FE Phase B is tracked separately.

152 +- **Rollback after framework+POC PR can leave metadata declarations behind**: POC collector `metadata.yaml` may carry new `dynamic_*` declarations even if POC taxonomy files are reverted. Mitigation: Rollback Matrix records whether reverted `dynamic_*` fields are removed with the taxonomy revert or intentionally retained as accurate emission facts.

153 +- **Full collector author cost is not "routine"**: at the postgres rate (70 explicit context lines), 150 collectors implies ~10,500 explicit context lines plus structure/overrides/comments. Mitigation: `gen_taxonomy_seed.py` is an initial PR hard requirement so follow-up migration is seed + human review, not hand authoring from scratch.

154 +- **Stale `check_collector_metadata.py` reuse trap**: it imports `SINGLE_PATTERN`, `MULTI_PATTERN`, `SINGLE_VALIDATOR`, `MULTI_VALIDATOR` from `gen_integrations` (`integrations/check_collector_metadata.py:8-9`), but the current generator defines none of those symbols; it is not wired into any active workflow. Mitigation: Decision 4 amended; `check_collector_taxonomy.py` is fresh, not a clone.

155 +- **Path-as-identity is brittle**: if a collector authored a path such as `[applications, postgres]`, moving postgres under an intermediate Databases section would mass-edit every collector taxonomy referencing it. Mitigation: Decision 8 locks `section_id` as an opaque stable handle; dots in an ID are namespace punctuation only and do not define parentage. Section moves are expressed in `sections.yaml` via `parent_id` changes, not by editing collector YAML; `taxonomy.json` carries both stable `id` and resolved `path`.

156 +- **Selector lookup performance**: naive `[k for k in CONTEXT_INDEX if k.startswith(prefix)]` per-prefix per-placement is O(P·C·placements). Mitigation: validation algorithm specifies sorted-context bisect for prefix and an index for collect-plugin; synthetic 10K-context performance fixture added to acceptance criteria.

157 +

158 +## Pre-Implementation Gate

159 +

160 +Status: **SATISFIED 2026-05-14** after the amended full-shape v1 contract review cleared and the framework+POC implementation resumed. The prior structural-only gate was superseded; the active gate is now satisfied for the single framework+POC PR scope.

161 +

162 +Problem / root-cause model:

163 +

164 +- Two repositories (public netdata, private cloud-frontend) own complementary halves of the same taxonomy: collectors emit chart contexts in `metadata.yaml`; cloud-frontend renders them via hand-maintained JS modules under `domains/charts/toc/taxonomy/`. There is no machine-readable contract between the two halves, no CI gate that crosses the boundary, and no validation that taxonomy entries reference real contexts. Drift accumulates silently until a chart fails to render or a collector renames a context that some taxonomy file still references. The fix is structural: move the dashboard TOC taxonomy contract (sections, paths, contexts, families flag, icon keys, view-conditional rendering, and full-shape item bodies where needed for parity) into the public repo as collector-adjacent YAML; build a Python aggregator that fails CI on any cross-reference mismatch; emit a JSON artifact the cloud-frontend consumes.

165 +

166 +Evidence reviewed:

167 +

168 +- `cloud-frontend/src/domains/charts/toc/taxonomy/index.js:13-100` (root dashboards map), `applications.js` (5,885 LoC), `kubernetes.js`, `system.js`, `containersAndVms.js`, `netdata.js`, `icons.js` — all read in full by 3 agents.

169 +- `netdata/integrations/gen_integrations.py:1-1469`, `integrations/schemas/collector.json:1-627`, `integrations/categories.yaml`.

170 +- `netdata/src/go/plugin/go.d/collector/mysql/metadata.yaml` and 5+ other representative collectors.

171 +- `.github/workflows/generate-integrations.yml`, `check-markdown.yml`.

172 +- `.agents/skills/integrations-lifecycle/` — existing pipeline knowledge.

173 +- 3 independent Opus 4.7 analysis reports: `.local/audits/taxonomy-design/agent-{1,2,3}-analysis.md`.

174 +- 3 schema-readability persona reviews: `.local/audits/taxonomy-design/schema-review-{1,2,3}.md`.

175 +- 7-collector empirical schema validation: `.local/audits/taxonomy-design/schema-validation-drafts.md`.

176 +- External Codex review (6 parallel GPT-5.5-xhigh subagents + synthesis, 2026-05-10):

177 + - Synthesis: `.local/audits/taxonomy-design/external-review-codex/SYNTHESIS.md`

178 + - 01 Schema correctness: `.local/audits/taxonomy-design/external-review-codex/01-schema-correctness.md`

179 + - 02 Readability: `.local/audits/taxonomy-design/external-review-codex/02-readability.md`

180 + - 03 Pipeline: `.local/audits/taxonomy-design/external-review-codex/03-pipeline.md`

181 + - 04 Migration: `.local/audits/taxonomy-design/external-review-codex/04-migration.md`

182 + - 05 Cross-domain: `.local/audits/taxonomy-design/external-review-codex/05-cross-domain.md`

183 + - 06 Readiness: `.local/audits/taxonomy-design/external-review-codex/06-readiness.md`

184 +- Direct file:line evidence cited in this gate (selected, non-exhaustive):

185 + - `cloud-frontend/.../taxonomy/index.js:13-62` — root dashboards map; `dynamicSections: true` for room and node.

186 + - `cloud-frontend/.../charts/toc/getMenu.js:50-75` — regex/test entries; `:77-82` and `:91-98` first-available array semantics; `:89` `isSingleNode` derivation; `:100-102` function-typed grid items; `:171-180` virtual-context rendering; `:238-350` family hierarchy; `:425-439` dynamic fallback for unmatched contexts.

187 + - `cloud-frontend/.../taxonomy/getMenu.test.js:326-343` overview/single-node flavor; `:369-423` menu ancestry; `:450-506` regex sections; `:509-517` grids; `:520-528` virtual contexts.

188 + - `cloud-frontend/.../taxonomy/systemStorage.js:3-31` non-empty `virtualContexts`; `:103` virtual-context referenced inside taxonomy structure.

189 + - `cloud-frontend/.../taxonomy/netdata.js:79-83` negative-lookahead regex over `netdata.*`.

190 + - `cloud-frontend/.../taxonomy/system.js:44-68, 117-168` and `systemMemory.js:35-80` and `remoteDevices.js:5-15, 93-131` and `containersAndVms.js:3-152, 311-365, 574-712` and `applications.js:3902-4001` (Pulsar) — function-typed entries and array alternatives.

191 + - `cloud-frontend/.github/workflows/sync-to-s3.yaml:47-67` — current FE artifact ingestion path; copies `integrations.js`, not taxonomy JSON.

192 + - `cloud-frontend/package.json:151-165` — no current taxonomy artifact import path.

193 + - `netdata/integrations/gen_integrations.py:27-37` `COLLECTOR_SOURCES`; `:155-174` warning-fatal; `:177-238` Draft7+Registry; `:326-339` collector globbing; `:379-407` `load_collectors`; `:822-830` ID synthesis; `:839-842` deterministic sort; `:861-996` mutating render; `:1414-1428` artifact emission; `:1431-1465` exit/fail handling.

194 + - `netdata/integrations/check_collector_metadata.py:8-9` — imports stale symbols from `gen_integrations`; not wired into active workflows.

195 + - `netdata/integrations/schemas/collector.json:264-395` `metrics` block; `:283-395` static metric context schema; no existing `additionalProperties: false`.

196 + - `netdata/.github/workflows/generate-integrations.yml:1-24, 47-66, 64-81` — current path triggers, generation, artifact cleanup.

197 + - `netdata/.github/workflows/check-markdown.yml:3-11` — current changed-file path triggers (no taxonomy paths).

198 + - `netdata/src/go/plugin/ibm.d/AGENTS.md:28-51` — generated files, source-of-truth files (`contexts.yaml`, `config.go`, `module.yaml`).

199 + - `netdata/src/go/plugin/ibm.d/docgen/main.go:27-47` — current `Context` struct (no taxonomy fields); `:119-153` and `:360-387` and `:389-529` generation flow.

200 + - `netdata/src/go/plugin/ibm.d/modules/db2/generate.go:1-3` and `modules/db2/contexts/doc.go:1-5` — module-level `go generate` invocations.

201 + - `netdata/src/collectors/statsd.plugin/example.conf:6-14, 30-43` — user-supplied app names and arbitrary user-defined chart contexts.

202 + - `netdata/src/collectors/statsd.plugin/statsd.c:1596-1628, 2254-2274` — `statsd.plugin` plugin label.

203 + - `netdata/src/database/rrdset-index-id.c:23-27` — `_collect_plugin` and `_collect_module` labels on RRDSETs.

204 + - `netdata/src/go/plugin/go.d/collector/mysql/metadata.yaml:1138-1149` — `mysql.galera_open_transactions` exists; no `mysql.open_transactions` (drift evidence vs `applications.js:1957`).

205 +- No external open-source repositories were consulted as design references; cross-domain comparisons (Kubernetes CRDs, OpenAPI/JSON Schema, Prometheus relabel, OpenTelemetry semconv, VS Code/JetBrains marketplace) are documented in `.local/audits/taxonomy-design/external-review-codex/05-cross-domain.md` for context only.

206 +

207 +Affected contracts and surfaces:

208 +

209 +- **New schemas**: `integrations/schemas/taxonomy_collector.json`, `taxonomy_sections.json`, `taxonomy_output.json`.

210 +- **Modified schema**: `integrations/schemas/collector.json` — adds two optional dynamic-context declarations under `metrics:`:

211 + - `dynamic_context_prefixes: [{prefix, reason}, ...]` — for collectors whose dynamic contexts share a namespace prefix (snmp, prometheus scraper, cgroup, apps).

212 + - `dynamic_collect_plugins: [{plugin, reason}, ...]` — for collectors whose dynamic contexts have no shared prefix (statsd.plugin, charts.d.plugin, python.d.plugin).

213 +- **New runtime artifacts**: `integrations/taxonomy/sections.yaml`, `integrations/taxonomy/icons.yaml`, `integrations/taxonomy.json` (emitted; ephemeral gitignored per Decision 3 — matches `integrations.js` precedent).

214 +- **New code**: `integrations/_common.py` (narrow shared extract — see the implementation plan for the explicit allow-list), `integrations/gen_taxonomy.py`, `integrations/gen_taxonomy_seed.py` (Decision 9), `integrations/check_collector_taxonomy.py` (fresh; not a clone of the stale `check_collector_metadata.py`, see Decision 4 amendment).

215 +- **Modified**: `integrations/gen_integrations.py` (refactored to use `_common.py`; byte-identical output requirement is acceptance-gate-fatal), `.github/workflows/generate-integrations.yml` (new step + path triggers including `**/taxonomy.yaml`, `integrations/taxonomy/**`, `integrations/schemas/taxonomy*.json`, `integrations/gen_taxonomy*.py`, `integrations/_common.py`), `check-markdown.yml` (changed-collector taxonomy gate per Decision 12).

216 +- **New per-collector files in this PR**: a small POC set of `<collector>/taxonomy.yaml` files. Full collector coverage is follow-up work.

217 +- **ibm.d framework**: production `contexts.yaml` schema/codegen extension is follow-up unless an ibm.d collector is selected as a POC.

218 +- **Cloud-frontend (OUT OF SCOPE of SOW-0016)**: FE team owns consumption — JSON consumer module, legacy taxonomy module removal, renderer adapter, `dynamicSections` removal, regex-catchall removal, rollback. Tracked as a downstream FE-team SOW. SOW-0016 publishes `integrations/taxonomy.json` and the `taxonomy_output.json` schema; that is the entire Netdata-FE contract surface.

219 +- **Project policy**: `AGENTS.md` collector-consistency rule includes `taxonomy.yaml`.

220 +- **Skills**: `integrations-lifecycle/` updated; `project-writing-collectors/` updated.

221 +- **Docs**: collector contributor docs reference `taxonomy.yaml` as a new required file.

222 +

223 +Existing patterns to reuse:

224 +

225 +- `gen_integrations.py:177-238` — Draft7Validator + Registry pattern.

226 +- `gen_integrations.py:155-174` — `WARNINGS` accumulator + `fail_on_warnings()` for CI-fatal gating; **wrap into a structured `Finding` model with multiple renderers** (text, valid GitHub Actions annotation `::error file=...,line=...,title=TAX003::`, JSON, optional SARIF) — the existing prefix `:warning file=...:` and `:error file=...:` strings are NOT valid GitHub Actions annotations and must not be propagated.

227 +- `gen_integrations.py:326-339` — collector-source globbing.

228 +- `gen_integrations.py:822-830` — collector-id synthesis pattern; reused so taxonomy and integrations agree on collector identity.

229 +- `gen_integrations.py:839-842` — explicit deterministic sort (`_index`, `_src_path`, `id`); taxonomy needs an equivalent locked merge key (Decision 12).

230 +- `integrations/categories.yaml` — frozen-list registry pattern (mirror for `sections.yaml`).

231 +- ibm.d `go generate` codegen pattern remains the production target, but only enters this PR if an ibm.d POC is selected.

232 +- **Do NOT mirror `integrations/check_collector_metadata.py`**: it imports symbols (`SINGLE_PATTERN`, `MULTI_PATTERN`, `SINGLE_VALIDATOR`, `MULTI_VALIDATOR`) the current `gen_integrations.py` no longer defines and is not invoked by any active workflow. Build `check_collector_taxonomy.py` fresh against `_common.py` and the new taxonomy validators.

233 +

234 +Risk and blast radius:

235 +

236 +- **Regression**: existing `gen_integrations.py` is refactored to use `_common.py`. Risk: introducing a regression in the existing pipeline. Mitigation: refactor as pure code-motion (no behavior change); diff the rendered `integrations.json` before/after to confirm byte-identical output.

237 +- **CI runtime**: new pipeline adds another pass. Estimated <5s. Negligible.

238 +- **Compatibility**: `taxonomy.json` is a new artifact; consumers (cloud-frontend) opt in. No breakage of existing artifacts.

239 +- **Performance**: aggregator builds an in-memory context index (~10K entries max). Linear in collector count; well within budget.

240 +- **Security**: no secret-handling involved; all data is public collector metadata.

241 +- **Data loss**: zero. New artifacts; no destructive edits.

242 +- **Migration**: initial PR touches only POC collectors. Full migration of ~150 collectors is follow-up work once the framework shape is proven.

243 +- **Rollout**: one framework+POC PR is reversible by revert. FE Phase B is downstream and outside this SOW.

244 +- **Operational**: zero impact on running agents. Taxonomy is build-time metadata, not runtime.

245 +

246 +Sensitive data handling plan:

247 +

248 +- This work involves zero credentials, secrets, customer data, or private endpoints. All inputs (`metadata.yaml`, cloud-frontend taxonomy JS) are non-sensitive structural metadata. All outputs (schemas, YAML files, generated JSON) are non-sensitive structural metadata. No redaction required in the SOW, specs, skills, code comments, or commits.

249 +- The cloud-frontend repo is private but its taxonomy source files are not sensitive in content; they merely live in a private repo for org-policy reasons. Downstream FE Phase B work is referenced only at a high level; no private source from that repo will be pasted into public artifacts beyond structural references already cited in this SOW.

250 +

251 +Implementation plan (active, 2026-05-11 one-PR framework+POC scope):

252 +

253 +1. **Inline audit evidence before dependent code**: produce the minimum audit outputs needed by the next implementation step under `.local/audits/taxonomy-design/audit/`. The implementation must not encode unresolved `to-decide` rows. Audit 1.10 remains a non-blocking FE coordination note. Audit 1.5 is required only if production ibm.d codegen or an ibm.d POC enters this PR.

254 +2. **Refactor shared integration helpers**: extract the narrow `_common.py` allow-list from `gen_integrations.py`; prove `integrations.{json,js}` are byte-identical before/after using `diff -u` against `.local/audits/taxonomy-design/refactor-baseline/` and `.local/audits/taxonomy-design/refactor-after/`.

255 +3. **Add taxonomy schemas and registries**: author `taxonomy_collector.json`, `taxonomy_sections.json`, `taxonomy_output.json`, `collector.json` dynamic declaration extensions, `integrations/taxonomy/sections.yaml`, and `integrations/taxonomy/icons.yaml`. Use `section_id:` as the only v1 authoring form.

256 +4. **Add generator/checker/seed tooling**: implement `gen_taxonomy.py`, `gen_taxonomy_seed.py`, and fresh `check_collector_taxonomy.py`; include deterministic ordering, selector overlap detection, schema self-validation, structured `Finding` renderers, and a 10K-context performance fixture.

257 +5. **Add POC collector taxonomies**: add a small representative POC set, defaulting to the previously selected reference collectors `mysql`, `postgres`, `apache`, `nvidia_smi`, and `snmp` unless implementation evidence shows one should be swapped. Do not add taxonomy for all collectors in this PR.

258 +6. **Wire CI for this scope**: `check-markdown.yml` is the PR-blocking gate; `generate-integrations.yml` is the post-merge artifact-generation path. Changed/touched collector coverage is fatal; global all-collector coverage remains warning/informational.

259 +7. **Update contributor-facing artifacts**: update `AGENTS.md`, `.agents/skills/integrations-lifecycle/`, `.agents/skills/project-writing-collectors/`, `.agents/sow/specs/taxonomy.md`, and an integrations contributor doc.

260 +8. **Review checkpoints**: after each major step, decide whether an external Claude review is useful. If yes, provide the user a focused prompt with exact files and questions.

261 +9. **Close this SOW**: SOW-0016 completes when the single framework+POC PR merges green and the SOW validation/artifact gates are filled. Full collector migration, production ibm.d sweep, global fatal coverage, drift triage, and FE consumption become follow-up SOWs unless explicitly pulled into this PR.

262 +

263 +Historical superseded PR-A1 / PR-A1.5 / PR-A2 implementation details were removed from the active plan on 2026-05-11 after the user locked the one-PR framework+POC scope. The reasoning remains in the Execution Log for provenance.

264 +

265 +### Drift Triage Process (Follow-Up Full Migration)

266 +

267 +Full legacy drift triage is not part of the initial framework+POC PR. When the follow-up full migration runs `diff_legacy.py`, each finding uses this disposition model:

268 +

270 +|---|---|---|---|---|

276 +

277 +Bulk migration implementer does NOT silently decide product semantics. Every finding has a named owner; the full-migration SOW cannot close while any finding is `to-decide`.

278 +

279 +**Escalation clock (NEW 2026-05-11 per readiness reviewer 1)**: each finding's named owner has **5 business days** from notification to either sign off on the proposed disposition or push back with an alternative. After 5 business days without response, the bulk-migration implementer escalates to the user (project lead) for tie-break. This prevents the full-migration SOW from stalling indefinitely on owner PTO or unresponsiveness. Notifications are recorded in the drift inventory row with a date stamp.

280 +

281 +### Rollback Matrix (Netdata-only)

282 +

284 +|---|---|---|---|---|

285 +| Single framework+POC PR | `_common.py`, gen_taxonomy, schemas, sections.yaml, icons.yaml, seed/check tooling, CI wiring, docs/skills/specs, POC collector taxonomies | Nothing if fully reverted; `gen_integrations.py` returns to pre-refactor state via the same revert | Remove any POC `metadata.yaml.metrics.dynamic_*` declarations if the corresponding POC taxonomy is reverted and the declarations are not intentionally retained as emission facts | Pre-taxonomy-framework state |

286 +| Follow-up full migration | Remaining collector `taxonomy.yaml` files; possible no-metadata plugin handling; global all-collector fatality | If reverted, collector `metadata.yaml` extensions may become **zombie fields** unless explicitly removed in the same revert | Coordinated revert of `metadata.yaml` `dynamic_*` fields or explicit decision to retain them as accurate emission facts | Framework+POC state |

287 +

288 +Cloud-frontend rollback is the FE team's responsibility (Phase B out of scope of SOW-0016).

289 +

290 +### Audit Outputs (Support Evidence, Not A Separate Gate)

291 +

292 +These outputs remain the evidence checklist. Produce each before the implementation step that consumes it; no separate PR-A1 gate exists after the 2026-05-11 one-PR scope correction.

293 +

294 +- `1.1-dsl-inventory.md` — structured taxonomy-DSL inventory.

295 +- `1.2-families-semantics.md` — `families: true` contract from FE consumer + tests.

296 +- `1.3-icons-allowlist.md` — canonical icon-key list.

297 +- `1.4-virtualcontexts.md` — `virtualContexts` classification + dispositions.

298 +- `1.5-ibmd-prototype.md` — `db2` prototype + round-trip evidence (required only before production ibm.d codegen or an ibm.d POC enters scope).

299 +- `1.6-dynamic-collectors.md` — prefix-friendly vs plugin-label-friendly + no-metadata plugin dispositions.

300 +- `1.8-netdata-negative-selector.md` — chosen disposition for `netdata.js:79-83`.

301 +- `1.10-collect-plugin-question.md` — coordination note to FE team (non-blocking).

302 +- `1.11-section-id-map.md` — camelCase → kebab-case section ID map (top-level only).

303 +- `1.11b-sections-tree.md` — full sections.yaml draft (~80–150 entries with parent_id, section_order). NEW 2026-05-11.

304 +

305 +Total: 10 audit outputs (was 9 before reviewer pass surfaced the 1.11.b need), now consumed inline rather than as a standalone kickoff gate.

306 +

307 +**Removed 2026-05-11** (FE work out of scope of SOW-0016):

308 +- ~~`1.7-fe-adapter-spike.md`~~ — FE team's responsibility.

309 +- ~~`1.9-chart-recipes-seed.md`~~ — no chart-recipe manifest in v1 (Decision 14 removed).

310 +

311 +If an implementation step depends on an audit output, that output must exist and contain no unresolved `to-decide` rows before the step lands. Audit 1.10 is non-blocking; we proceed with `collect_plugin:` as the v1 selector unless the FE team raises a structural objection in time to change this PR.

312 +

313 +Validation plan:

314 +

315 +- **Unit / integration tests**:

316 + - `gen_taxonomy.py` runs against fixture `taxonomy.yaml` files; expected JSON output is golden-tested AND deterministic across runs.

317 + - Cross-reference validator: positive test (valid taxonomy passes), negative tests for active validation families including TAX003 (unknown context), TAX021 (unknown view-override key), TAX022 (`multi_node:` block), TAX023 (list-merge attempt), TAX024 (empty `single_node:`), TAX031/TAX035 (selector not declared in metadata), TAX036 (selector ownership overlap), TAX037 (referenced literal context has no owner), and TAX038 (unresolved escape hatch is now stale because the context resolves). TAX002 and TAX032 are reserved follow-up codes, not emitted in the framework+POC PR.

318 + - Selector overlap validator: positive and negative tests for cross-type overlap (static `contexts:` claimed by one collector AND `context_prefix:` matching the same context claimed by another → fatal).

319 + - Schema validator: positive AND negative tests for each schema file, including typo cases (`single-node` vs `single_node`, `include_chart` vs `include_charts`, etc.) — each must fail under the closed core schema (Decision 13).

320 + - Performance: synthetic 10K-context fixture must validate in <10 seconds.

321 + - Deterministic merge: re-run `gen_taxonomy.py` 10× on the same input; output must be byte-identical.

322 + - `_common.py` refactor: byte-identical `integrations.json` AND `integrations.js` before/after via the Plan step 2.2 `diff -u` baseline-copy procedure. **Gate-fatal.**

323 + - `Finding` renderers: text format, GitHub Actions annotation format (must match `::error file=PATH,line=N,title=TAXNNN::MESSAGE` regex), JSON sidecar shape, optional SARIF.

324 + - ibm.d round-trip: required only if production ibm.d codegen or an ibm.d POC enters this PR; touched generated outputs must be clean after `go generate`.

325 + - Drift triage: follow-up full migration requirement, not required for the initial framework+POC PR.

326 +- **Real-use evidence**:

327 + - Single framework+POC PR lands; CI green; `integrations/taxonomy.json` is produced locally/CI and validates against `taxonomy_output.json`.

328 + - POC collectors validate when `gen_taxonomy.py` runs in CI; their `taxonomy.json` slices validate against `taxonomy_output.json`. FE-side rendering verification is FE-team's responsibility (downstream SOW).

329 + - Changed/touched collector coverage is fatal; global all-collector coverage remains warning/informational.

330 + - ibm.d round-trip evidence captured only if an ibm.d POC/codegen change is included.

331 +- **Reviewer findings**: address all maintainer review comments on the single implementation PR. Apply the per-thread iteration discipline from `.agents/skills/pr-reviews/`.

332 +- **Same-failure search**: grep for any other place in either repo that hand-maintains a taxonomy-shaped data structure (e.g. `dashboards.json`, `menu.json`, `routes.json`); confirm none exist or document each.

333 +

334 +Artifact impact plan:

335 +

336 +- **AGENTS.md**: updated. Collector-consistency rule extended to include `taxonomy.yaml`. Project skills index updated to reference taxonomy work.

337 +- **Runtime project skills**:

338 + - `.agents/skills/integrations-lifecycle/` — major update. New section on taxonomy pipeline; pipeline.md, schema-reference.md, per-type-matrix.md, in-app-contract.md all touched.

339 + - `.agents/skills/project-writing-collectors/` — adds `taxonomy.yaml` as a required artifact for new collectors.

340 +- **Specs**: new `.agents/sow/specs/taxonomy.md` describing schema, validation rules, JSON output contract, frozen v1 top-level list, condition vocabulary.

341 +- **End-user/operator docs**:

342 + - `integrations/README.md` (or equivalent) — documents `taxonomy.yaml` for collector contributors.

343 + - Per-collector READMEs — no individual update needed; the new file is mentioned in the collector consistency rule docs.

344 +- **End-user/operator skills**: `docs/netdata-ai/skills/` — no direct impact (skills don't consume the TOC).

345 +- **SOW lifecycle**: this SOW has moved to `.agents/sow/current/` and will move to `.agents/sow/done/` on completion. Status transitions: open → in-progress → completed. No PR split is planned for this SOW after the 2026-05-11 one-PR correction.

346 +

347 +Open-source reference evidence:

348 +

349 +- This work is internal to Netdata's own repositories. No external open-source projects were consulted as design references. The `cloud-frontend` repository is a private Netdata repository, not an open-source dependency.

350 +

351 +Open decisions:

352 +

353 +ALL RESOLVED 2026-05-11. Recorded for traceability:

354 +

355 +1. **Section identity model**: **8.A locked** — stable `section_id` first-class in `sections.yaml`; collector taxonomy references `section_id:` directly; `section_path:` is not accepted in v1. Section IDs are opaque immutable handles; dots are namespace punctuation only. Section moves are `parent_id` edits in `sections.yaml`; collector YAML unaffected.

356 +2. **`only_views:` v1 inclusion**: **DROPPED from v1** per user — "I don't think we need only_views". If audit 1.1 finds an S2 case (whole-section visibility gating), that case spawns a follow-up SOW to add `only_views:` later. Does not block the initial framework+POC PR.

357 +3. **Drift triage owner assignment**: **A locked** — bulk-migration implementer assigns owners themselves based on finding type; named owners review and sign off. Pre-soliciting per finding rejected as overhead.

358 +

359 +Tactical items resolved without user input (recorded as defaults):

360 +

361 +- `taxonomy.json` artifact: gitignored ephemeral, generated in CI, consumed by FE-team's pipeline exactly as `integrations.js` is consumed today (Decision 3 correction).

362 +- Superseded 2026-05-14: no `include_charts:` or chart-recipe manifest in v1, but explicit full-shape typed `items:` are now in scope for v1 parity (Decision 5 reopened by user).

363 +- Phase B (FE switchover) is out of scope; FE team owns consumption on their schedule.

364 +- ibm.d production codegen is follow-up unless an ibm.d collector is selected as a POC.

365 +- Full collector taxonomy coverage and global-all-collector fatality are follow-up work; per-changed-collector coverage is fatal in the implementation PR (Decision 12).

366 +- Icon-key naming: kebab-case in YAML to match `categories.yaml` style.

367 +- No cloud-frontend freeze imposed (FE team owns their repo); any legacy snapshot pinning belongs to the future full-migration/drift-triage SOW.

368 +

369 +## Implications And Decisions

370 +

371 +User decisions locked on 2026-05-10:

372 +

373 +1. **Sibling vs embed**: sibling `taxonomy.yaml` next to `metadata.yaml`. Reason: ibm.d generator-on-generator avoidance; audience separation; file-size pragmatics. All 3 independent agents converged.

374 +2. **Matcher and reference policy (AMENDED 2026-05-14 for full-shape `items:`)**: drop regex entirely. Context ownership and display references are explicit item semantics.

375 + - **Structural literal owners**: a plain string in a structural `items:` array, or `type: owned_context` with `context:`, owns exactly one context. Every owned literal must resolve to the owning collector metadata (TAX003).

376 + - **Display references**: `type: context` widgets carry `contexts:` arrays. These references do not own contexts; each literal must resolve to metadata or carry the explicit `unresolved: {reason, owner, expires}` escape hatch. A resolved literal reference with no structural owner anywhere is TAX037.

377 + - **Selector items**: `type: selector` owns the contexts matched by `context_prefix:` or `collect_plugin:`. Prefix/plugin selectors still require `metadata.yaml.metrics.dynamic_context_prefixes:` or `metadata.yaml.metrics.dynamic_collect_plugins:` declarations. TAX031/TAX035 fire without opt-in.

378 + - **Selector objects inside widgets**: widget `contexts:` arrays may include `{context_prefix: [...]}` or `{collect_plugin: [...]}` selector objects. These reference contexts but do not own them.

379 + - **`context_prefix_exclude:`** is valid only alongside `context_prefix:` on the same item/reference; invalid pairings are TAX029.

380 + - **Overlap detection**: duplicate ownership between non-selector owners is TAX033. Ownership overlap involving a selector is TAX036. Referencing an already-owned context is expected and valid.

381 + - **Resolved/reference snapshots**: every generated placement and item carries `resolved_contexts` (owned contexts) and `referenced_contexts` (display references). For dynamic-context collectors, selector snapshots include only statically-known contexts; runtime selectors cover future emitted contexts.

382 + - **No-metadata collector handling**: a collector without `metadata.yaml` may declare dynamic opt-ins under top-level `inline_dynamic_declarations:`. The validator treats those declarations as equivalent to `metadata.yaml.metrics.*`; if sibling `metadata.yaml` exists, inline declarations are TAX029.

383 + - **Frontend label-access coordination**: audit 1.10 records whether `_collect_plugin` is already reachable from the FE chart-selection data path. If not, SOW-0016 still publishes `collect_plugin:`; FE consumption either exposes the label downstream or opens a future selector-replacement SOW.

384 + - The guardrail preserves drift-elimination: static collectors (mysql, postgres, ...) use structural literal ownership and explicit widgets, not broad dynamic selectors.

385 + - Cross-engine portability concerns are eliminated by string-prefix-only and label-equality semantics (no JS/Python regex divergence).

386 +3. **Output artifact + lifecycle (CORRECTED 2026-05-11 to match existing `integrations.js` precedent; earlier vendoring framing was over-engineered)**: separate `integrations/taxonomy.json`, **gitignored ephemeral**, generated by `gen_taxonomy.py` in netdata CI, consumed by cloud-frontend exactly as `integrations.js` is consumed today (`cloud-frontend/.github/workflows/sync-to-s3.yaml:47-67`).

387 + - **Generation policy**: `gen_taxonomy.py` writes `integrations/taxonomy.json` during CI; `.github/workflows/generate-integrations.yml` cleanup step removes it together with `integrations.{js,json}` (matches existing pipeline at `:64-66`). Local generation for inspection is supported. The file is `.gitignore`d.

388 + - **Versioning (contract-discipline IN the artifact, not in the delivery mechanism)**: output JSON carries `taxonomy_schema_version: 1` at root level (separate from per-file `taxonomy_version: 1` which is the input schema version). The FE consumer is expected to fail its build on unsupported `taxonomy_schema_version` or unknown required fields. This closes the gap that exists for `integrations.js` today (`.agents/skills/integrations-lifecycle/in-app-contract.md:117-120`) without changing the delivery mechanism.

389 + - **Compatibility policy**: within v1, additive optional fields are non-breaking; field removal is forbidden; enum value additions are reviewed; selector semantics, override merge semantics, list-replacement behavior, section path/ID identity, and generated JSON field names are FROZEN. Changing any of those bumps `taxonomy_schema_version`.

390 + - **Deprecation fields**: every section and selector type may carry `status: active|deprecated`, `deprecation: { replacement_id, since, removal_in }`. Consumers warn on deprecated entries; remove-after window is one major schema version.

391 + - **Source commit metadata**: every emitted `taxonomy.json` includes `source: { netdata_commit: <sha>, generated_at: <iso8601> }` for traceability. Whether the FE pins to a specific commit is the FE team's choice, not this SOW's contract.

392 + - **Cloud-frontend consumption**: out of scope of SOW-0016; tracked as a downstream FE-team SOW. This SOW publishes the artifact and the contract; consumption is the FE team's responsibility.

393 + - Reason for the correction: the existing `integrations.js` model is proven, well-understood, and operationally simple. The version-discipline concerns Codex subagents 04/05 raised are addressed by `taxonomy_schema_version` IN the output JSON, regardless of delivery mechanism. Inventing a separate vendoring strategy traded simplicity for theoretical robustness the artifact metadata already provides.

394 +4. **Pipeline (AMENDED per Change 10)**: new `integrations/gen_taxonomy.py` + a NARROWLY scoped extracted `integrations/_common.py` (allow-list in the active implementation plan). `check_collector_taxonomy.py` is fresh, NOT a clone of the stale `check_collector_metadata.py`. Byte-identical `integrations.{js,json}` output before/after the `_common.py` refactor is a gate-fatal acceptance criterion.

395 +5. **Full-shape TOC item contract in v1 (REOPENED 2026-05-14 by user decision)**:

396 + - `taxonomy.yaml` v1 must represent the full cloud-frontend TOC shape needed for parity: ordered `items:`, structural groups, explicit `owned_context` leaves, flattening groups for legacy `justGroup` semantics, selector leaves, grids, context/table widgets, first-available alternatives, and view-conditioned item bodies.

397 + - `include_charts:` remains absent. v1 does not use opaque chart-handle references or a chart-recipe manifest; it carries explicit typed item bodies where the legacy FE taxonomy carries explicit widget bodies.

398 + - Cloud-frontend consumption remains downstream/out of scope. The FE team owns the renderer adapter and legacy taxonomy removal. The current FE code is evidence for semantics, not a frozen object-shape contract; the Netdata artifact may require clean downstream FE changes.

399 + - Ownership and display references are separate: structural `owned_context` and `selector` items own contexts; `context` widgets, grids, alternatives, and view-switch widget bodies reference contexts and must validate them without tripping duplicate-ownership checks.

400 + - Widget `contexts:` arrays may contain literal context strings, explicit unresolved-reference objects with reason/owner/expiry, or selector objects (`context_prefix`, `context_prefix_exclude`, `collect_plugin`). Selector objects inside widgets reference contexts; selector items under `items:` own contexts.

401 + - TAX003 and TAX037 are fatal by default. Intentional staged/legacy unresolved references require the explicit unresolved-reference escape hatch; warning-by-default drift is rejected. TAX036 remains reserved for existing selector-overlap conflicts.

402 + - TAX038 warns when an unresolved-reference escape hatch has become stale because the context now resolves in metadata.

403 + - Renderer-private payloads are fenced under `renderer:`. Core item objects remain closed to preserve typo detection; known renderer keys are `overlays`, `url_options`, and `toolbox_elements`, and future renderer-only additions use `x_*`.

404 + - String shorthand is allowed only in structural positions (`placement.items`, `group.items`, `flatten.items`). It is rejected inside `grid.items`, `first_available.items`, and `view_switch` branches.

405 + - `flatten` is rejected inside `view_switch` branches; `first_available` alternatives are display-only object items and cannot own contexts.

406 + - The executable schema contract must include a per-type closed-field matrix and recursion matrix equivalent to the amended design artifact.

407 + - Reason: user directive 2026-05-14 — "There is no need to narrow the scope of v1, it should cover everything. Less churn is not our concern."

408 +6. **ibm.d**: production codegen remains the target architecture, but after the 2026-05-11 one-PR scope correction it is not required in the initial framework+POC PR unless an ibm.d collector is selected as a POC. Reason: full ibm.d generation belongs with full collector coverage, not with the minimal framework proof.

409 +7. **`dynamicSections`**: legacy FE root fallback, not a collector `taxonomy.yaml` field. If downstream FE needs dynamic fallback during migration, the clean Netdata-side home is a generated root/section option derived from `sections.yaml`; the FE team still owns the renderer behavior and removal timing. No Netdata-side feature flag.

410 +8. **Frozen v1 top-level sections + stable section identity (LOCKED 8.A on 2026-05-11)**: top-level frozen list `system, kubernetes, containers-vms, synthetic-checks, remote-devices, otel, azure-monitor, applications, netdata` (mirrors `cloud-frontend/.../taxonomy/index.js:25-35` per audit 1.11 spelling map). Adding a new top-level requires a PR on `sections.yaml`.

411 + - **Identity model: 8.A (locked by user 2026-05-11)**. Stable `section_id` is first-class in `sections.yaml`. Collector `taxonomy.yaml` references `section_id:` directly. **`section_id:` is the canonical and ONLY accepted authoring form in v1.** `section_path:` as a list-of-segments is NOT accepted in v1 schema — closed schema rejects it (`additionalProperties: false`). Rationale: two authoring forms create a typo/ambiguity surface; one canonical form keeps `gen_taxonomy_seed.py` output deterministic and reviews easy. Output `taxonomy.json` carries BOTH stable `id` and a generator-resolved dotted `path` for FE convenience. If contributor demand emerges for path-style authoring later, a follow-up SOW must design that alternate input shape explicitly.

412 + - **ID semantics**: `section_id` is an opaque immutable handle. Dots in IDs are allowed for readability and namespace grouping, but they do not define parentage and are not recomputed when a section moves. `parent_id` is the only source of hierarchy. Example: moving `applications.postgres` under a new `applications.databases` parent edits the `parent_id` of the existing `applications.postgres` section; collector `taxonomy.yaml` remains unchanged. Renaming the ID to `applications.databases.postgres` would be a deprecation/replacement, not a move.

413 + - **`sections.yaml` shape**: each entry has `id` (stable, immutable, kebab-case), `parent_id` (root entries omit), `title`, `short_name?`, `icon?`, `section_order` (for top-level ordering), `status` (`active` | `deprecated`), `deprecation?: { replacement_id, since, removal_in }`.

414 + - **Move semantics**: a section moves by changing `parent_id` in `sections.yaml`. Collector `taxonomy.yaml` files referencing the moved section need NO edit because they reference `section_id`, not the path. The resolved `path` in `taxonomy.json` updates automatically.

415 + - **Deprecation semantics**: a section may be marked `status: deprecated` with `deprecation: { replacement_id, since, removal_in }`. Consumers warn; new collectors cannot place charts under deprecated sections (TAX028).

416 + - **Stable IDs anchor**: FE state persistence (saved view layouts), URL query params, and deprecation tracking all key off stable section IDs, not paths.

417 +9. **Delivery shape (CORRECTED 2026-05-11 by user decision)**: one implementation PR for the framework plus POC collector taxonomies. No PR-A1 / PR-A1.5 / PR-A2 split for this SOW. FE Phase B remains downstream and out of SOW-0016. Full collector coverage, production ibm.d sweep, legacy drift triage, and global all-collector fatality are follow-up work.

418 +10. **Categories axis** (`meta.monitored_instance.categories`) is orthogonal to TOC. Preserved untouched. Documented in skills + AGENTS.md so contributors do not conflate the two axes.

419 +11. **View-conditional rendering (AMENDED 2026-05-14; `only_views:` still dropped per user decision)**: sparse `single_node:` deltas and whole-body `view_switch` have separate roles. No whole-node visibility gate in v1.

420 + - **Top-level fields ARE the multi-node rendering** (the canonical/dominant case).

421 + - **`single_node:` is a sparse same-kind override block** — declared only when single-node view differs from multi-node by scalar/list/display/renderer field deltas on the same item type. It may not contain `type`, `items`, `multi_node`, `single_node`, or change an owner into a widget.

422 + - **`view_switch` is for whole-item replacement** — use it when branches have different item kinds, different child trees, or widget bodies where sparse override would be unclear. `view_switch.multi_node` and `view_switch.single_node` are both required and contain concrete items. `single_node:` and `view_switch` cannot appear on the same item.

423 + - **No `only_views:` in v1 (locked by user 2026-05-11)**: whole-node visibility gating is not part of the v1 schema. If audit 1.1 surfaces a real structure-level visibility-gate case (Scenario S2), that case becomes a **follow-up SOW** to add `only_views:` later — it does NOT block the initial framework+POC PR. The implementation PR ships without the field. Schema's permitted `x_*` extension namespace (Decision 13) does not back-door this; adding `only_views:` later requires a `taxonomy_schema_version` minor bump and the follow-up SOW's design review.

424 + - **Allowed override fields in `single_node:` (CLOSED set, v1)**: the set is derived from the same item type's field matrix in `.local/audits/taxonomy-design/full-shape-v1-redesign.md`. `include_charts`, `only_views`, `type`, and `items` are still NOT valid in `single_node:`. The schema closes `single_node:` properties with `additionalProperties: false` plus the permitted `x_*` extension namespace per Decision 13.

425 + - **List replacement examples (canonical, in spec doc)**:

426 + - Scalar override: `single_node: { title: "Average CPU" }` — replaces top-level `title` only for single-node view.

427 + - List replacement on a `type: context` widget: `contexts: [a, b, c]` + `single_node: { contexts: [a, b] }` → single-node renders only `a, b` (the top-level list does not extend).

428 + - Explicit clear: `group_by: [label:node]` at top + `single_node: { group_by: [] }` → single-node has no grouping.

429 + - **Lint rules (orthogonal, normalized after Codex review subagent 02)**:

430 + - **TAX021** — unknown override key under `single_node:` (closed enum violation).

431 + - **TAX022** — `multi_node:` override block declared (multi-node IS top level).

432 + - **TAX023** — list-merge attempt (educational error: lists replace, not merge; if extend is needed, use `*_extend:` field — not in v1).

433 + - **TAX024** — empty `single_node:` block (warning; equivalent to omitting it).

434 + - **TAX025** — `single_node:` override field equals top-level value (redundant override; warning).

435 + - ~~**TAX026**~~ — REMOVED: previously "dead override under `only_views: [multi_node]`"; no longer applicable without `only_views:`.

436 + - ~~**TAX027**~~ — REMOVED: previously "`only_views:` value not in closed enum"; no longer applicable.

437 + - **Why no per-node `condition:` everywhere (Variant A rejected)**: 2 of 3 reviewers ranked it last for maintenance; typos silently render in both views.

438 + - **Why no duplicate placements (Variant B rejected)**: invisible pair-link, copy-paste drift bait.

439 + - **Why no `variants:` block (Variant C rejected)**: critical ambiguity around "missing branch = base or = hidden".

440 + - **Why no pure handle-level conditioning (Variant E rejected)**: shifts source-of-truth into FE; title text (most common per-view difference) leaves the YAML.

441 + - **Why no `views:` wrapper**: YAGNI; `single_node:` handles simple same-kind deltas and `view_switch` handles whole-body replacement without hiding multi-node defaults inside a wrapper. Future view types require an explicit schema/version amendment.

442 + - The amended design keeps the maintainer's sparse-override preference for simple deltas while covering full FE body switches. Whole-section visibility gates are deliberately excluded from v1; any real S2 case from audit 1.1 becomes a follow-up SOW rather than implicit schema surface.

443 + - Reason for pinning at user request: cloud-frontend's `({ isSingleNode }) => ...` pattern produces view-dependent chart specs (per user "Different view depends on the view, we need this in the taxonomy"); user further clarified "multi node is the default and single node is the same, we need to support override syntax".

444 + - Reviewer reports: `.local/audits/taxonomy-design/schema-review-{1,2,3}.md`; external review: `.local/audits/taxonomy-design/external-review-codex/`.

445 +

446 +12. **Deterministic merge / order rules (NEW 2026-05-11 per Change 8)**. With ~150 separate YAML files, file-system traversal order, YAML author order, and Python dict insertion order must NOT be correctness inputs. The locked merge algorithm:

447 + - **Top-level section order**: `sections.yaml` declares `section_order` field per top-level entry; sort ascending. Frozen v1 ordering: `system, kubernetes, containers-vms, synthetic-checks, remote-devices, otel, azure-monitor, applications, netdata` (mirrors current FE).

448 + - **Parent/leaf ordering**: per parent, children sort by `priority` ASC (lower = earlier; default 1000), then by normalized title (`unicodedata.normalize("NFC", title).casefold()`, Python default binary string ordering; no locale collation), then by `placement_id` (lex), then by source path (lex) as final tiebreaker.

449 + - **Explicit item ordering**: `items:` arrays preserve author order at every depth. The deterministic sort applies only when merging independently-authored placement/section siblings under the same parent. The generator must not recursively sort author-provided item trees.

450 + - **Selector and reference ordering**: structural `items:` arrays and widget `contexts:` arrays preserve author order. `context_prefix:` and `collect_plugin:` selector lists are sorted lex on emit so the JSON snapshot is stable across machines.

451 + - **Duplicate ownership policy**: a leaf `(section_id, leaf_id)` has EXACTLY ONE owner (TAX006 fatal). Implicit multi-owner merge is forbidden. If two collectors must contribute to a shared parent (e.g. multiple databases under `applications.databases`), they own distinct leaves under it; the parent metadata comes from `sections.yaml` (or a single explicit `section_overrides:` per audit-resolved policy).

452 + - **Finding emission cadence**: TAX033 and TAX036 emit once per conflicting context per unordered owner pair, sorted by context, owner key, and source path. Multiple selector mechanisms for the same pair are folded into one finding message. TAX037 emits once per referenced-only literal context per nearest item path. TAX038 emits once per stale unresolved reference per item path.

453 + - **JSON serialization**: `gen_taxonomy.py` emits `taxonomy.json` with sorted object keys, fixed indentation, no trailing whitespace. Re-running the generator on identical input produces byte-identical output (golden test required).

454 + - Reason: avoids noisy diffs in POC and later full-migration PRs; eliminates a class of CI flakiness; makes later legacy-vs-generated diff tooling reliable.

455 +

456 +13. **Schema evolution posture (NEW 2026-05-11 per Change 15)**. Permissive `additionalProperties: true` is replaced by **closed core schemas + namespaced extension keys**.

457 + - **Closed core**: every taxonomy authoring object (placement, item, `single_node:` block, `sections.yaml` entry, opt-out object) declares `additionalProperties: false`. Unknown core keys fail TAX021/TAX028 (depending on context). This catches the typo class that already exists in cloud-frontend (`applications.js:53` has `icons:` instead of `icon:`; `systemHardware.js:69-70` has duplicate `title` keys).

458 + - **Renderer envelope**: FE-private renderer payloads may be carried only under a fenced `renderer:` object. Open pass-through fields directly on item bodies are rejected so typo detection remains meaningful.

459 + - **Namespaced extension keys**: `x_*` is the only permitted extension namespace on core nodes. Extensions are preserved into `taxonomy.json` under an `_extra` block, scoped to the originating placement/subsection. Extensions never alter rendering until a schema version claims them.

460 + - **Breaking-change boundary** (must be documented in `taxonomy.md` spec):

461 + - **Non-breaking**: adding optional fields with defined defaults; adding new `sections.yaml` entries; adding deprecation metadata.

462 + - **Non-breaking only if old consumers ignore them safely**: adding a new view type (closed enum extension); adding a new selector type; adding a new override field under `single_node:`. All of these require a `taxonomy_schema_version` minor bump even if the change is forward-compatible at the data level.

463 + - **Breaking (require major schema version bump)**: changing matcher semantics, override merge semantics, list-replacement behavior, section ID/path identity, generated JSON field renames, or removing/renaming existing output fields.

464 + - **Persisted shorthands forbidden**: no `contexts: all_from_metadata`. The seed tool (Decision 9, active implementation plan) generates explicit lists at author time; subsequent metric additions to `metadata.yaml` MUST be reflected in a taxonomy diff or coverage check fails.

465 + - Reason: open schemas preserve typos; the SOW's drift-elimination goal requires loud failure on misspelled fields, not silent acceptance.

466 +

467 +14. **Chart-recipe manifest remains removed; explicit item bodies replace it (UPDATED 2026-05-14)**: previously proposed as `integrations/taxonomy/chart_recipes.yaml` to validate `include_charts:` handle references. v1 still has no `include_charts:` field and no chart-recipe manifest. The reopened full-shape design models first-available alternatives, grids, context/table widgets, and view-conditioned item bodies directly inside ordered `items:` rather than through recipe handles.

468 +

469 +## Plan

470 +

471 +1. **Single framework+POC PR (active scope)** — narrow `_common.py` extract with byte-identical proof; new schemas with closed-core posture; `sections.yaml` with stable IDs; `icons.yaml`; `gen_taxonomy.py` with deterministic merge and structured `Finding` renderers; `gen_taxonomy_seed.py`; fresh `check_collector_taxonomy.py`; a small POC collector set; CI wired with changed-collector fatal gate; documentation, specs, and skills updated.

472 +2. **Inline audit evidence** — audit outputs under `.local/audits/taxonomy-design/audit/` are produced before the implementation step that depends on them. Audit 1.10 is a non-blocking FE coordination note. Audit 1.5 is only required if production ibm.d codegen or an ibm.d POC is included.

473 +3. **Review checkpoints** — after each major implementation step, decide whether a Claude review is useful. If yes, provide a focused prompt with exact files and questions.

474 +4. **Follow-up work (not this PR)** — full collector taxonomy coverage, production ibm.d sweep, legacy drift triage, global-all-collector fatality, and cloud-frontend consumption.

475 +

476 +Total for this SOW is now the single Netdata framework+POC PR. SOW-0016 closes when that PR merges green and the validation/artifact gates are filled.

477 +

478 +**Cloud-frontend Phase B is OUT of scope of SOW-0016** (FE team owns it on their schedule; tracked as a downstream FE-team SOW). The Netdata-side deliverables are: published `taxonomy.json` artifact shape, `taxonomy_output.json` schema, generator/checker/seed framework, changed-collector CI gate, docs/spec/skills, and POC taxonomy files.

479 +

480 +## Execution Log

481 +

482 +### 2026-05-10

483 +

484 +- 3 independent Opus 4.7 analysis agents launched in parallel; architecture reports stored at `.local/audits/taxonomy-design/agent-{1,2,3}-analysis.md`.

485 +- Synthesized recommendation produced; user reviewed and locked decisions 1–10 plus added the view-condition requirement (decision 11).

486 +- TODO file `TODO-collector-taxonomy-unification.md` updated to reference this SOW.

487 +- This SOW created and Pre-Implementation Gate filled.

488 +- User flagged the initial per-node `condition:` schema design (Variant A) as a UX risk; 3 independent reviewers ran in parallel as taxonomy-author personas (Marta/Pavel/Lina) across 5 schema variants and 3 scenarios. Reports at `.local/audits/taxonomy-design/schema-review-{1,2,3}.md`.

489 +- Schema-review synthesis: hybrid `only_views:` + `views:` was initially chosen.

490 +- User refined #1 (YAGNI): drop `only_views:` from v1; ship pure Variant D (`views:` overrides only). Schema's `additionalProperties: true` posture allows non-breaking addition of `only_views:` later if pre-audit step 1.1 finds an S2 case.

491 +- User refined #2 (avoid duplication): pure D's symmetric `views: { single_node, multi_node }` requires writing field defaults somewhere awkward. User feedback: "multi node is the default and single node is the same, we need to support override syntax — take multi and override some stuff". Schema refined to curated-and-override: top-level fields ARE multi-node; `single_node:` block holds the sparse delta. No `views:` wrapper, no `multi_node:` block. Decision 11 updated. Validator codes: TAX021 (unknown view-override key), TAX022 (`multi_node:` block illegal — fields go at top level), TAX023 (list-merge attempt), TAX024 (empty `single_node:` block).

492 +- User refined #3 (dynamic contexts): SNMP, prometheus, cgroup, apps emit contexts whose names share a namespace prefix. Decision 2 updated to allow `context_prefix:` (string prefix only, NOT regex) with an opt-in guardrail: collector must declare `metrics.dynamic_context_prefixes: [...]` in its `metadata.yaml`. Adds TAX031 (prefix not declared), TAX033/TAX036 ownership-overlap checks, and TAX034 (redundant explicit context under prefix). TAX032 remains a reserved follow-up code for a narrower prefix-overlap diagnostic if the project later needs one.

493 +- Empirical validation (7 go.d collectors against locked schema, report at `.local/audits/taxonomy-design/schema-validation-drafts.md`): 6/7 pass cleanly. statsd exposed a real gap — its user-app synthetic charts use user-supplied names (`name = myapp` → `myapp.*` chart names) with no shared prefix. Same shape applies to `charts.d.plugin` (bash scripts pick their own names) and `python.d.plugin` (legacy). User confirmed adding `collect_plugin:` selector to v1 (Decision A on 2026-05-10).

494 +- User refined #4 (label selector): added `collect_plugin: [<plugin-name>]` selector parallel to `context_prefix:`. Selects any chart whose `_collect_plugin` label matches. Same opt-in pattern: collector declares `metrics.dynamic_collect_plugins: [{plugin, reason}, ...]` in `metadata.yaml` (or in `taxonomy.yaml` for plugins lacking `metadata.yaml` like statsd.plugin). Adds TAX035 (collect_plugin not declared), TAX036 (overlap between collectors).

495 +- Validation also surfaced a real-world drift: `mysql.open_transactions` referenced in `cloud-frontend/.../applications.js:1957` does NOT exist in `mysql/metadata.yaml`. This is exactly the failure TAX003 catches at PR time. PR-A2 plan extended: run a diff-tool sweep over the legacy taxonomy to inventory similar drifts before bulk migration.

496 +- Other empirical findings (some superseded by later decisions): non-leaf sections must live in `sections.yaml`; exactly one collector may use `section_overrides:` for any given `section_id`; empty `contexts:` is valid when `context_prefix:` or `collect_plugin:` is present; postgres-style 70-context enumeration requires `gen_taxonomy_seed.py` to amortize author cost. The earlier `section_path:` and `include_charts:` draft shapes are explicitly superseded by Decisions 5 and 8.A.

497 +- Schema impact: `integrations/schemas/collector.json` extended with two optional dynamic declarations. This is the first material change to the existing collector schema in this SOW.

498 +

499 +### 2026-05-11

500 +

501 +- External Codex review run (6 parallel GPT-5.5-xhigh subagents per the prompt at `/tmp/codex-taxonomy-review-prompt.md`); reports stored at `.local/audits/taxonomy-design/external-review-codex/`. Synthesis verdict: GO WITH CHANGES. Architecture approved; SOW not ready for PR-A1 kickoff until 15 contract-level gaps closed.

502 +- SOW amended this date to incorporate all 15 required changes:

503 + - **Change 1 (audit 1.4 false premise)**: rewrote step 1.4 from "confirm none" to a `virtualContexts` classification table with per-row owner. The initial disposition name `frontend-recipe-handle` was later normalized to `keep-frontend-only` after Decision 14 was removed. `systemStorage.js:3-31, 103` cited as the disproof of the previous premise.

504 + - **Change 2 (chart handle contract)**: added Decision 14 — `integrations/taxonomy/chart_recipes.yaml` manifest with stable handle IDs, consumed_contexts, ordered alternatives (for `getMenu.js:77-82` first-available semantics), supported_views, owner, status, deprecation. Validators TAX040–TAX042 added.

505 + - **Change 3 (Netdata negative selector)**: amended Decision 2 with `context_prefix_exclude:` constrained-exclusion field; audit 1.8 chooses static enumeration vs prefix+exclude; regex remains forbidden.

506 + - **Change 4 (selector semantics)**: amended Decision 2 with explicit union behavior, cross-type overlap detection, resolved-snapshot-vs-runtime contract, no-metadata collector inline declarations, FE label-access proof requirement (audit 1.10).

507 + - **Change 5 (taxonomy.json lifecycle)**: amended Decision 3 — gitignored ephemeral in PR-A1; vendored build-pinned in Phase B; `taxonomy_schema_version`, source commit metadata, deprecation fields, FE build-time validation.

508 + - **Change 6 (stable section identity)**: amended Decision 8 — originally proposed path 8.A (stable `section_id`) vs 8.B (immutable path segments); later user sign-off locked 8.A and removed `section_path:` as an accepted v1 authoring form.

509 + - **Change 7 (view-conditional hardening)**: amended Decision 11 — originally reconsidered `only_views:`; later user sign-off dropped it from v1. Closed allowed-override-fields set remains; canonical examples are no override, scalar override, and list replacement; lint codes TAX026/TAX027 are removed.

510 + - **Change 8 (deterministic merge/order)**: added Decision 12 — locked sort key `(section_order, priority, normalized_title, placement_id, src_path)`; deterministic JSON serialization; one-owner-per-leaf rule; 10-run byte-identical golden test.

511 + - **Change 9 (seed tooling)**: promoted `gen_taxonomy_seed.py` to PR-A1 hard requirement; persisted `contexts: all_from_metadata` shorthand explicitly forbidden (Decision 13).

512 + - **Change 10 (fresh checker)**: explicit "do not mirror `check_collector_metadata.py`" guidance; `check_collector_taxonomy.py` is a fresh wrapper around `_common.py` and taxonomy validators.

513 + - **Change 11 (PR-A1.5 blocking)**: PR-A1.5 elevated from "may merge as part of A1" to a SEPARATE BLOCKING PR between PR-A1 and PR-A2; audit 1.5 produces a working `db2` prototype with round-trip evidence; escalation path defined if prototype fails in <1 week.

514 + - **Change 12 (changed-collector fatal gate)**: from PR-A1 onward, taxonomy coverage is fatal for changed `metadata.yaml`/`taxonomy.yaml` files; global-all-collector coverage stays warning until PR-A2 final.

515 + - **Change 13 (drift triage process)**: PR-A2 step 4.6 introduces a drift inventory with closed disposition set `{drop-frontend-entry, fix-metadata, fix-collector, keep-frontend-only, defer-with-sow}` and per-finding owner; bulk migration implementer cannot silently decide product semantics.

516 + - **Change 14 (Phase B realism)**: original wording held Phase B estimate at 2 weeks; reviewer pass updated to 6–8 weeks and required FE rollback/design artifacts. **2026-05-11 scope correction superseded all of this**: Phase B is OUT of SOW-0016; FE-team owns it. Those FE artifacts are downstream FE-SOW responsibilities. SOW-0016 closes when the single framework+POC PR lands with green validation.

517 + - **Change 15 (schema evolution)**: added Decision 13 — closed core schemas (`additionalProperties: false`); only `x_*` namespaced extension keys allowed at core nodes; explicit breaking-change boundary documentation.

518 +- Pre-Implementation Audit Outputs section was added during hardening and later normalized to 10 required audit deliverables + 1 non-blocking coordination note after audit 1.7 and 1.9 were removed; PR-A1 is blocked until all required outputs are non-`to-decide`.

519 +- Drift Triage Process and Rollback Matrix subsections added.

520 +- Open decisions section temporarily rewritten with 5 user-facing residual decisions (Phase B fallback shape; artifact lifecycle path; section identity model 8.A vs 8.B; `only_views:` v1 inclusion default; drift triage owner-assignment policy). Later same-day scope correction and user sign-off resolved all of them.

521 +- Acceptance criteria expanded from 9 items to 27 items reflecting the new gates.

522 +- Plan total was temporarily revised from "~6–8 weeks" to "15–19 weeks calendar" when Phase B was still included; later same-day scope correction narrowed SOW-0016 back to Netdata-only.

523 +- Sub-state updated to "design hardened after external Codex review (GO WITH CHANGES); pre-implementation audit blocking; PR-A1 cannot start until audit produces written evidence."

524 +- **2026-05-11 scope correction (later same day)**: user clarified two points:

525 + - (a) "we don't need to change FE - the FE guys will do it. We just need to prepare everything in Netdata repo." Phase B is moved OUT of scope of SOW-0016; cloud-frontend consumption is tracked as a downstream FE-team SOW on their schedule.

526 + - (b) The previously-recommended "build-pinned vendored" artifact lifecycle was over-engineered relative to the existing `integrations.js` precedent. Decision 3 corrected: gitignored ephemeral generation, consumed by FE-team's pipeline exactly as `integrations.js` is consumed today; contract-discipline lives in `taxonomy_schema_version` IN the artifact, not in the delivery mechanism.

527 +- Consequent SOW edits made in the same session:

528 + - Decision 3 rewritten to match `integrations.js` precedent.

529 + - Historical note, superseded 2026-05-14: Decision 5 was then corrected to keep chart bodies FE-side. The later 2026-05-14 user decision reopens this boundary and requires explicit full-shape typed `items:` in v1.

530 + - Decision 14 REMOVED from v1: chart-recipe manifest unnecessary with `include_charts:` removed. This remains true after the 2026-05-14 reopening because explicit typed item bodies replace recipe handles.

531 + - Phase B Plan section replaced with a one-paragraph "OUT OF SCOPE" reference to the downstream FE-team SOW.

532 + - Audit 1.7 (FE adapter spike) and 1.9 (chart-recipe manifest seed) REMOVED. Audit 1.10 (`_collect_plugin` feasibility) downgraded from blocking to a non-blocking coordination question sent to FE team.

533 + - Acceptance criteria pruned of FE-side gates (vendoring proof, rollback runbook, staging fixture, FE adapter spike, chart-recipe handle validation).

534 + - Rollback Matrix Phase B row removed; matrix simplified to Netdata-only (PR-A1, PR-A1.5, PR-A2).

535 + - Plan total revised from "15–19 weeks" to "7–9 weeks calendar of Netdata-side work" honestly reflecting the narrower scope.

536 + - Open Decisions reduced from 5 user items to 3 (Phase B fallback shape and artifact lifecycle path are now resolved by the scope correction itself).

537 + - Followup mapping updated: Phase B becomes a downstream FE-team SOW. Historical chart-recipe-handle follow-up was superseded on 2026-05-14 by explicit full-shape typed item bodies.

538 +- **2026-05-11 residual-decision sign-off (final design lock)**:

539 + - Decision 1 (section identity model) → **8.A locked**. Stable `section_id` first-class in `sections.yaml`; collector taxonomy references `section_id:`; section moves are `parent_id` edits.

540 + - Decision 2 (`only_views:` v1 inclusion) → **dropped from v1** per user ("I don't think we need only_views"). If audit 1.1 finds an S2 case, that case spawns a follow-up SOW (non-blocking for PR-A1). Decision 11 amended; canonical examples reduced from 4 to 3; lint codes TAX026 and TAX027 removed; `single_node:` allowed-fields set updated accordingly.

541 + - Decision 3 (drift triage owner assignment) → **A locked**. Bulk-migration implementer assigns owners based on finding type; named owners sign off or push back.

542 + - Sub-state updated: "all design decisions locked 2026-05-11; SOW ready to move from `pending/` to `current/` on user go-ahead". Open Decisions section now records all 3 resolutions for traceability.

543 + - No further user decisions block PR-A1 start. Only the audit remains.

544 +- **2026-05-11 final readiness review (3 parallel Opus 4.7 reviewers; all returned READY WITH NOTES)**:

545 + - Reports: `.local/audits/taxonomy-design/final-review/agent-{1,2,3}-readiness.md`.

546 + - 9 patches applied to close all surfaced gaps:

547 + - **Patch 1 (Critical, R3)**: Step 2.11 CI wiring corrected — `generate-integrations.yml` runs on `push: master` (post-merge); `check-markdown.yml` is the PR-time gate. Taxonomy validation must run in BOTH workflows; "fatal in CI" gates explicitly live in `check-markdown.yml`. Path triggers extended on both workflows.

548 + - **Patch 2 (High, R1+R3)**: Decision 8.A clarified — `section_id:` is canonical and only accepted authoring form in v1. `section_path: [list]` is NOT accepted in v1 schema. Eliminates the typo/ambiguity surface.

549 + - **Patch 3 (High, R1+R2)**: Step 2.12 "touched-collector" gate definition tightened — fatal only when diff modifies `metrics.*` keys OR `taxonomy.yaml` OR adds/removes either file. Edits to `overview`, `setup`, `troubleshooting`, `alerts`, `related_resources` do NOT trigger the gate. Prevents PR-A1→PR-A2 window from blocking unrelated metadata edits.

550 + - **Patch 4 (High, R2)**: Audit 1.11.b added — full `sections.yaml` tree draft (~80–150 entries) walking every cloud-frontend taxonomy file. Without this, PR-A1 step 2.4 was a developer guessing the topology from 18 JS files for a day.

551 + - **Patch 5 (High, R1)**: Banner added to `schema-validation-drafts.md` warning it is superseded for authoring (25+ stale `include_charts:` uses; ~20 stale `section_path:` uses). First developer reading it as template no longer gets the wrong shape.

552 + - **Patch 6 (Medium, R2)**: Step 2.2 byte-identical proof phrasing fixed — `git diff --exit-code` does NOT work on gitignored ephemeral files; correct procedure uses `diff -u` against captured pre-refactor copies stored under `.local/audits/taxonomy-design/refactor-baseline/`.

553 + - **Patch 7 (Medium, R1)**: Drift Triage Process — 5-business-day escalation clock added; if a named owner doesn't sign off within 5 business days, implementer escalates to user for tie-break. Prevents PR-A2 final stalling on PTO.

554 + - **Patch 8 (Medium, R2+R3)**: Stale residue swept — Step 2.14 "chart-recipe-alternatives" fixture replaced with a Netdata-negative-selector fixture (Decision 14 was removed; old fixture name was leftover); `keep-frontend-virtual` spelling normalized to `keep-frontend-only`; `getMenu.js` path corrected from `taxonomy/getMenu.js` to `charts/toc/getMenu.js`; stale Phase-B-coupling references in Risks and Execution Log replaced with downstream-FE-SOW language.

555 + - **Patch 9 (Medium, R3)**: Decision 2 shapes locked — `inline_dynamic_declarations:` block shape and `resolved_contexts` snapshot shape both written into the SOW with concrete YAML/JSON examples. No remaining schema TBDs before PR-A1 step 2.3.

556 + - Total audit outputs now 10 (was 9): added `1.11b-sections-tree.md`. Total reviewer-flagged ambiguities resolved: all surfaced concerns either patched in SOW or explicitly assigned to in-flight PR-A1 work.

557 + - SOW is ready to move from `pending/` to `current/`. Audit phase begins on the next user action.

558 +- **2026-05-11 external Codex post-patch readiness recheck (4 parallel subagents; synthesis verdict NOT READY as artifact bundle, no architecture blocker)**:

559 + - Reports stored under `.local/audits/taxonomy-design/external-review-codex-2/`.

560 + - Findings: 5 of 9 patches landed cleanly; 4 partially landed due stale text in acceptance criteria, plan, validation, TODO, and walkthrough.

561 + - Sweep applied: normalized audit count to 10 required outputs + 1 non-blocking coordination note; removed active `section_path` authoring language; removed active `include_charts` / chart-recipe / Phase-B validation residue from Netdata execution gates; replaced stale `git diff --exit-code` proof wording for gitignored artifacts with the `diff -u` baseline-copy procedure; locked PR-A1 clarifications for `taxonomy_optout`, deterministic title normalization, seed output, audit 1.11.b dependency, and opaque stable section-ID semantics.

562 + - Targeted stale-reference grep validated the sweep; SOW is ready to move from `pending/` to `current/` on user go-ahead.

563 +- **2026-05-11 audit phase start**:

564 + - User approved moving SOW-0016 from `pending/` to `current/` and starting the audit output phase.

565 + - Status changed from `open` to `in-progress`.

566 + - Audit output directory initialized at `.local/audits/taxonomy-design/audit/` with index `00-audit-index.md`.

567 +- **2026-05-11 one-PR execution correction**:

568 + - User decided: "we will do everything in one PR (framework - w/o adding taxonomy for all collectors, can add a few as a POC)."

569 + - Branch created and checked out: `sow-0016-taxonomy-framework-poc`.

570 + - Active SOW scope corrected: single framework+POC PR; no PR-A1 / PR-A1.5 / PR-A2 split for this SOW.

571 + - Full collector coverage, production ibm.d sweep, legacy drift triage, and global all-collector fatality moved to follow-up work unless explicitly pulled into this PR.

572 + - User requested review checkpoints: after each major step, provide a Claude prompt if an external review would be useful.

573 +- **2026-05-11 framework+POC implementation pass**:

574 + - `_common.py` extracted from `gen_integrations.py`; byte-identical `integrations.{json,js}` proof completed before later schema/metadata changes.

575 + - Taxonomy schemas, section/icon registries, generator, checker, seed helper, and unittest coverage added.

576 + - CI wired in both PR-time `check-markdown.yml` and post-merge `generate-integrations.yml`.

577 + - POC collector taxonomies added for apache, mysql, postgres, nvidia_smi, and snmp.

578 + - SNMP metadata extended with `metrics.dynamic_context_prefixes` for `snmp.`.

579 + - Contributor docs, project skills, AGENTS.md, and taxonomy spec updated.

580 + - Local `.venv` validation passed; details recorded in the Validation section.

581 +- **2026-05-11 Claude review follow-up**:

582 + - User provided external review verdict READY WITH NOTES.

583 + - Fixed accepted pre-merge items F1/F2/F4/F6/F7/F8/F12: no-metadata `taxonomy_optout` no longer emits a misleading missing-metadata fatal; metadata files whose metrics block was removed are treated as touched; 10-run determinism unittest added; seed/checker docs added to README, pipeline, and add-go-collector recipe; title normalization aligned to NFC.

584 + - Deferred per user/review scope: YAML-aware metrics span parsing, broader TAX negative-test matrix, global gate severity policy, invalid-metadata surfacing, schema-error message-code mapping hardening, artifacts-and-banners taxonomy entry, and opt-out POC example.

585 +- **2026-05-14 full-shape redesign pause**:

586 + - User rejected structural-only POC depth and required v1 to cover full Cloud FE TOC shapes.

587 + - Implementation paused; no further code changes until the amended full-shape contract is reviewed.

588 + - Design artifact updated at `.local/audits/taxonomy-design/full-shape-v1-redesign.md`.

589 + - Claude review prompt updated at `.local/audits/taxonomy-design/full-shape-v1-adversarial-review-prompt.md`.

590 +- **2026-05-14 full-shape implementation resume**:

591 + - External review returned READY TO IMPLEMENT after B1-B7 plus R1/R2 amendments.

592 + - `taxonomy_collector.json`, `taxonomy_output.json`, `gen_taxonomy.py`, `gen_taxonomy_seed.py`, `test_taxonomy.py`, docs/spec/skills, and all five POC `taxonomy.yaml` files were updated to the ordered recursive `items:` contract.

593 + - MySQL POC now models summary grid widgets, table widgets, nested structural groups, owned context leaves, `referenced_contexts`, and legacy FE drift correction for `mysql.galera_open_transactions`.

594 + - Local `.venv` validation passed for py_compile, generator check-only, touched-collector checker, seed helper, determinism diff, performance spot check, and 18 taxonomy unit tests.

595 +

596 +## Validation

597 +

598 +Acceptance criteria evidence:

599 +

600 +- Implemented locally:

601 + - `integrations/_common.py` extracted and `integrations/gen_integrations.py` refactored to reuse it.

602 + - `integrations/schemas/taxonomy_collector.json`, `taxonomy_sections.json`, and `taxonomy_output.json` added with closed-core v1 authoring; on 2026-05-14 `taxonomy_collector.json` and `taxonomy_output.json` were updated from the structural-only POC shape to the full ordered recursive `items:` shape.

603 + - `integrations/taxonomy/sections.yaml` and `icons.yaml` added for the POC section/icon registry.

604 + - `integrations/gen_taxonomy.py`, `integrations/gen_taxonomy_seed.py`, and `integrations/check_collector_taxonomy.py` added. On 2026-05-14, `gen_taxonomy.py` was updated to emit local `resolved_contexts`, `referenced_contexts`, and `unresolved_references` snapshots for every placement/item, enforce TAX037 referenced-but-not-owned, preserve TAX036 for selector ownership overlap, deduplicate/sort TAX033/TAX036 conflict emission, and support `owned_context`, `group`, `flatten`, `selector`, `context`, `grid`, `first_available`, and `view_switch` item kinds.

605 + - POC `taxonomy.yaml` files added for apache, mysql, postgres, nvidia_smi, and snmp. On 2026-05-14, all five were migrated to `items:`; MySQL became the full-shape proof with summary grid, table widgets, nested groups, owned structural context leaves, and legacy drift correction from `mysql.open_transactions` to `mysql.galera_open_transactions`.

606 + - `snmp/metadata.yaml` declares `metrics.dynamic_context_prefixes: [{prefix: snmp., reason: ...}]` for the SNMP dynamic-prefix POC.

607 + - CI wiring added to `check-markdown.yml` and `generate-integrations.yml`.

608 + - `integrations/taxonomy.json` added to `.gitignore`.

609 +

610 +Tests or equivalent validation:

611 +

612 +- Passing locally with repo-local `.venv`:

613 + - `.venv/bin/python -m py_compile integrations/_common.py integrations/gen_integrations.py integrations/gen_taxonomy.py integrations/gen_taxonomy_seed.py integrations/check_collector_taxonomy.py integrations/tests/test_taxonomy.py`

614 + - `.venv/bin/python integrations/gen_integrations.py`

615 + - `.venv/bin/python integrations/gen_taxonomy.py --check-only`

616 + - `.venv/bin/python integrations/check_collector_taxonomy.py`

617 + - `.venv/bin/python -m unittest integrations.tests.test_taxonomy` (40 tests, including old-shape rejection, recursion-matrix rejection, renderer-envelope rejection, positive coverage for item kinds, TAX003 unknown-context fatal, TAX036 selector-overlap preservation, TAX037 referenced-but-not-owned enforcement, TAX038 stale-unresolved warning, unresolved payload output, dynamic-prefix narrowing, metadata-warning surfacing, YAML-aware touched-collector span parsing, deleted-collector gate handling, and 10-run deterministic taxonomy output check)

618 + - `.venv/bin/python integrations/gen_taxonomy_seed.py src/go/plugin/go.d/collector/apache/metadata.yaml --module-name apache --section-id applications.apache --placement-id apache --icon apache` (emits flat `items:`)

619 + - Determinism proof: two consecutive `gen_taxonomy.py --output /private/tmp/netdata-taxonomy-{1,2}.json` runs compared cleanly with `diff -u`.

620 + - 2026-05-14 full-shape determinism proof: two consecutive `gen_taxonomy.py --output /private/tmp/netdata-taxonomy-fullshape-{1,2}.json` runs compared cleanly with `diff -u`.

621 + - Performance spot check after Claude-blocker fixes: `/usr/bin/time -p .venv/bin/python integrations/gen_taxonomy.py --check-only` completed in `real 3.99` seconds.

622 +

623 +Real-use evidence:

624 +

625 +- `integrations/gen_taxonomy.py` emitted a valid local full-shape taxonomy artifact containing the five POC placements. The generated `section_path` values are `applications.apache`, `applications.mysql`, `applications.postgres`, `system.hardware.gpus.nvidia`, and `remote-devices.snmp`. The artifact is gitignored and not intended for commit.

626 +- MySQL POC coverage proof from `/private/tmp/netdata-taxonomy-fullshape-1.json`: MySQL placement has `families: null`, 75 `resolved_contexts`, 16 `referenced_contexts`, 0 `unresolved_references`, a summary `grid` first item with 8 widget references, and no missing or extra owned contexts compared with `mysql/metadata.yaml` (75 metadata contexts, 75 owned).

627 +- MySQL intentionally owns `mysql.handlers` once in the `Handlers` structural group even though the legacy FE listed it in more than one visual grouping; the v1 contract requires single ownership and display widgets can reference owned contexts separately.

628 +- NVIDIA intentionally adds structural Bus / Utilization / Memory / Sensors / MIG grouping around the legacy FE table coverage. This is a Netdata-side taxonomy improvement, not an accidental FE parity miss.

629 +- `integrations/gen_docs_integrations.py -c go.d.plugin/snmp` produced no committed doc drift after adding the SNMP dynamic declaration, confirming the metadata extension does not alter generated user docs.

630 +

631 +Reviewer findings:

632 +

633 +- External readiness reviews are complete before implementation.

634 +- Post-implementation Claude review returned READY WITH NOTES and identified accepted fixes F1/F2/F4/F6/F7/F8/F12; all seven were applied.

635 +- Claude re-review returned READY: all seven accepted fixes are applied, tested, and cross-referenced in SOW/TODO/spec/recipe. Deferred caveats F3/F5/F9/F10/F11/F13/F14 remain non-blocking by design.

636 +- 2026-05-14 full-shape contract review returned READY TO IMPLEMENT after B1-B7 and R1/R2 amendments. The local implementation now targets that full-shape contract.

637 +- 2026-05-14 full-shape implementation review returned NOT READY with three blockers: MB1 unresolved payload dropped, MB2 MySQL `families: true`, MB3 missing TAX003 negative test. All three are fixed. Same-PR improvements also landed for TAX033/TAX036 dedupe/sort, recursion-matrix tests, renderer-envelope tests, item-kind positive tests, and schema-reference matrix depth.

638 +- 2026-05-14 post-blocker Claude re-review returned READY. It confirmed MB1/MB2/MB3 closure and same-PR fixes for TAX033/TAX036 cadence, recursion-matrix tests, item-kind positive tests, renderer-envelope tests, and schema-reference depth. Remaining polish is non-blocking: generic TAX001 stale-shape diagnostics and no artificial `view_switch`/`first_available`/`flatten`/renderer examples in POC YAMLs.

639 +

640 +Same-failure scan:

641 +

642 +- Targeted stale-shape scan completed during implementation via schema/unit coverage:

643 + - `section_path` authoring is rejected by `taxonomy_collector.json`.

644 + - `multi_node:` is rejected by TAX022 prescan.

645 + - `context_prefix_exclude:` without same-node `context_prefix:` raises TAX029.

646 + - top-level/placement `contexts:` authoring is rejected; POC collectors now use `items:`.

647 + - strings inside `grid.items` are rejected by schema; display positions require object items.

648 + - forbidden recursion-matrix cases are covered by unit tests: owning items in grid bodies, nested flatten, string first-available alternatives, string/flatten/nested-view-switch branches.

649 + - renderer pass-through is covered by unit tests: unknown non-`x_*` renderer keys and item-body renderer fields are rejected; `x_*` inside `renderer` is accepted.

650 + - Existing stale references to the old five-file shorthand were updated in integration lifecycle docs/recipes except the historical note that the shorthand is stale.

651 +

652 +Sensitive data gate:

653 +

654 +- Confirmed: this SOW, the linked TODO, the 3 agent reports, and all anticipated artifacts contain no raw secrets, credentials, bearer tokens, SNMP communities, customer data, personal data, customer-identifying IPs, private endpoints, or proprietary incident details. The work is structural metadata only. No redaction required.

655 +

656 +Artifact maintenance gate:

657 +

658 +- AGENTS.md: updated — collector-consistency rule now includes `taxonomy.yaml`; integrations-lifecycle trigger includes taxonomy files/artifacts.

659 +- Runtime project skills: updated — `integrations-lifecycle/` and `project-writing-collectors/` now document taxonomy authoring, generator/checker flow, artifact contract, and consistency impact.

660 +- Specs: updated — new `.agents/sow/specs/taxonomy.md` records source files, authoring contract, selectors, output artifact, CI contract, finding-code matrix, and contributor rule.

661 +- End-user/operator docs: updated — `integrations/README.md` documents `gen_taxonomy.py --check-only`, dynamic selector opt-ins, and the gitignored taxonomy artifact.

662 +- End-user/operator skills: no impact expected (skills do not consume the TOC).

663 +- SOW lifecycle: open → in-progress when moved to `current/` → completed after the single framework+POC PR validates and lands. Final move to done/ together with the work commit per project rule "Do not create a separate commit just to mark or move the SOW".

664 +

665 +Specs update:

666 +

667 +- Complete locally — `.agents/sow/specs/taxonomy.md`.

668 +

669 +Project skills update:

670 +

671 +- Complete locally — `integrations-lifecycle/`, `project-writing-collectors/`.

672 +

673 +End-user/operator docs update:

674 +

675 +- Complete locally — `integrations/README.md`.

676 +

677 +End-user/operator skills update:

678 +

679 +- No impact expected.

680 +

681 +Lessons:

682 +

683 +- Captured in `## Lessons Extracted`.

684 +

685 +Follow-up mapping:

686 +

687 +- Updated. Anticipated follow-ups:

688 + - **Claude review deferred items (not blocking this POC PR)**: broader TAX negative-test matrix; global gate severity policy as taxonomy coverage grows; schema-error message-code mapping hardening; opt-out POC example.

689 + - **Audit evidence (in-scope this SOW)**: 10 supporting outputs under `.local/audits/taxonomy-design/audit/` plus 1 non-blocking FE coordination note; consumed inline before dependent implementation steps.

690 + - **Cloud-frontend Phase B SOW (downstream, FE-team-owned, separate private-repo SOW)**: FE team consumes the published `taxonomy.json` on their schedule. They own: consumer module, taxonomy module removal, chart-spec extraction, `dynamicSections` removal, regex-catchall removal, rollback strategy. Out of scope of SOW-0016.

691 + - **Full collector taxonomy migration**: follow-up SOW/PR plan for the remaining collectors, global all-collector fatality, and legacy drift inventory.

692 + - **Drift triage follow-ups**: any `defer-with-sow` disposition from the future full-migration drift inventory spawns a new SOW with the named owner before that SOW closes.

693 + - **`virtualContexts` follow-ups**: any `defer-with-sow` disposition from audit 1.4 spawns a new SOW.

694 + - **Chart-recipe handle manifest**: no longer planned for v1. The 2026-05-14 full-shape redesign uses explicit typed item bodies instead of recipe handles. A future handle system would require a separate SOW and schema bump.

695 + - **Any new top-level section additions** to `sections.yaml` after v1: each is a small new SOW.

696 + - **`only_views:` schema feature reopens** if audit 1.1 surfaces multi-axis view conditioning (more than `single_node | multi_node`): future SOW with `taxonomy_version` bump.

697 +

698 +## Outcome

699 +

700 +In progress for the current PR. Delivered locally on branch `sow-0016-taxonomy-framework-poc`:

701 +

702 +- collector-adjacent `taxonomy.yaml` authoring contract;

703 +- closed authoring/output schemas and section/icon registries;

704 +- `gen_taxonomy.py`, `gen_taxonomy_seed.py`, and `check_collector_taxonomy.py`;

705 +- PR and master-regeneration workflow wiring;

706 +- integrations README, SOW spec, and project-skill updates;

707 +- five full-shape POC collector taxonomies for Apache, MySQL, Postgres, NVIDIA, and SNMP.

708 +

709 +## Lessons Extracted

710 +

711 +- Stable `section_id` values need generated path segments derived from the final ID component; otherwise opaque IDs with namespace punctuation produce duplicated paths such as `applications.applications.apache`.

712 +- Full-shape POC files must exercise real dashboard structures. Schema-valid flat lists are not enough to prove the contract.

713 +

714 +## Followup

715 +

716 +- Full collector taxonomy migration for the remaining collectors, including global all-collector fatality timing.

717 +- Downstream cloud-frontend consumption SOW: fetch/copy `integrations/taxonomy.json`, implement the adapter, and remove legacy taxonomy modules on the FE team's schedule.

718 +- Production ibm.d taxonomy generation from module source data.

719 +- Broader TAX negative-test matrix and more precise stale-shape diagnostics.

720 +- Invalid-metadata surfacing and schema-error message-code hardening beyond the fixes included in the framework PR.

721 +- Opt-out POC example for a no-metadata/dynamic plugin.

722 +- Future top-level section additions or `only_views:`-style view-axis expansion require separate SOWs and schema-version changes.

723 +

724 +## Regression Log

725 +

726 +None yet.

727 +

728 +Append regression entries here only after this SOW was completed or closed and later testing or use found broken behavior. Use a dated `## Regression - YYYY-MM-DD` heading at the end of the file. Never prepend regression content above the original SOW narrative.

.agents/sow/specs/taxonomy.md new

+197

@@ -0,0 +1,197 @@

1 +# Collector Taxonomy

2 +

3 +Collector chart taxonomy is authored in public repo source files and

4 +generated into a dashboard-consumable JSON artifact.

5 +

6 +## Source Files

7 +

8 +- Collector taxonomy authoring file:

9 + `<collector>/taxonomy.yaml`, sibling to `metadata.yaml`.

10 +- Section registry:

11 + `integrations/taxonomy/sections.yaml`.

12 +- Icon registry:

13 + `integrations/taxonomy/icons.yaml`.

14 +- Schemas:

15 + `integrations/schemas/taxonomy_collector.json`,

16 + `integrations/schemas/taxonomy_sections.json`,

17 + `integrations/schemas/taxonomy_output.json`.

18 +- Generator:

19 + `integrations/gen_taxonomy.py`.

20 +- Touched-collector checker:

21 + `integrations/check_collector_taxonomy.py`.

22 +- Seed helper:

23 + `integrations/gen_taxonomy_seed.py`.

24 +

25 +## Authoring Contract

26 +

27 +`taxonomy.yaml` v1 is a closed schema. Unknown core keys fail

28 +validation; extension keys must be prefixed with `x_`.

29 +

30 +Required top-level fields:

31 +

32 +- `taxonomy_version: 1`

33 +- `plugin_name`

34 +- `module_name`

35 +- either `placements` or `taxonomy_optout`, not both

36 +

37 +Each placement requires `id`, `section_id`, `title`, and `items`.

38 +`section_id` is the only accepted v1 section reference.

39 +`section_path` is rejected in authoring files. Section IDs are stable

40 +opaque handles; hierarchy is defined by `parent_id` in

41 +`sections.yaml`.

42 +

43 +`items:` is an ordered recursive tree. The allowed item kinds are:

44 +

45 +- context-string shorthand such as `mysql.queries`; this is an owning

46 + structural context leaf and normalizes to `type: owned_context`;

47 +- `type: owned_context` with one literal `context`;

48 +- `type: group` with stable hand-authored `id`, `title`, and nested

49 + structural `items`;

50 +- `type: flatten`, the structural equivalent of the legacy FE

51 + `properties.justGroup` behavior;

52 +- `type: selector` with one selector mechanism;

53 +- `type: context`, a display widget that references contexts and

54 + requires `contexts` plus `chart_library`;

55 +- `type: grid`, `type: first_available`, and `type: view_switch` for

56 + dashboard widget composition.

57 +

58 +Strings are allowed only in structural positions: placement `items`,

59 +`group.items`, and `flatten.items`. Grid bodies, first-available

60 +alternatives, and view-switch branches must use explicit object

61 +items. Nested `flatten` under `flatten.items` is rejected. The

62 +recursion matrix is exhaustive: unlisted container/item combinations

63 +are invalid.

64 +

65 +`single_node:` is a sparse same-kind delta only. It may override

66 +display, selector, or renderer fields allowed on the same item type.

67 +It may not contain `type`, `items`, `multi_node`, `single_node`, or

68 +change an owning item into a display widget. Whole-body single-vs-

69 +multi differences use `type: view_switch`; `view_switch` and sparse

70 +`single_node` are mutually exclusive on the same item.

71 +

72 +Renderer-private payloads live only under `renderer:`. Current known

73 +keys are `renderer.overlays`, `renderer.url_options`, and

74 +`renderer.toolbox_elements`; future renderer-only additions must use

75 +`x_*` under `renderer`. `toolbox_elements`, `overlays`, and

76 +`url_options` are not valid item-body siblings.

77 +

78 +## Selectors And Context References

79 +

80 +Every literal context owned by `owned_context` or referenced by a

81 +`context` widget must exist in the owning collector's `metadata.yaml`

82 +under `metrics.scopes[].metrics[].name`, unless the exact reference

83 +uses the explicit unresolved-reference escape hatch:

84 +

85 +```yaml

86 +contexts:

87 + - context: mysql.future_context

88 + unresolved:

89 + reason: staged downstream rollout

90 + owner: cloud-frontend

91 + expires: "2026-08-01"

92 +```

93 +

94 +Dynamic collectors must opt in from metadata:

95 +

96 +```yaml

97 +metrics:

98 + dynamic_context_prefixes:

99 + - prefix: snmp.

100 + reason: SNMP profiles emit vendor-specific contexts at runtime.

101 + dynamic_collect_plugins:

102 + - plugin: statsd.plugin

103 + reason: statsd synthetic charts are operator-defined.

104 +```

105 +

106 +`type: selector` owns the contexts it resolves from `context_prefix`

107 +or `collect_plugin`. Selector objects inside a widget `contexts:`

108 +array reference contexts but do not own them. `context_prefix_exclude`

109 +is valid only on the same item/reference that also has

110 +`context_prefix`. A `context_prefix:` value may narrow a declared

111 +metadata dynamic namespace; for example, a collector that declares

112 +`dynamic_context_prefixes: [{prefix: snmp., ...}]` may use

113 +`context_prefix: [snmp.device_prof_]` in taxonomy authoring.

114 +

115 +`collect_plugin:` selects by Agent `_collect_plugin` label. It is for

116 +dynamic contexts that do not share a stable context-name prefix.

117 +

118 +## Output Artifact

119 +

120 +`integrations/gen_taxonomy.py` emits gitignored

121 +`integrations/taxonomy.json`:

122 +

123 +- `taxonomy_schema_version`

124 +- `source.netdata_commit`

125 +- `source.generated_at`

126 +- normalized `sections`

127 +- normalized `placements`

128 +- `opted_out_collectors`

129 +

130 +Each placement and item includes:

131 +

132 +- `resolved_contexts`: contexts owned by structural strings,

133 + `owned_context`, and selector items.

134 +- `referenced_contexts`: contexts referenced by display widgets,

135 + grids, first-available alternatives, and view-switch widget

136 + branches.

137 +- `unresolved_references`: explicit unresolved-reference escape

138 + hatches with `context`, `reason`, `owner`, `expires`, and

139 + `item_path`. This is the durable signal that a widget reference is

140 + intentionally unresolved instead of accidentally missing. `expires`

141 + uses `YYYY-MM-DD`.

142 +

143 +The snapshots are deterministic for identical repository input and

144 +preserve author item order for the recursive tree.

145 +

146 +## CI Contract

147 +

148 +Pull requests run `integrations/check_collector_taxonomy.py` from

149 +`.github/workflows/check-markdown.yml`. The checker:

150 +

151 +- validates all committed `taxonomy.yaml` files;

152 +- validates the generated artifact shape;

153 +- fails when a PR adds/removes a collector `taxonomy.yaml`;

154 +- fails when a PR edits a collector `metadata.yaml` metrics block

155 + without a sibling `taxonomy.yaml`.

156 +

157 +The master regeneration workflow runs `gen_taxonomy.py` and removes

158 +the gitignored artifact during cleanup.

159 +

160 +## Finding Codes

161 +

162 +Active v1 codes:

163 +

164 +| Code | Severity | Meaning |

165 +|---|---|---|

166 +| TAX001 | fatal | Schema/load failure or missing matching metadata. |

167 +| TAX002 | reserved | Reserved for a future empty-effective-node lint; current empty authoring shapes fail schema/load validation as TAX001. |

168 +| TAX003 | fatal | Literal context is not declared by the owning collector metadata. |

169 +| TAX006 | fatal | Duplicate section or placement ownership key. |

170 +| TAX021 | fatal | Invalid `single_node` override key or shape. |

171 +| TAX022 | fatal | `multi_node:` used outside `type: view_switch`. |

172 +| TAX023 | fatal | List-merge syntax such as `*_extend` used; v1 lists replace. |

173 +| TAX024 | warning | Empty `single_node:` block. |

174 +| TAX025 | warning | `single_node` override equals the top-level value. |

175 +| TAX028 | fatal | Unknown/deprecated section, unknown icon, or invalid section authoring shape. |

176 +| TAX029 | fatal | Invalid dynamic declaration location or `context_prefix_exclude` usage. |

177 +| TAX030 | fatal | Touched collector needs `taxonomy.yaml` coverage. |

178 +| TAX031 | fatal | `context_prefix` used without metadata opt-in. |

179 +| TAX032 | reserved | Reserved for a future narrower prefix-overlap diagnostic; current selector ownership overlap conflicts emit TAX036. |

180 +| TAX033 | fatal | Resolved context owned by more than one placement. |

181 +| TAX034 | warning | Literal context is redundant because a prefix already covers it. |

182 +| TAX035 | fatal | `collect_plugin` used without metadata opt-in. |

183 +| TAX036 | fatal | Selector overlap conflict across collector/type boundaries. |

184 +| TAX037 | fatal | Literal context is referenced by a widget but not owned by any structural item. |

185 +| TAX038 | warning | `unresolved` escape hatch is stale because the context now resolves. |

186 +

187 +Removed v1 codes:

188 +

189 +- TAX026 and TAX027 were removed with `only_views`.

190 +- TAX040 through TAX042 were removed with chart-recipe manifests.

191 +

192 +## Contributor Rule

193 +

194 +Collector context changes and taxonomy changes move together. A PR

195 +that adds, removes, or renames chart contexts must update

196 +`metadata.yaml` and `taxonomy.yaml` in the same change unless the

197 +collector uses a declared dynamic selector that covers the context.

.github/workflows/check-markdown.yml

+11

@@ -7,6 +7,7 @@ on:

7 - '**/*.mdx'

8 - 'docs/**'

9 - '**/metadata.yaml'

10 + - '**/taxonomy.yaml'

11 - 'integrations/**'

13 concurrency:

@@ -46,6 +47,16 @@ jobs:

47 source ./venv/bin/activate

48 cd netdata && python3 integrations/gen_integrations.py

50 + - name: Check Collector Taxonomy

51 + run: |

52 + source ./venv/bin/activate

53 + cd netdata && python3 integrations/check_collector_taxonomy.py --pr-diff "${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}"

54 +

55 + - name: Test Collector Taxonomy Tooling

56 + run: |

57 + source ./venv/bin/activate

58 + cd netdata && python3 -m unittest integrations.tests.test_taxonomy

59 +

60 - name: Generate Integrations Documentation

61 run: |

62 source ./venv/bin/activate

.github/workflows/generate-integrations.yml

+20 -1

@@ -9,15 +9,22 @@ on:

9 - 'src/collectors/**/metadata.yaml'

10 - 'src/crates/**/metadata.yaml'

11 - 'src/go/plugin/**/metadata.yaml'

12 + - '**/taxonomy.yaml'

13 - 'src/exporting/**/metadata.yaml'

14 - 'src/health/notifications/**/metadata.yaml'

15 + - 'integrations/taxonomy/**'

16 - 'integrations/templates/**'

17 - 'integrations/schemas/**'

18 - 'integrations/categories.yaml'

19 - 'integrations/deploy.yaml'

20 - 'integrations/cloud-notifications/metadata.yaml'

21 - 'integrations/cloud-authentication/metadata.yaml'

22 + - 'integrations/_common.py'

23 - 'integrations/gen_integrations.py'

24 + - 'integrations/gen_taxonomy.py'

25 + - 'integrations/gen_taxonomy_seed.py'

26 + - 'integrations/check_collector_taxonomy.py'

27 + - 'integrations/tests/**'

28 - 'integrations/gen_docs_integrations.py'

29 - 'integrations/gen_doc_collector_page.py'

30 - 'integrations/gen_doc_secrets_page.py'

@@ -49,6 +56,16 @@ jobs:

56 run: |

57 source ./virtualenv/bin/activate

58 python3 integrations/gen_integrations.py

59 + - name: Generate Collector Taxonomy

60 + id: generate-taxonomy

61 + run: |

62 + source ./virtualenv/bin/activate

63 + python3 integrations/gen_taxonomy.py

64 + - name: Test Collector Taxonomy Tooling

65 + id: test-taxonomy

66 + run: |

67 + source ./virtualenv/bin/activate

68 + python3 -m unittest integrations.tests.test_taxonomy

69 - name: Generate Integrations Documentation

70 id: generate-integrations-documentation

71 run: |

@@ -63,7 +80,7 @@ jobs:

80 python3 integrations/gen_doc_secrets_page.py

81 - name: Clean Up Temporary Data

82 id: clean

66 - run: rm -rf go.d.plugin virtualenv integrations/integrations.js integrations/integrations.json

83 + run: rm -rf go.d.plugin virtualenv integrations/integrations.js integrations/integrations.json integrations/taxonomy.json

84 - name: Create PR

85 id: create-pr

86 uses: peter-evans/create-pull-request@v8

@@ -92,6 +109,8 @@ jobs:

109 Checkout Agent: ${{ steps.checkout-agent.outcome }}

110 Prep python env and deps: ${{ steps.prep-deps.outcome }}

111 Generate Integrations: ${{ steps.generate.outcome }}

112 + Generate Collector Taxonomy: ${{ steps.generate-taxonomy.outcome }}

113 + Test Collector Taxonomy Tooling: ${{ steps.test-taxonomy.outcome }}

114 Generate Integrations Documentation: ${{ steps.generate-integrations-documentation.outcome }}

115 Generate src/collectors/COLLECTORS.md: ${{ steps.generate-collectors-md.outcome }}

116 Generate src/collectors/SECRETS.md: ${{ steps.generate-secrets-md.outcome }}

.gitignore

@@ -161,6 +161,7 @@ python.d/python-modules-installer.sh

161 # integration generated files

162 integrations/integrations.js

163 integrations/integrations.json

164 +integrations/taxonomy.json

165

166 # documentation generated files

167 docs/generator/src

AGENTS.md

+5 -1

@@ -309,7 +309,7 @@ Runtime input skills:

309 Purpose: mirror maintainer-preferred framework V2 patterns from accepted collectors so new or migrated modules blend with repository style.

310

311 - `.agents/skills/integrations-lifecycle/`

312 - Trigger: editing any `metadata.yaml`; modifying `integrations/` generators, schemas, or templates; working with `integrations.js` / `integrations.json` / per-integration `.md` files / `COLLECTORS.md` / `SECRETS.md` / `SERVICE-DISCOVERY.md`; ibm.d module generation (`contexts.yaml` -> `metadata.yaml`); CI workflows `generate-integrations.yml` and `check-markdown.yml`; the five-file collector-consistency rule.

312 + Trigger: editing any `metadata.yaml` or collector `taxonomy.yaml`; modifying `integrations/` generators, schemas, taxonomy registries, or templates; working with `integrations.js` / `integrations.json` / `integrations/taxonomy.json` / per-integration `.md` files / `COLLECTORS.md` / `SECRETS.md` / `SERVICE-DISCOVERY.md`; ibm.d module generation (`contexts.yaml` -> `metadata.yaml`); CI workflows `generate-integrations.yml` and `check-markdown.yml`; the collector-consistency rule.

313 Status: live. SKILL.md plus per-domain guides (`pipeline.md`, `schema-reference.md`, `per-type-matrix.md`, `artifacts-and-banners.md`, `ibm-d.md`, `consistency.md`, `in-app-contract.md`, `gotchas.md`) and `recipes/`, `how-tos/` directories.

314

315 - `.agents/skills/learn-site-structure/`

@@ -406,11 +406,15 @@ When working on collectors (especially Go collectors), ALL of the following file

406 - How it works

407 - Configuration options

408 - Troubleshooting

409 +7. **taxonomy.yaml** - Dashboard table-of-contents placement for the collector's chart contexts

410

411 These files MUST be consistent with each other. For example:

412 - If units change in code, they MUST be updated in metadata.yaml

413 - If new metrics are added, they MUST be documented in metadata.yaml and README.md

414 - If configuration options change, they MUST be updated in config_schema.json, stock config, and documentation

415 +- If chart contexts are added, removed, or renamed, taxonomy.yaml MUST still resolve to real metadata.yaml contexts or declared dynamic selectors

416 +

417 +Unlike the other consistency artifacts, taxonomy.yaml coverage is enforced fatally in CI by `integrations/check_collector_taxonomy.py` running in `.github/workflows/check-markdown.yml`.

418

419 ## C code

420 - gcc, clang, glibc and muslc

integrations/README.md

+29 -1

@@ -1,4 +1,5 @@

1 -To generate a copy of `integrations.js` locally, you will need:

1 +To generate a copy of `integrations.js` and validate collector

2 +taxonomy locally, you will need:

4 - Python 3.6 or newer (only tested on Python 3.10 currently, should work

5 on any version of Python newer than 3.6).

@@ -24,9 +25,36 @@ Once the environment is set up, run the documentation generators from

25 the Agent repo root:

27 - `integrations/gen_integrations.py`

28 +- `integrations/gen_taxonomy.py --check-only`

29 +- `integrations/check_collector_taxonomy.py`

30 - `integrations/gen_docs_integrations.py`

31 - `integrations/gen_doc_collector_page.py`

32 - `integrations/gen_doc_secrets_page.py`

34 These scripts must be run _from this specific location_, as they use

35 their own path to figure out where all the files they need are.

36 +

37 +Collector dashboard taxonomy is authored in sibling `taxonomy.yaml`

38 +files next to collector `metadata.yaml` files. Static collectors use

39 +ordered `items:` trees; a plain context string in `items:` owns that

40 +chart context and normalizes to `type: owned_context`. Display widgets

41 +use `type: context` with `contexts:` and `chart_library`, and every

42 +referenced literal context must be owned somewhere in the structural

43 +tree. Dynamic collectors use `type: selector` with `context_prefix:`

44 +or `collect_plugin:` and must opt in from `metadata.yaml` with

45 +`metrics.dynamic_context_prefixes:` or

46 +`metrics.dynamic_collect_plugins:`; a taxonomy `context_prefix:` may

47 +narrow a declared metadata namespace. The generated

48 +`integrations/taxonomy.json` artifact is gitignored like

49 +`integrations/integrations.js`.

50 +

51 +To seed a static collector taxonomy from existing metadata contexts:

52 +

53 +```bash

54 +python3 integrations/gen_taxonomy_seed.py src/go/plugin/go.d/collector/apache/metadata.yaml --module-name apache --section-id applications.apache --placement-id apache --icon apache

55 +```

56 +

57 +Pull requests run `integrations/check_collector_taxonomy.py` from

58 +`.github/workflows/check-markdown.yml`. The gate validates committed

59 +taxonomy files and fails when a collector `metadata.yaml` metrics block

60 +or `taxonomy.yaml` changes without matching taxonomy coverage.

integrations/_common.py new

+160

@@ -0,0 +1,160 @@

1 +import json

2 +import os

3 +from pathlib import Path

4 +

5 +from jsonschema import Draft7Validator, ValidationError

6 +from referencing import Registry, Resource

7 +from referencing.jsonschema import DRAFT7

8 +from ruamel.yaml import YAML, YAMLError

9 +

10 +

11 +AGENT_REPO = 'netdata/netdata'

12 +

13 +INTEGRATIONS_PATH = Path(__file__).parent

14 +REPO_PATH = INTEGRATIONS_PATH.parent

15 +SCHEMA_PATH = INTEGRATIONS_PATH / 'schemas'

16 +METADATA_PATTERN = '*/metadata.yaml'

17 +

18 +COLLECTOR_SOURCES = [

19 + (AGENT_REPO, REPO_PATH / 'src' / 'collectors', True),

20 + (AGENT_REPO, REPO_PATH / 'src' / 'collectors' / 'charts.d.plugin', True),

21 + (AGENT_REPO, REPO_PATH / 'src' / 'collectors' / 'python.d.plugin', True),

22 + (AGENT_REPO, REPO_PATH / 'src' / 'collectors' / 'guides', True),

23 + (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'go.d' / 'collector', True),

24 + (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'scripts.d' / 'collector', True),

25 + (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'ibm.d' / 'modules', True),

26 + (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'ibm.d' / 'modules' / 'websphere', True),

27 + (AGENT_REPO, REPO_PATH / 'src' / 'crates' / 'netdata-otel', True),

28 +]

29 +

30 +GITHUB_ACTIONS = os.environ.get('GITHUB_ACTIONS', False)

31 +DEBUG = os.environ.get('DEBUG', False)

32 +WARNINGS = []

33 +

34 +

35 +def debug(msg):

36 + if GITHUB_ACTIONS:

37 + print(f'::debug::{msg}')

38 + elif DEBUG:

39 + print(f'>>> {msg}')

40 + else:

41 + pass

42 +

43 +

44 +def warn(msg, path):

45 + WARNINGS.append((str(path), msg))

46 +

47 + if GITHUB_ACTIONS:

48 + print(f'::warning file={path}::{msg}')

49 + else:

50 + print(f'!!! WARNING:{path}:{msg}')

51 +

52 +

53 +def fail_on_warnings():

54 + if not WARNINGS:

55 + return 0

56 +

57 + warned_files = sorted({path for path, _ in WARNINGS})

58 + print(f'::error::Integrations generation failed with {len(WARNINGS)} warning(s) across {len(warned_files)} file(s).')

59 +

60 + for path in warned_files:

61 + print(f'::error file={path}::Metadata warnings in this file are now fatal for integrations generation.')

62 +

63 + return 1

64 +

65 +

66 +def retrieve_from_filesystem(uri):

67 + path = SCHEMA_PATH / Path(uri)

68 + contents = json.loads(path.read_text())

69 + return Resource.from_contents(contents, DRAFT7)

70 +

71 +

72 +registry = Registry(retrieve=retrieve_from_filesystem)

73 +

74 +

75 +def make_validator(schema_ref):

76 + return Draft7Validator(

77 + {'$ref': schema_ref},

78 + registry=registry,

79 + )

80 +

81 +

82 +COLLECTOR_VALIDATOR = make_validator('./collector.json#')

83 +

84 +

85 +def get_collector_metadata_entries():

86 + ret = []

87 +

88 + for r, d, m in COLLECTOR_SOURCES:

89 + if d.exists() and d.is_dir() and m:

90 + for item in d.glob(METADATA_PATTERN):

91 + ret.append((r, item))

92 + elif d.exists() and d.is_file() and not m:

93 + if d.match(METADATA_PATTERN):

94 + ret.append((r, d))

95 +

96 + return ret

97 +

98 +

99 +def load_yaml(src):

100 + yaml = YAML(typ='safe')

101 +

102 + if not src.is_file():

103 + warn(f'{src} is not a file.', src)

104 + return False

105 +

106 + try:

107 + contents = src.read_text()

108 + except (IOError, OSError):

109 + warn(f'Failed to read {src}.', src)

110 + return False

111 +

112 + try:

113 + data = yaml.load(contents)

114 + except YAMLError:

115 + warn(f'Failed to parse {src} as YAML.', src)

116 + return False

117 +

118 + return data

119 +

120 +

121 +def load_collectors():

122 + ret = []

123 +

124 + entries = get_collector_metadata_entries()

125 +

126 + for repo, path in entries:

127 + debug(f'Loading {path}.')

128 + data = load_yaml(path)

129 +

130 + if not data:

131 + continue

132 +

133 + try:

134 + COLLECTOR_VALIDATOR.validate(data)

135 + except ValidationError as e:

136 + warn(

137 + f'Failed to validate {path} against the schema: {e.message} (path: {"/".join(str(p) for p in e.absolute_path)})',

138 + path)

139 + continue

140 +

141 + for idx, item in enumerate(data['modules']):

142 + item['meta']['plugin_name'] = data['plugin_name']

143 + item['integration_type'] = 'collector'

144 + item['_src_path'] = path

145 + item['_repo'] = repo

146 + item['_index'] = idx

147 + ret.append(item)

148 +

149 + return ret

150 +

151 +

152 +def make_id(meta):

153 + if 'monitored_instance' in meta:

154 + instance_name = meta['monitored_instance']['name'].replace(' ', '_')

155 + elif 'instance_name' in meta:

156 + instance_name = meta['instance_name']

157 + else:

158 + instance_name = '000_unknown'

159 +

160 + return f'{meta["plugin_name"]}-{meta["module_name"]}-{instance_name}'

integrations/check_collector_taxonomy.py new

+156

@@ -0,0 +1,156 @@

1 +#!/usr/bin/env python3

2 +

3 +import argparse

4 +import re

5 +import subprocess

6 +import sys

7 +from pathlib import Path

8 +

9 +from ruamel.yaml import YAML, YAMLError

10 +

11 +from gen_taxonomy import FATAL, Finding, build_taxonomy, relpath

12 +from _common import REPO_PATH

13 +

14 +HUNK_RE = re.compile(r'^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@')

15 +

16 +

17 +def run_git(*args):

18 + return subprocess.check_output(['git', '-C', str(REPO_PATH), *args], text=True)

19 +

20 +

21 +def metadata_metrics_spans(path):

22 + text = path.read_text()

23 + lines = text.splitlines()

24 + yaml = YAML(typ='rt')

25 + try:

26 + data = yaml.load(text)

27 + except YAMLError:

28 + return []

29 +

30 + spans = []

31 + modules = data.get('modules', []) if isinstance(data, dict) else []

32 + for module_index, module in enumerate(modules):

33 + if not isinstance(module, dict) or 'metrics' not in module:

34 + continue

35 + try:

36 + start = module.lc.key('metrics')[0] + 1

37 + except (AttributeError, KeyError, TypeError):

38 + continue

39 +

40 + next_lines = []

41 + for key in module:

42 + if key == 'metrics':

43 + continue

44 + try:

45 + key_line = module.lc.key(key)[0] + 1

46 + except (AttributeError, KeyError, TypeError):

47 + continue

48 + if key_line > start:

49 + next_lines.append(key_line)

50 +

51 + if next_lines:

52 + end = min(next_lines) - 1

53 + else:

54 + next_module_line = None

55 + try:

56 + if module_index + 1 < len(modules):

57 + next_module_line = modules.lc.item(module_index + 1)[0] + 1

58 + except (AttributeError, KeyError, TypeError):

59 + next_module_line = None

60 + end = (next_module_line - 1) if next_module_line else len(lines)

61 +

62 + spans.append((start, end))

63 + return spans

64 +

65 +

66 +def range_intersects_spans(start, length, spans):

67 + if length == 0:

68 + changed_start = start

69 + changed_end = start

70 + else:

71 + changed_start = start

72 + changed_end = start + length - 1

73 + return any(changed_start <= span_end and changed_end >= span_start for span_start, span_end in spans)

74 +

75 +

76 +def metadata_metrics_touched(diff_range, path):

77 + if not path.exists():

78 + return True

79 +

80 + diff = run_git('diff', '--unified=0', diff_range, '--', relpath(path))

81 + if not diff.strip():

82 + return False

83 +

84 + spans = metadata_metrics_spans(path)

85 + if not spans:

86 + return True

87 +

88 + for line in diff.splitlines():

89 + match = HUNK_RE.match(line)

90 + if not match:

91 + continue

92 + start = int(match.group(1))

93 + length = int(match.group(2) or '1')

94 + if range_intersects_spans(start, length, spans):

95 + return True

96 + return False

97 +

98 +

99 +def touched_collectors(diff_range):

100 + output = run_git('diff', '--name-status', diff_range)

101 + touched = set()

102 + for line in output.splitlines():

103 + if not line.strip():

104 + continue

105 + fields = line.split('\t')

106 + status = fields[0]

107 + path = REPO_PATH / fields[-1]

108 + name = path.name

109 +

110 + if name == 'taxonomy.yaml':

111 + touched.add(path.parent)

112 + elif name == 'metadata.yaml':

113 + if status.startswith(('A', 'D')):

114 + touched.add(path.parent)

115 + elif metadata_metrics_touched(diff_range, path):

116 + touched.add(path.parent)

117 + return sorted(touched)

118 +

119 +

120 +def check_touched_coverage(diff_range):

121 + findings = []

122 + for collector_dir in touched_collectors(diff_range):

123 + taxonomy_path = collector_dir / 'taxonomy.yaml'

124 + metadata_path = collector_dir / 'metadata.yaml'

125 + if not taxonomy_path.exists() and not metadata_path.exists():

126 + continue

127 + if not taxonomy_path.exists():

128 + findings.append(Finding(

129 + code='TAX030',

130 + severity=FATAL,

131 + path=taxonomy_path,

132 + message='Collector metrics or taxonomy changed, but taxonomy.yaml is missing.',

133 + ))

134 + return findings

135 +

136 +

137 +def main():

138 + parser = argparse.ArgumentParser(description='Validate collector taxonomy coverage and taxonomy artifact generation.')

139 + parser.add_argument('--pr-diff', help='Git diff range for touched-collector coverage, for example origin/master...HEAD.')

140 + args = parser.parse_args()

141 +

142 + findings = []

143 + if args.pr_diff:

144 + findings.extend(check_touched_coverage(args.pr_diff))

145 +

146 + _, taxonomy_findings = build_taxonomy()

147 + findings.extend(taxonomy_findings)

148 +

149 + for finding in findings:

150 + print(finding.render(), file=sys.stderr)

151 +

152 + return 1 if any(finding.severity == FATAL for finding in findings) else 0

153 +

154 +

155 +if __name__ == '__main__':

156 + sys.exit(main())

integrations/gen_integrations.py

+25 -200

@@ -1,40 +1,31 @@

1 #!/usr/bin/env python3

3 import json

4 -import os

4 import re

5 import sys

6 from copy import deepcopy

8 -from pathlib import Path

10 -from jsonschema import Draft7Validator, ValidationError

11 -from referencing import Registry, Resource

12 -from referencing.jsonschema import DRAFT7

13 -from ruamel.yaml import YAML, YAMLError

14 -

15 -AGENT_REPO = 'netdata/netdata'

8 +from jsonschema import ValidationError

9 +

10 +from _common import (

11 + AGENT_REPO,

12 + INTEGRATIONS_PATH,

13 + METADATA_PATTERN,

14 + REPO_PATH,

15 + debug,

16 + fail_on_warnings,

17 + load_collectors,

18 + load_yaml,

19 + make_id,

20 + make_validator,

21 + warn,

22 +)

17 -INTEGRATIONS_PATH = Path(__file__).parent

24 TEMPLATE_PATH = INTEGRATIONS_PATH / 'templates'

25 OUTPUT_PATH = INTEGRATIONS_PATH / 'integrations.js'

26 JSON_PATH = INTEGRATIONS_PATH / 'integrations.json'

27 CATEGORIES_FILE = INTEGRATIONS_PATH / 'categories.yaml'

22 -REPO_PATH = INTEGRATIONS_PATH.parent

23 -SCHEMA_PATH = INTEGRATIONS_PATH / 'schemas'

28 DISTROS_FILE = REPO_PATH / '.github' / 'data' / 'distros.yml'

25 -METADATA_PATTERN = '*/metadata.yaml'

26 -

27 -COLLECTOR_SOURCES = [

28 - (AGENT_REPO, REPO_PATH / 'src' / 'collectors', True),

29 - (AGENT_REPO, REPO_PATH / 'src' / 'collectors' / 'charts.d.plugin', True),

30 - (AGENT_REPO, REPO_PATH / 'src' / 'collectors' / 'python.d.plugin', True),

31 - (AGENT_REPO, REPO_PATH / 'src' / 'collectors' / 'guides', True),

32 - (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'go.d' / 'collector', True),

33 - (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'scripts.d' / 'collector', True),

34 - (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'ibm.d' / 'modules', True),

35 - (AGENT_REPO, REPO_PATH / 'src' / 'go' / 'plugin' / 'ibm.d' / 'modules' / 'websphere', True),

36 - (AGENT_REPO, REPO_PATH / 'src' / 'crates' / 'netdata-otel', True),

37 -]

30 FLOWS_SOURCES = [

31 (AGENT_REPO, REPO_PATH / 'src' / 'crates' / 'netflow-plugin' / 'metadata.yaml', False),

@@ -138,104 +129,16 @@ SERVICE_DISCOVERY_RENDER_KEYS = [

129 CUSTOM_TAG_PATTERN = re.compile('\\{% if .*?%\\}.*?\\{% /if %\\}|\\{%.*?%\\}', flags=re.DOTALL)

130 FIXUP_BLANK_PATTERN = re.compile('\\\\\\n *\\n')

131

141 -GITHUB_ACTIONS = os.environ.get('GITHUB_ACTIONS', False)

142 -DEBUG = os.environ.get('DEBUG', False)

143 -WARNINGS = []

144 -

145 -

146 -def debug(msg):

147 - if GITHUB_ACTIONS:

148 - print(f':debug:{msg}')

149 - elif DEBUG:

150 - print(f'>>> {msg}')

151 - else:

152 - pass

153 -

154 -

155 -def warn(msg, path):

156 - WARNINGS.append((str(path), msg))

157 -

158 - if GITHUB_ACTIONS:

159 - print(f':warning file={path}:{msg}')

160 - else:

161 - print(f'!!! WARNING:{path}:{msg}')

162 -

163 -

164 -def fail_on_warnings():

165 - if not WARNINGS:

166 - return 0

167 -

168 - warned_files = sorted({path for path, _ in WARNINGS})

169 - print(f':error:Integrations generation failed with {len(WARNINGS)} warning(s) across {len(warned_files)} file(s).')

170 -

171 - for path in warned_files:

172 - print(f':error file={path}:Metadata warnings in this file are now fatal for integrations generation.')

173 -

174 - return 1

175 -

176 -

177 -def retrieve_from_filesystem(uri):

178 - path = SCHEMA_PATH / Path(uri)

179 - contents = json.loads(path.read_text())

180 - return Resource.from_contents(contents, DRAFT7)

181 -

182 -

183 -registry = Registry(retrieve=retrieve_from_filesystem)

184 -

185 -CATEGORY_VALIDATOR = Draft7Validator(

186 - {'$ref': './categories.json#'},

187 - registry=registry,

188 -)

189 -

190 -DEPLOY_VALIDATOR = Draft7Validator(

191 - {'$ref': './deploy.json#'},

192 - registry=registry,

193 -)

194 -

195 -EXPORTER_VALIDATOR = Draft7Validator(

196 - {'$ref': './exporter.json#'},

197 - registry=registry,

198 -)

199 -

200 -AGENT_NOTIFICATION_VALIDATOR = Draft7Validator(

201 - {'$ref': './agent_notification.json#'},

202 - registry=registry,

203 -)

204 -

205 -CLOUD_NOTIFICATION_VALIDATOR = Draft7Validator(

206 - {'$ref': './cloud_notification.json#'},

207 - registry=registry,

208 -)

209 -

210 -LOGS_VALIDATOR = Draft7Validator(

211 - {'$ref': './logs.json#'},

212 - registry=registry,

213 -)

214 -

215 -AUTHENTICATION_VALIDATOR = Draft7Validator(

216 - {'$ref': './authentication.json#'},

217 - registry=registry,

218 -)

219 -

220 -COLLECTOR_VALIDATOR = Draft7Validator(

221 - {'$ref': './collector.json#'},

222 - registry=registry,

223 -)

224 -

225 -FLOWS_VALIDATOR = Draft7Validator(

226 - {'$ref': './flows.json#'},

227 - registry=registry,

228 -)

229 -

230 -SECRETSTORE_VALIDATOR = Draft7Validator(

231 - {'$ref': './secretstore.json#'},

232 - registry=registry,

233 -)

234 -

235 -SERVICE_DISCOVERY_VALIDATOR = Draft7Validator(

236 - {'$ref': './service_discovery.json#'},

237 - registry=registry,

238 -)

132 +CATEGORY_VALIDATOR = make_validator('./categories.json#')

133 +DEPLOY_VALIDATOR = make_validator('./deploy.json#')

134 +EXPORTER_VALIDATOR = make_validator('./exporter.json#')

135 +AGENT_NOTIFICATION_VALIDATOR = make_validator('./agent_notification.json#')

136 +CLOUD_NOTIFICATION_VALIDATOR = make_validator('./cloud_notification.json#')

137 +LOGS_VALIDATOR = make_validator('./logs.json#')

138 +AUTHENTICATION_VALIDATOR = make_validator('./authentication.json#')

139 +FLOWS_VALIDATOR = make_validator('./flows.json#')

140 +SECRETSTORE_VALIDATOR = make_validator('./secretstore.json#')

141 +SERVICE_DISCOVERY_VALIDATOR = make_validator('./service_discovery.json#')

142

143 _jinja_env = False

144

@@ -323,42 +226,6 @@ def get_category_sets(categories):

226 return (default, valid)

227

228

326 -def get_collector_metadata_entries():

327 - ret = []

328 -

329 - for r, d, m in COLLECTOR_SOURCES:

330 - if d.exists() and d.is_dir() and m:

331 - for item in d.glob(METADATA_PATTERN):

332 - ret.append((r, item))

333 - elif d.exists() and d.is_file() and not m:

334 - if d.match(METADATA_PATTERN):

335 - ret.append((r, d))

336 -

337 - return ret

338 -

339 -

340 -def load_yaml(src):

341 - yaml = YAML(typ='safe')

342 -

343 - if not src.is_file():

344 - warn(f'{src} is not a file.', src)

345 - return False

346 -

347 - try:

348 - contents = src.read_text()

349 - except (IOError, OSError):

350 - warn(f'Failed to read {src}.', src)

351 - return False

352 -

353 - try:

354 - data = yaml.load(contents)

355 - except YAMLError:

356 - warn(f'Failed to parse {src} as YAML.', src)

357 - return False

358 -

359 - return data

360 -

361 -

229 def load_categories():

230 categories = load_yaml(CATEGORIES_FILE)

231

@@ -376,37 +243,6 @@ def load_categories():

243 return categories

244

245

379 -def load_collectors():

380 - ret = []

381 -

382 - entries = get_collector_metadata_entries()

383 -

384 - for repo, path in entries:

385 - debug(f'Loading {path}.')

386 - data = load_yaml(path)

387 -

388 - if not data:

389 - continue

390 -

391 - try:

392 - COLLECTOR_VALIDATOR.validate(data)

393 - except ValidationError as e:

394 - warn(

395 - f'Failed to validate {path} against the schema: {e.message} (path: {"/".join(str(p) for p in e.absolute_path)})',

396 - path)

397 - continue

398 -

399 - for idx, item in enumerate(data['modules']):

400 - item['meta']['plugin_name'] = data['plugin_name']

401 - item['integration_type'] = 'collector'

402 - item['_src_path'] = path

403 - item['_repo'] = repo

404 - item['_index'] = idx

405 - ret.append(item)

406 -

407 - return ret

408 -

409 -

246 def load_flows():

247 ret = []

248

@@ -819,17 +655,6 @@ def load_service_discoveries():

655 return ret

656

657

822 -def make_id(meta):

823 - if 'monitored_instance' in meta:

824 - instance_name = meta['monitored_instance']['name'].replace(' ', '_')

825 - elif 'instance_name' in meta:

826 - instance_name = meta['instance_name']

827 - else:

828 - instance_name = '000_unknown'

829 -

830 - return f'{meta["plugin_name"]}-{meta["module_name"]}-{instance_name}'

831 -

832 -

658 def make_edit_link(item):

659 item_path = item['_src_path'].relative_to(REPO_PATH)

660

integrations/gen_taxonomy.py new

+929

@@ -0,0 +1,929 @@

1 +#!/usr/bin/env python3

2 +

3 +import argparse

4 +import bisect

5 +import json

6 +import subprocess

7 +import sys

8 +import unicodedata

9 +from dataclasses import dataclass

10 +from pathlib import Path

11 +

12 +from _common import (

13 + COLLECTOR_SOURCES,

14 + GITHUB_ACTIONS,

15 + INTEGRATIONS_PATH,

16 + REPO_PATH,

17 + WARNINGS,

18 + load_collectors,

19 + load_yaml,

20 + make_id,

21 + make_validator,

22 +)

23 +

24 +TAXONOMY_PATH = INTEGRATIONS_PATH / 'taxonomy'

25 +SECTIONS_PATH = TAXONOMY_PATH / 'sections.yaml'

26 +ICONS_PATH = TAXONOMY_PATH / 'icons.yaml'

27 +OUTPUT_PATH = INTEGRATIONS_PATH / 'taxonomy.json'

28 +

29 +SECTIONS_VALIDATOR = make_validator('./taxonomy_sections.json#')

30 +COLLECTOR_TAXONOMY_VALIDATOR = make_validator('./taxonomy_collector.json#')

31 +OUTPUT_VALIDATOR = make_validator('./taxonomy_output.json#')

32 +

33 +FATAL = 'fatal'

34 +WARNING = 'warning'

35 +

36 +DISPLAY_KEYS = (

37 + 'title',

38 + 'short_name',

39 + 'icon',

40 + 'priority',

41 + 'families',

42 + 'tooltip',

43 + 'menu_pattern',

44 + 'hide_sub_icon',

45 + 'force_visibility',

46 + 'fallback_icon',

47 + 'include_grand_parents',

48 + 'properties',

49 +)

50 +

51 +WIDGET_KEYS = (

52 + 'chart_library',

53 + 'group_by',

54 + 'group_by_label',

55 + 'aggregation_method',

56 + 'selected_dimensions',

57 + 'dimensions_sort',

58 + 'colors',

59 + 'layout',

60 + 'table_columns',

61 + 'table_sort_by',

62 + 'labels',

63 + 'value_range',

64 + 'eliminate_zero_dimensions',

65 + 'context_items',

66 + 'post_group_by',

67 + 'show_post_aggregations',

68 + 'grouping_method',

69 + 'sparkline',

70 + 'renderer',

71 +)

72 +

73 +SELECTOR_KEYS = ('context_prefix', 'context_prefix_exclude', 'collect_plugin')

74 +SORTED_LIST_KEYS = set(SELECTOR_KEYS)

75 +

76 +ITEM_COPY_KEYS = {

77 + 'owned_context': ('context', *DISPLAY_KEYS, 'single_node'),

78 + 'group': ('id', *DISPLAY_KEYS, 'section_filters', 'dyncfg', 'single_node'),

79 + 'flatten': ('id', *DISPLAY_KEYS, 'single_node'),

80 + 'selector': ('id', *DISPLAY_KEYS, *SELECTOR_KEYS, 'single_node'),

81 + 'context': ('id', *DISPLAY_KEYS, 'contexts', *WIDGET_KEYS, 'single_node'),

82 + 'grid': ('id', *DISPLAY_KEYS, 'renderer', 'single_node'),

83 + 'first_available': ('id', *DISPLAY_KEYS, 'single_node'),

84 + 'view_switch': ('id',),

85 +}

86 +

87 +PLACEMENT_COPY_KEYS = (

88 + 'short_name',

89 + 'icon',

90 + 'priority',

91 + 'families',

92 + 'tooltip',

93 + 'menu_pattern',

94 + 'hide_sub_icon',

95 + 'force_visibility',

96 + 'fallback_icon',

97 + 'include_grand_parents',

98 + 'properties',

99 + 'single_node',

100 +)

101 +

102 +

103 +@dataclass(frozen=True)

104 +class Finding:

105 + code: str

106 + severity: str

107 + path: Path

108 + message: str

109 + line: int | None = None

110 +

111 + def render(self):

112 + location = str(self.path)

113 + if GITHUB_ACTIONS:

114 + line = f',line={self.line}' if self.line else ''

115 + level = 'error' if self.severity == FATAL else 'warning'

116 + return f'::{level} file={location}{line},title={self.code}::{self.message}'

117 +

118 + line = f':{self.line}' if self.line else ''

119 + return f'{location}{line}: {self.severity.upper()} {self.code}: {self.message}'

120 +

121 +

122 +def relpath(path):

123 + try:

124 + return path.relative_to(REPO_PATH).as_posix()

125 + except ValueError:

126 + return path.as_posix()

127 +

128 +

129 +def normalize_title(value):

130 + normalized = unicodedata.normalize('NFC', value or '')

131 + return normalized.casefold()

132 +

133 +

134 +def path_segment(section):

135 + return section['id'].rsplit('.', 1)[-1]

136 +

137 +

138 +def run_git(*args):

139 + try:

140 + return subprocess.check_output(

141 + ['git', '-C', str(REPO_PATH), *args],

142 + text=True,

143 + stderr=subprocess.DEVNULL,

144 + ).strip()

145 + except (subprocess.CalledProcessError, FileNotFoundError):

146 + return 'unknown'

147 +

148 +

149 +def source_info():

150 + return {

151 + 'netdata_commit': run_git('rev-parse', 'HEAD'),

152 + 'generated_at': run_git('log', '-1', '--format=%cI'),

153 + }

154 +

155 +

156 +def discover_taxonomy_files():

157 + files = []

158 + for _, root, recursive in COLLECTOR_SOURCES:

159 + if root.exists() and root.is_dir() and recursive:

160 + files.extend(root.glob('*/taxonomy.yaml'))

161 + elif root.exists() and root.is_file() and root.name == 'taxonomy.yaml':

162 + files.append(root)

163 + return sorted(set(files), key=lambda p: relpath(p))

164 +

165 +

166 +def validate_schema(validator, data, path, default_code, findings):

167 + valid = True

168 + for error in sorted(validator.iter_errors(data), key=lambda e: list(e.absolute_path)):

169 + valid = False

170 + code = default_code

171 + absolute_path = [str(part) for part in error.absolute_path]

172 + if 'single_node' in absolute_path:

173 + code = 'TAX021'

174 + elif error.validator == 'additionalProperties' and 'section_path' in error.message:

175 + code = 'TAX028'

176 + findings.append(Finding(

177 + code=code,

178 + severity=FATAL,

179 + path=path,

180 + message=f'{error.message} (schema path: {"/".join(str(p) for p in error.absolute_schema_path)})',

181 + ))

182 + return valid

183 +

184 +

185 +def load_icons(findings):

186 + data = load_yaml(ICONS_PATH)

187 + if not data:

188 + findings.append(Finding('TAX001', FATAL, ICONS_PATH, 'Unable to load taxonomy icon registry.'))

189 + return set()

190 + icons = data.get('icons', [])

191 + seen = set()

192 + for icon in icons:

193 + if icon in seen:

194 + findings.append(Finding('TAX028', FATAL, ICONS_PATH, f'Duplicate icon id: {icon}'))

195 + seen.add(icon)

196 + return seen

197 +

198 +

199 +def load_sections(findings, icons):

200 + data = load_yaml(SECTIONS_PATH)

201 + if not data:

202 + findings.append(Finding('TAX001', FATAL, SECTIONS_PATH, 'Unable to load taxonomy sections registry.'))

203 + return [], {}

204 +

205 + if not validate_schema(SECTIONS_VALIDATOR, data, SECTIONS_PATH, 'TAX001', findings):

206 + return [], {}

207 +

208 + sections = data['sections']

209 + by_id = {}

210 + for section in sections:

211 + section_id = section['id']

212 + if section_id in by_id:

213 + findings.append(Finding('TAX006', FATAL, SECTIONS_PATH, f'Duplicate section id: {section_id}'))

214 + by_id[section_id] = section

215 +

216 + for section in sections:

217 + icon = section.get('icon')

218 + if icon and icon not in icons:

219 + findings.append(Finding('TAX028', FATAL, SECTIONS_PATH, f'Section {section["id"]} references unknown icon: {icon}'))

220 +

221 + parent_id = section.get('parent_id')

222 + if parent_id and parent_id not in by_id:

223 + findings.append(Finding('TAX028', FATAL, SECTIONS_PATH, f'Section {section["id"]} references unknown parent_id: {parent_id}'))

224 +

225 + deprecation = section.get('deprecation', {})

226 + replacement_id = deprecation.get('replacement_id')

227 + if replacement_id and replacement_id not in by_id:

228 + findings.append(Finding('TAX028', FATAL, SECTIONS_PATH, f'Section {section["id"]} references unknown replacement_id: {replacement_id}'))

229 +

230 + paths = {}

231 +

232 + def resolve_path(section_id, visiting):

233 + if section_id in paths:

234 + return paths[section_id]

235 + if section_id in visiting:

236 + findings.append(Finding('TAX028', FATAL, SECTIONS_PATH, f'Section parent cycle includes: {section_id}'))

237 + return section_id

238 + section = by_id[section_id]

239 + parent_id = section.get('parent_id')

240 + if not parent_id:

241 + paths[section_id] = section_id

242 + return section_id

243 + paths[section_id] = f'{resolve_path(parent_id, visiting | {section_id})}.{path_segment(section)}'

244 + return paths[section_id]

245 +

246 + for section_id in by_id:

247 + resolve_path(section_id, set())

248 +

249 + emitted = []

250 + for section in sorted(sections, key=lambda s: (s['section_order'], normalize_title(s['title']), s['id'])):

251 + item = {

252 + 'id': section['id'],

253 + 'path': paths[section['id']],

254 + 'title': section['title'],

255 + 'section_order': section['section_order'],

256 + 'status': section['status'],

257 + }

258 + for key in ('parent_id', 'short_name', 'icon', 'deprecation'):

259 + if key in section:

260 + item[key] = section[key]

261 + extras = {k: v for k, v in section.items() if k.startswith('x_')}

262 + if extras:

263 + item['_extra'] = extras

264 + emitted.append(item)

265 +

266 + return emitted, {section_id: (section, paths[section_id]) for section_id, section in by_id.items()}

267 +

268 +

269 +def module_contexts(module):

270 + contexts = []

271 + for scope in module.get('metrics', {}).get('scopes', []):

272 + for metric in scope.get('metrics', []):

273 + name = metric.get('name')

274 + if name and name not in contexts:

275 + contexts.append(name)

276 + return contexts

277 +

278 +

279 +def dynamic_declarations(module):

280 + metrics = module.get('metrics', {})

281 + prefixes = {item['prefix'] for item in metrics.get('dynamic_context_prefixes', [])}

282 + plugins = {item['plugin'] for item in metrics.get('dynamic_collect_plugins', [])}

283 + return prefixes, plugins

284 +

285 +

286 +def build_metadata_indexes(findings):

287 + warning_start = len(WARNINGS)

288 + modules = load_collectors()

289 + for path, message in WARNINGS[warning_start:]:

290 + findings.append(Finding('TAX001', FATAL, Path(path), message))

291 +

292 + by_path_module = {}

293 + context_to_modules = {}

294 + contexts_by_plugin = {}

295 + all_contexts = set()

296 +

297 + for module in modules:

298 + src_path = Path(module['_src_path'])

299 + meta = module['meta']

300 + key = (src_path, meta['plugin_name'], meta['module_name'])

301 + by_path_module.setdefault(key, []).append(module)

302 +

303 + contexts = module_contexts(module)

304 + contexts_by_plugin.setdefault(meta['plugin_name'], set()).update(contexts)

305 + for context in contexts:

306 + all_contexts.add(context)

307 + context_to_modules.setdefault(context, set()).add(key)

308 +

309 + return {

310 + 'modules': modules,

311 + 'by_path_module': by_path_module,

312 + 'context_to_modules': context_to_modules,

313 + 'contexts_by_plugin': contexts_by_plugin,

314 + 'all_contexts': sorted(all_contexts),

315 + }

316 +

317 +

318 +def prescan_removed_shapes(data, path, findings):

319 + def walk(node):

320 + if isinstance(node, dict):

321 + if 'multi_node' in node and node.get('type') != 'view_switch':

322 + findings.append(Finding('TAX022', FATAL, path, '`multi_node:` is accepted only inside `type: view_switch`.'))

323 + for key, value in node.items():

324 + if key.endswith('_extend'):

325 + findings.append(Finding('TAX023', FATAL, path, f'List-merge field `{key}:` is not accepted in taxonomy v1.'))

326 + walk(value)

327 + elif isinstance(node, list):

328 + for item in node:

329 + walk(item)

330 +

331 + walk(data)

332 +

333 +

334 +def collector_ids(modules):

335 + ids = []

336 + for module in modules:

337 + ids.append(make_id(module['meta']))

338 + return sorted(ids)

339 +

340 +

341 +def merged_module_contexts(modules):

342 + contexts = set()

343 + for module in modules:

344 + contexts.update(module_contexts(module))

345 + return contexts

346 +

347 +

348 +def merged_dynamic_declarations(modules, inline):

349 + prefixes = set()

350 + plugins = set()

351 + for module in modules:

352 + module_prefixes, module_plugins = dynamic_declarations(module)

353 + prefixes.update(module_prefixes)

354 + plugins.update(module_plugins)

355 +

356 + if inline:

357 + prefixes.update(item['prefix'] for item in inline.get('dynamic_context_prefixes', []))

358 + plugins.update(item['plugin'] for item in inline.get('dynamic_collect_plugins', []))

359 +

360 + return prefixes, plugins

361 +

362 +

363 +def resolve_prefix(prefix, all_contexts):

364 + start = bisect.bisect_left(all_contexts, prefix)

365 + stop = bisect.bisect_left(all_contexts, prefix + chr(0x10ffff))

366 + return all_contexts[start:stop]

367 +

368 +

369 +def is_context_prefix_declared(prefix, allowed_prefixes):

370 + return any(prefix.startswith(allowed) for allowed in allowed_prefixes)

371 +

372 +

373 +def ordered_union(*sequences):

374 + seen = set()

375 + result = []

376 + for sequence in sequences:

377 + for item in sequence:

378 + if item not in seen:

379 + seen.add(item)

380 + result.append(item)

381 + return result

382 +

383 +

384 +def ordered_dict_union(*sequences):

385 + seen = set()

386 + result = []

387 + for sequence in sequences:

388 + for item in sequence:

389 + key = json.dumps(item, sort_keys=True)

390 + if key not in seen:

391 + seen.add(key)

392 + result.append(item)

393 + return result

394 +

395 +

396 +def node_label(node, fallback):

397 + if isinstance(node, str):

398 + return node

399 + return node.get('id') or node.get('context') or node.get('title') or fallback

400 +

401 +

402 +def validate_icons(node, path, icons, findings, label):

403 + for key in ('icon', 'fallback_icon'):

404 + icon = node.get(key)

405 + if icon and icon not in icons:

406 + findings.append(Finding('TAX028', FATAL, path, f'Item `{label}` references unknown {key}: {icon}'))

407 +

408 +

409 +def validate_override(parent, path, findings):

410 + if parent.get('type') == 'view_switch':

411 + return

412 + single_node = parent.get('single_node')

413 + if single_node is None:

414 + return

415 + if not single_node:

416 + findings.append(Finding('TAX024', WARNING, path, 'Empty `single_node:` block is equivalent to omitting it.'))

417 + return

418 + for key, value in single_node.items():

419 + if parent.get(key) == value:

420 + findings.append(Finding('TAX025', WARNING, path, f'`single_node.{key}` is identical to the top-level value.'))

421 +

422 +

423 +def copy_fields(node, output, keys):

424 + for key in keys:

425 + if key in node:

426 + value = node[key]

427 + if key in SORTED_LIST_KEYS:

428 + value = sorted(value)

429 + output[key] = value

430 +

431 +

432 +def emit_extra(node, output):

433 + extras = {k: v for k, v in node.items() if k.startswith('x_')}

434 + if extras:

435 + output['_extra'] = extras

436 +

437 +

438 +def resolve_selectors(node, known_contexts, allowed_prefixes, allowed_plugins, metadata_indexes, path, findings):

439 + resolved = set()

440 + explicit = node.get('contexts', [])

441 + prefixes = node.get('context_prefix', [])

442 + excludes = node.get('context_prefix_exclude', [])

443 + collect_plugins = node.get('collect_plugin', [])

444 +

445 + for context in explicit:

446 + if context not in known_contexts:

447 + findings.append(Finding('TAX003', FATAL, path, f'Unknown context for this collector: {context}'))

448 + resolved.add(context)

449 +

450 + if excludes and not prefixes:

451 + findings.append(Finding('TAX029', FATAL, path, '`context_prefix_exclude:` requires `context_prefix:` on the same node.'))

452 +

453 + for prefix in prefixes:

454 + if not is_context_prefix_declared(prefix, allowed_prefixes):

455 + findings.append(Finding('TAX031', FATAL, path, f'context_prefix `{prefix}` is not declared in metadata.yaml metrics.dynamic_context_prefixes.'))

456 + for context in resolve_prefix(prefix, metadata_indexes['all_contexts']):

457 + resolved.add(context)

458 +

459 + for exclude in excludes if prefixes else []:

460 + if not any(exclude.startswith(prefix) for prefix in prefixes):

461 + findings.append(Finding('TAX029', FATAL, path, f'context_prefix_exclude `{exclude}` is not covered by context_prefix.'))

462 + for context in list(resolved):

463 + if context.startswith(exclude):

464 + resolved.remove(context)

465 +

466 + for plugin in collect_plugins:

467 + if plugin not in allowed_plugins:

468 + findings.append(Finding('TAX035', FATAL, path, f'collect_plugin `{plugin}` is not declared in metadata.yaml metrics.dynamic_collect_plugins.'))

469 + resolved.update(metadata_indexes['contexts_by_plugin'].get(plugin, set()))

470 +

471 + for context in explicit:

472 + if any(context.startswith(prefix) for prefix in prefixes):

473 + findings.append(Finding('TAX034', WARNING, path, f'Context `{context}` is redundant because it is covered by context_prefix.'))

474 +

475 + return sorted(resolved)

476 +

477 +

478 +def resolve_node_contexts(node, known_contexts, allowed_prefixes, allowed_plugins, metadata_indexes, path, findings):

479 + return resolve_selectors(node, known_contexts, allowed_prefixes, allowed_plugins, metadata_indexes, path, findings)

480 +

481 +

482 +def validate_literal_context(context, known_contexts, unresolved, path, findings):

483 + known = context in known_contexts

484 + if unresolved:

485 + if known:

486 + findings.append(Finding('TAX038', WARNING, path, f'Unresolved escape hatch is stale because context now exists: {context}'))

487 + elif not known:

488 + findings.append(Finding('TAX003', FATAL, path, f'Unknown context for this collector: {context}'))

489 + return known

490 +

491 +

492 +def resolve_context_references(

493 + refs,

494 + known_contexts,

495 + allowed_prefixes,

496 + allowed_plugins,

497 + metadata_indexes,

498 + path,

499 + findings,

500 + referenced_literals,

501 + item_path):

502 + referenced = []

503 + unresolved_references = []

504 + for ref in refs:

505 + if isinstance(ref, str):

506 + known = validate_literal_context(ref, known_contexts, unresolved=False, path=path, findings=findings)

507 + referenced = ordered_union(referenced, [ref])

508 + referenced_literals.append((ref, item_path, path, False, known))

509 + continue

510 +

511 + if 'context' in ref:

512 + context = ref['context']

513 + known = validate_literal_context(context, known_contexts, unresolved=True, path=path, findings=findings)

514 + referenced = ordered_union(referenced, [context])

515 + referenced_literals.append((context, item_path, path, True, known))

516 + unresolved_references.append({

517 + 'context': context,

518 + 'reason': ref['unresolved']['reason'],

519 + 'owner': ref['unresolved']['owner'],

520 + 'expires': ref['unresolved']['expires'],

521 + 'item_path': item_path,

522 + })

523 + continue

524 +

525 + resolved = resolve_selectors(ref, known_contexts, allowed_prefixes, allowed_plugins, metadata_indexes, path, findings)

526 + referenced = ordered_union(referenced, resolved)

527 +

528 + return referenced, unresolved_references

529 +

530 +

531 +def register_ownership(contexts, ownership, current, owner_kind, path, ownership_conflicts):

532 + for context in contexts:

533 + previous = ownership.get(context)

534 + if previous and previous['owner'] != current:

535 + code = 'TAX036' if 'selector' in {previous['kind'], owner_kind} else 'TAX033'

536 + owners = tuple(sorted([previous['owner'], current]))

537 + ownership_conflicts[(code, context, owners)] = {

538 + 'path': path,

539 + }

540 + ownership[context] = {

541 + 'owner': current,

542 + 'kind': owner_kind,

543 + }

544 +

545 +

546 +def emit_ownership_conflicts(ownership_conflicts, findings):

547 + for code, context, owners in sorted(ownership_conflicts):

548 + findings.append(Finding(

549 + code,

550 + FATAL,

551 + ownership_conflicts[(code, context, owners)]['path'],

552 + f'Context `{context}` is owned by both {owners[0]} and {owners[1]}.',

553 + ))

554 +

555 +

556 +def emit_item(

557 + node,

558 + position,

559 + known_contexts,

560 + allowed_prefixes,

561 + allowed_plugins,

562 + metadata_indexes,

563 + icons,

564 + ownership,

565 + ownership_conflicts,

566 + referenced_literals,

567 + owner_label,

568 + path,

569 + findings,

570 + index):

571 + if isinstance(node, str):

572 + label = f'{owner_label}.{node}'

573 + validate_literal_context(node, known_contexts, unresolved=False, path=path, findings=findings)

574 + register_ownership([node], ownership, f'{relpath(path)}:{label}', 'literal', path, ownership_conflicts)

575 + return {

576 + 'type': 'owned_context',

577 + 'context': node,

578 + 'resolved_contexts': [node],

579 + 'referenced_contexts': [],

580 + 'unresolved_references': [],

581 + }

582 +

583 + kind = node['type']

584 + label = f'{owner_label}.{node_label(node, str(index))}'

585 + validate_icons(node, path, icons, findings, label)

586 + validate_override(node, path, findings)

587 +

588 + output = {'type': kind}

589 + copy_fields(node, output, ITEM_COPY_KEYS[kind])

590 + emit_extra(node, output)

591 +

592 + resolved_contexts = []

593 + referenced_contexts = []

594 + unresolved_references = []

595 +

596 + if kind == 'owned_context':

597 + context = node['context']

598 + validate_literal_context(context, known_contexts, unresolved=False, path=path, findings=findings)

599 + resolved_contexts = [context]

600 + register_ownership(resolved_contexts, ownership, f'{relpath(path)}:{label}', 'literal', path, ownership_conflicts)

601 +

602 + elif kind == 'selector':

603 + resolved_contexts = resolve_selectors(node, known_contexts, allowed_prefixes, allowed_plugins, metadata_indexes, path, findings)

604 + register_ownership(resolved_contexts, ownership, f'{relpath(path)}:{label}', 'selector', path, ownership_conflicts)

605 +

606 + elif kind in ('group', 'flatten'):

607 + children = []

608 + for child_index, child in enumerate(node.get('items', [])):

609 + emitted = emit_item(

610 + child,

611 + 'structural',

612 + known_contexts,

613 + allowed_prefixes,

614 + allowed_plugins,

615 + metadata_indexes,

616 + icons,

617 + ownership,

618 + ownership_conflicts,

619 + referenced_literals,

620 + label,

621 + path,

622 + findings,

623 + child_index,

624 + )

625 + children.append(emitted)

626 + resolved_contexts = ordered_union(resolved_contexts, emitted['resolved_contexts'])

627 + referenced_contexts = ordered_union(referenced_contexts, emitted['referenced_contexts'])

628 + unresolved_references = ordered_dict_union(unresolved_references, emitted['unresolved_references'])

629 + output['items'] = children

630 +

631 + elif kind == 'context':

632 + referenced_contexts, unresolved_references = resolve_context_references(

633 + node['contexts'],

634 + known_contexts,

635 + allowed_prefixes,

636 + allowed_plugins,

637 + metadata_indexes,

638 + path,

639 + findings,

640 + referenced_literals,

641 + label,

642 + )

643 +

644 + elif kind == 'grid':

645 + children = []

646 + for child_index, child in enumerate(node.get('items', [])):

647 + emitted = emit_item(

648 + child,

649 + 'display',

650 + known_contexts,

651 + allowed_prefixes,

652 + allowed_plugins,

653 + metadata_indexes,

654 + icons,

655 + ownership,

656 + ownership_conflicts,

657 + referenced_literals,

658 + label,

659 + path,

660 + findings,

661 + child_index,

662 + )

663 + children.append(emitted)

664 + resolved_contexts = ordered_union(resolved_contexts, emitted['resolved_contexts'])

665 + referenced_contexts = ordered_union(referenced_contexts, emitted['referenced_contexts'])

666 + unresolved_references = ordered_dict_union(unresolved_references, emitted['unresolved_references'])

667 + output['items'] = children

668 +

669 + elif kind == 'first_available':

670 + children = []

671 + for child_index, child in enumerate(node.get('items', [])):

672 + emitted = emit_item(

673 + child,

674 + 'display',

675 + known_contexts,

676 + allowed_prefixes,

677 + allowed_plugins,

678 + metadata_indexes,

679 + icons,

680 + ownership,

681 + ownership_conflicts,

682 + referenced_literals,

683 + label,

684 + path,

685 + findings,

686 + child_index,

687 + )

688 + children.append(emitted)

689 + resolved_contexts = ordered_union(resolved_contexts, emitted['resolved_contexts'])

690 + referenced_contexts = ordered_union(referenced_contexts, emitted['referenced_contexts'])

691 + unresolved_references = ordered_dict_union(unresolved_references, emitted['unresolved_references'])

692 + output['items'] = children

693 +

694 + elif kind == 'view_switch':

695 + branch_position = 'structural' if position == 'structural' else 'display'

696 + for branch in ('multi_node', 'single_node'):

697 + emitted = emit_item(

698 + node[branch],

699 + branch_position,

700 + known_contexts,

701 + allowed_prefixes,

702 + allowed_plugins,

703 + metadata_indexes,

704 + icons,

705 + ownership,

706 + ownership_conflicts,

707 + referenced_literals,

708 + f'{label}.{branch}',

709 + path,

710 + findings,

711 + 0,

712 + )

713 + output[branch] = emitted

714 + resolved_contexts = ordered_union(resolved_contexts, emitted['resolved_contexts'])

715 + referenced_contexts = ordered_union(referenced_contexts, emitted['referenced_contexts'])

716 + unresolved_references = ordered_dict_union(unresolved_references, emitted['unresolved_references'])

717 +

718 + if position != 'structural' and resolved_contexts:

719 + findings.append(Finding('TAX001', FATAL, path, f'Item `{label}` owns contexts from a display-only position.'))

720 +

721 + output['resolved_contexts'] = resolved_contexts

722 + output['referenced_contexts'] = referenced_contexts

723 + output['unresolved_references'] = unresolved_references

724 + return output

725 +

726 +

727 +def emit_referenced_only_findings(referenced_literals, ownership, findings):

728 + emitted = set()

729 + rows = sorted(referenced_literals, key=lambda row: (row[0], row[1], relpath(row[2])))

730 + for context, item_path, path, unresolved, known in rows:

731 + if unresolved or not known or context in ownership:

732 + continue

733 + key = (context, item_path, path)

734 + if key in emitted:

735 + continue

736 + emitted.add(key)

737 + findings.append(Finding(

738 + 'TAX037',

739 + FATAL,

740 + path,

741 + f'Context `{context}` is referenced by widget `{item_path}` but is not owned by any taxonomy item.',

742 + ))

743 +

744 +

745 +def process_taxonomy_file(path, sections, icons, metadata_indexes, ownership, findings, referenced_literals=None, ownership_conflicts=None):

746 + if referenced_literals is None:

747 + referenced_literals = []

748 + if ownership_conflicts is None:

749 + ownership_conflicts = {}

750 +

751 + data = load_yaml(path)

752 + if not data:

753 + findings.append(Finding('TAX001', FATAL, path, 'Unable to load taxonomy file.'))

754 + return [], []

755 +

756 + prescan_removed_shapes(data, path, findings)

757 + if not validate_schema(COLLECTOR_TAXONOMY_VALIDATOR, data, path, 'TAX001', findings):

758 + return [], []

759 +

760 + metadata_path = path.with_name('metadata.yaml')

761 + identity = (metadata_path, data['plugin_name'], data['module_name'])

762 + modules = metadata_indexes['by_path_module'].get(identity, [])

763 + inline = data.get('inline_dynamic_declarations')

764 +

765 + if modules and inline:

766 + findings.append(Finding('TAX029', FATAL, path, '`inline_dynamic_declarations:` is allowed only for collectors without metadata.yaml.'))

767 +

768 + known_contexts = merged_module_contexts(modules)

769 + allowed_prefixes, allowed_plugins = merged_dynamic_declarations(modules, inline)

770 + ids = collector_ids(modules) if modules else [f'{data["plugin_name"]}-{data["module_name"]}']

771 +

772 + if 'taxonomy_optout' in data:

773 + return [], [{

774 + 'collector_ids': ids,

775 + 'plugin_name': data['plugin_name'],

776 + 'module_name': data['module_name'],

777 + 'source_path': relpath(path),

778 + 'reason': data['taxonomy_optout']['reason'],

779 + }]

780 +

781 + if not modules and not inline:

782 + findings.append(Finding('TAX001', FATAL, path, 'No matching metadata.yaml module found and no inline_dynamic_declarations provided.'))

783 +

784 + emitted = []

785 + placement_keys = set()

786 +

787 + for placement in data['placements']:

788 + placement_key = (placement['section_id'], placement['id'])

789 + if placement_key in placement_keys:

790 + findings.append(Finding('TAX006', FATAL, path, f'Duplicate placement in collector taxonomy: {placement["section_id"]}.{placement["id"]}'))

791 + placement_keys.add(placement_key)

792 +

793 + section = sections.get(placement['section_id'])

794 + if not section:

795 + findings.append(Finding('TAX028', FATAL, path, f'Unknown section_id: {placement["section_id"]}'))

796 + section_path = placement['section_id']

797 + else:

798 + section_entry, section_path = section

799 + if section_entry['status'] == 'deprecated':

800 + findings.append(Finding('TAX028', FATAL, path, f'New placements cannot target deprecated section_id: {placement["section_id"]}'))

801 +

802 + validate_icons(placement, path, icons, findings, placement['id'])

803 + validate_override(placement, path, findings)

804 +

805 + items = []

806 + resolved_contexts = []

807 + referenced_contexts = []

808 + unresolved_references = []

809 + for index, child in enumerate(placement['items']):

810 + emitted_child = emit_item(

811 + child,

812 + 'structural',

813 + known_contexts,

814 + allowed_prefixes,

815 + allowed_plugins,

816 + metadata_indexes,

817 + icons,

818 + ownership,

819 + ownership_conflicts,

820 + referenced_literals,

821 + placement['id'],

822 + path,

823 + findings,

824 + index,

825 + )

826 + items.append(emitted_child)

827 + resolved_contexts = ordered_union(resolved_contexts, emitted_child['resolved_contexts'])

828 + referenced_contexts = ordered_union(referenced_contexts, emitted_child['referenced_contexts'])

829 + unresolved_references = ordered_dict_union(unresolved_references, emitted_child['unresolved_references'])

830 +

831 + item = {

832 + 'collector_ids': ids,

833 + 'plugin_name': data['plugin_name'],

834 + 'module_name': data['module_name'],

835 + 'source_path': relpath(path),

836 + 'id': placement['id'],

837 + 'section_id': placement['section_id'],

838 + 'section_path': section_path,

839 + 'title': placement['title'],

840 + 'items': items,

841 + 'resolved_contexts': resolved_contexts,

842 + 'referenced_contexts': referenced_contexts,

843 + 'unresolved_references': unresolved_references,

844 + }

845 + copy_fields(placement, item, PLACEMENT_COPY_KEYS)

846 + emit_extra(placement, item)

847 + emitted.append(item)

848 +

849 + return emitted, []

850 +

851 +

852 +def build_taxonomy():

853 + findings = []

854 + icons = load_icons(findings)

855 + section_entries, sections = load_sections(findings, icons)

856 + metadata_indexes = build_metadata_indexes(findings)

857 + ownership = {}

858 + ownership_conflicts = {}

859 + referenced_literals = []

860 + placements = []

861 + opted_out_collectors = []

862 +

863 + for path in discover_taxonomy_files():

864 + new_placements, new_optouts = process_taxonomy_file(path, sections, icons, metadata_indexes, ownership, findings, referenced_literals, ownership_conflicts)

865 + placements.extend(new_placements)

866 + opted_out_collectors.extend(new_optouts)

867 +

868 + emit_ownership_conflicts(ownership_conflicts, findings)

869 + emit_referenced_only_findings(referenced_literals, ownership, findings)

870 +

871 + placements.sort(key=lambda item: (

872 + sections.get(item['section_id'], ({'section_order': 100000}, item['section_path']))[0]['section_order'],

873 + item.get('priority', 1000),

874 + normalize_title(item['title']),

875 + item['id'],

876 + item['source_path'],

877 + ))

878 +

879 + taxonomy = {

880 + 'taxonomy_schema_version': 1,

881 + 'source': source_info(),

882 + 'sections': section_entries,

883 + 'placements': placements,

884 + 'opted_out_collectors': sorted(opted_out_collectors, key=lambda item: (item['plugin_name'], item['module_name'], item['source_path'])),

885 + }

886 +

887 + validate_schema(OUTPUT_VALIDATOR, taxonomy, OUTPUT_PATH, 'TAX001', findings)

888 + return taxonomy, findings

889 +

890 +

891 +def write_json(path, data):

892 + path.write_text(json.dumps(data, indent=2, sort_keys=True) + '\n')

893 +

894 +

895 +def main():

896 + parser = argparse.ArgumentParser(description='Generate Netdata collector taxonomy artifact.')

897 + parser.add_argument('--check-only', action='store_true', help='Validate taxonomy sources without writing taxonomy.json.')

898 + parser.add_argument('--output', type=Path, default=OUTPUT_PATH, help='Output JSON path.')

899 + parser.add_argument('--findings-json', type=Path, help='Optional path for machine-readable findings.')

900 + args = parser.parse_args()

901 +

902 + taxonomy, findings = build_taxonomy()

903 +

904 + for finding in findings:

905 + print(finding.render(), file=sys.stderr)

906 +

907 + if args.findings_json:

908 + write_json(args.findings_json, [

909 + {

910 + 'code': finding.code,

911 + 'severity': finding.severity,

912 + 'path': relpath(finding.path),

913 + 'line': finding.line,

914 + 'message': finding.message,

915 + }

916 + for finding in findings

917 + ])

918 +

919 + if any(finding.severity == FATAL for finding in findings):

920 + return 1

921 +

922 + if not args.check_only:

923 + write_json(args.output, taxonomy)

924 +

925 + return 0

926 +

927 +

928 +if __name__ == '__main__':

929 + sys.exit(main())

integrations/gen_taxonomy_seed.py new

+101

@@ -0,0 +1,101 @@

1 +#!/usr/bin/env python3

2 +

3 +import argparse

4 +import sys

5 +from pathlib import Path

6 +

7 +from ruamel.yaml import YAML

8 +

9 +from _common import load_yaml

10 +

11 +

12 +def contexts_from_metadata(metadata, module_name):

13 + modules = [

14 + module for module in metadata.get('modules', [])

15 + if module.get('meta', {}).get('module_name') == module_name

16 + ]

17 + contexts = []

18 + for module in modules:

19 + for scope in module.get('metrics', {}).get('scopes', []):

20 + for metric in scope.get('metrics', []):

21 + name = metric.get('name')

22 + if name and name not in contexts:

23 + contexts.append(name)

24 + return contexts

25 +

26 +

27 +def default_module_name(metadata):

28 + modules = metadata.get('modules', [])

29 + if len(modules) != 1:

30 + return None

31 + return modules[0]['meta']['module_name']

32 +

33 +

34 +def monitored_title(metadata, module_name):

35 + for module in metadata.get('modules', []):

36 + if module.get('meta', {}).get('module_name') == module_name:

37 + return module.get('meta', {}).get('monitored_instance', {}).get('name', module_name)

38 + return module_name

39 +

40 +

41 +def build_seed(metadata_path, module_name, section_id, placement_id, icon):

42 + metadata = load_yaml(metadata_path)

43 + if not metadata:

44 + raise SystemExit(f'Failed to load {metadata_path}')

45 +

46 + if not module_name:

47 + module_name = default_module_name(metadata)

48 + if not module_name:

49 + raise SystemExit('metadata.yaml has multiple modules; pass --module-name explicitly.')

50 +

51 + contexts = contexts_from_metadata(metadata, module_name)

52 + if not contexts:

53 + raise SystemExit(f'No metric contexts found for module {module_name}.')

54 +

55 + if not placement_id:

56 + placement_id = module_name.replace('_', '-')

57 + title = monitored_title(metadata, module_name)

58 +

59 + placement = {

60 + 'id': placement_id,

61 + 'section_id': section_id,

62 + 'title': title,

63 + 'items': contexts,

64 + }

65 + if icon:

66 + placement['icon'] = icon

67 +

68 + return {

69 + 'taxonomy_version': 1,

70 + 'plugin_name': metadata['plugin_name'],

71 + 'module_name': module_name,

72 + 'placements': [placement],

73 + }

74 +

75 +

76 +def main():

77 + parser = argparse.ArgumentParser(description='Seed a collector taxonomy.yaml from metadata.yaml contexts.')

78 + parser.add_argument('metadata_yaml', type=Path)

79 + parser.add_argument('--module-name', help='metadata.yaml module name to seed.')

80 + parser.add_argument('--section-id', default='TODO.section', help='Initial section_id value.')

81 + parser.add_argument('--placement-id', help='Initial placement id. Defaults to module-name with underscores replaced.')

82 + parser.add_argument('--icon', help='Optional icon id.')

83 + parser.add_argument('--output', type=Path, help='Write to this path instead of stdout.')

84 + args = parser.parse_args()

85 +

86 + seed = build_seed(args.metadata_yaml, args.module_name, args.section_id, args.placement_id, args.icon)

87 + yaml = YAML()

88 + yaml.default_flow_style = False

89 + yaml.indent(mapping=2, sequence=4, offset=2)

90 +

91 + if args.output:

92 + with args.output.open('w') as fp:

93 + yaml.dump(seed, fp)

94 + else:

95 + yaml.dump(seed, sys.stdout)

96 +

97 + return 0

98 +

99 +

100 +if __name__ == '__main__':

101 + sys.exit(main())

integrations/schemas/collector.json

+48

@@ -280,6 +280,54 @@

280 "description": "Availability condition name."

281 }

282 },

283 + "dynamic_context_prefixes": {

284 + "type": "array",

285 + "description": "Dynamic chart context prefixes this collector is allowed to reference from taxonomy.yaml. Use only when the collector emits contexts that are not fully listed in metadata.yaml but share a stable namespace prefix.",

286 + "items": {

287 + "type": "object",

288 + "additionalProperties": false,

289 + "properties": {

290 + "prefix": {

291 + "type": "string",

292 + "minLength": 1,

293 + "description": "Static prefix shared by dynamic chart contexts."

294 + },

295 + "reason": {

296 + "type": "string",

297 + "minLength": 1,

298 + "description": "Why this collector needs a dynamic context-prefix selector."

299 + }

300 + },

301 + "required": [

302 + "prefix",

303 + "reason"

304 + ]

305 + }

306 + },

307 + "dynamic_collect_plugins": {

308 + "type": "array",

309 + "description": "Agent collect_plugin labels this collector is allowed to reference from taxonomy.yaml. Use only when dynamic chart contexts cannot be selected by stable context prefix.",

310 + "items": {

311 + "type": "object",

312 + "additionalProperties": false,

313 + "properties": {

314 + "plugin": {

315 + "type": "string",

316 + "minLength": 1,

317 + "description": "Agent _collect_plugin label value."

318 + },

319 + "reason": {

320 + "type": "string",

321 + "minLength": 1,

322 + "description": "Why this collector needs a collect_plugin selector."

323 + }

324 + },

325 + "required": [

326 + "plugin",

327 + "reason"

328 + ]

329 + }

330 + },

331 "scopes": {

332 "type": "array",

333 "description": "List of scopes and their metrics.",

integrations/schemas/taxonomy_collector.json new

+1302

@@ -0,0 +1,1302 @@

1 +{

2 + "$schema": "http://json-schema.org/draft-07/schema#",

3 + "type": "object",

4 + "title": "Netdata collector taxonomy authoring file.",

5 + "additionalProperties": false,

6 + "patternProperties": {

7 + "^x_": {}

8 + },

9 + "properties": {

10 + "taxonomy_version": {

11 + "type": "integer",

12 + "const": 1

13 + },

14 + "plugin_name": {

15 + "type": "string",

16 + "minLength": 1

17 + },

18 + "module_name": {

19 + "type": "string",

20 + "minLength": 1

21 + },

22 + "inline_dynamic_declarations": {

23 + "$ref": "#/$defs/dynamic_declarations"

24 + },

25 + "taxonomy_optout": {

26 + "type": "object",

27 + "additionalProperties": false,

28 + "patternProperties": {

29 + "^x_": {}

30 + },

31 + "properties": {

32 + "reason": {

33 + "type": "string",

34 + "minLength": 1

35 + }

36 + },

37 + "required": [

38 + "reason"

39 + ]

40 + },

41 + "placements": {

42 + "type": "array",

43 + "items": {

44 + "$ref": "#/$defs/placement"

45 + },

46 + "minItems": 1

47 + }

48 + },

49 + "required": [

50 + "taxonomy_version",

51 + "plugin_name",

52 + "module_name"

53 + ],

54 + "oneOf": [

55 + {

56 + "required": [

57 + "placements"

58 + ],

59 + "not": {

60 + "required": [

61 + "taxonomy_optout"

62 + ]

63 + }

64 + },

65 + {

66 + "required": [

67 + "taxonomy_optout"

68 + ],

69 + "not": {

70 + "required": [

71 + "placements"

72 + ]

73 + }

74 + }

75 + ],

76 + "$defs": {

77 + "dynamic_declarations": {

78 + "type": "object",

79 + "additionalProperties": false,

80 + "properties": {

81 + "dynamic_context_prefixes": {

82 + "type": "array",

83 + "items": {

84 + "$ref": "#/$defs/dynamic_context_prefix"

85 + },

86 + "minItems": 1

87 + },

88 + "dynamic_collect_plugins": {

89 + "type": "array",

90 + "items": {

91 + "$ref": "#/$defs/dynamic_collect_plugin"

92 + },

93 + "minItems": 1

94 + }

95 + }

96 + },

97 + "dynamic_context_prefix": {

98 + "type": "object",

99 + "additionalProperties": false,

100 + "properties": {

101 + "prefix": {

102 + "type": "string",

103 + "minLength": 1

104 + },

105 + "reason": {

106 + "type": "string",

107 + "minLength": 1

108 + }

109 + },

110 + "required": [

111 + "prefix",

112 + "reason"

113 + ]

114 + },

115 + "dynamic_collect_plugin": {

116 + "type": "object",

117 + "additionalProperties": false,

118 + "properties": {

119 + "plugin": {

120 + "type": "string",

121 + "minLength": 1

122 + },

123 + "reason": {

124 + "type": "string",

125 + "minLength": 1

126 + }

127 + },

128 + "required": [

129 + "plugin",

130 + "reason"

131 + ]

132 + },

133 + "id": {

134 + "type": "string",

135 + "pattern": "^[a-z0-9][a-z0-9_.-]*$"

136 + },

137 + "non_empty_string": {

138 + "type": "string",

139 + "minLength": 1

140 + },

141 + "string_list": {

142 + "type": "array",

143 + "items": {

144 + "type": "string",

145 + "minLength": 1

146 + },

147 + "minItems": 1

148 + },

149 + "families": {

150 + "oneOf": [

151 + {

152 + "type": "boolean"

153 + },

154 + {

155 + "$ref": "#/$defs/string_list"

156 + }

157 + ]

158 + },

159 + "properties": {

160 + "type": "object",

161 + "additionalProperties": false,

162 + "properties": {

163 + "important": {

164 + "type": "boolean"

165 + },

166 + "grouping": {

167 + "type": "boolean"

168 + },

169 + "small": {

170 + "type": "boolean"

171 + },

172 + "is_placeholder": {

173 + "type": "boolean"

174 + }

175 + }

176 + },

177 + "layout": {

178 + "type": "object",

179 + "additionalProperties": false,

180 + "properties": {

181 + "left": {

182 + "type": "number"

183 + },

184 + "top": {

185 + "type": "number"

186 + },

187 + "width": {

188 + "type": "number"

189 + },

190 + "height": {

191 + "type": "number"

192 + }

193 + },

194 + "required": [

195 + "left",

196 + "top",

197 + "width",

198 + "height"

199 + ]

200 + },

201 + "table_sort_item": {

202 + "type": "object",

203 + "additionalProperties": false,

204 + "properties": {

205 + "id": {

206 + "$ref": "#/$defs/non_empty_string"

207 + },

208 + "desc": {

209 + "type": "boolean"

210 + }

211 + },

212 + "required": [

213 + "id",

214 + "desc"

215 + ]

216 + },

217 + "context_item": {

218 + "type": "object",

219 + "additionalProperties": false,

220 + "properties": {

221 + "value": {

222 + "$ref": "#/$defs/non_empty_string"

223 + },

224 + "label": {

225 + "$ref": "#/$defs/non_empty_string"

226 + }

227 + },

228 + "required": [

229 + "value",

230 + "label"

231 + ]

232 + },

233 + "renderer": {

234 + "type": "object",

235 + "additionalProperties": false,

236 + "patternProperties": {

237 + "^x_": {}

238 + },

239 + "properties": {

240 + "overlays": {

241 + "type": "array",

242 + "items": {

243 + "type": "object"

244 + }

245 + },

246 + "url_options": {

247 + "type": "object"

248 + },

249 + "toolbox_elements": {

250 + "type": "array",

251 + "items": {

252 + "type": "object"

253 + }

254 + }

255 + }

256 + },

257 + "labels": {

258 + "type": "object",

259 + "additionalProperties": {

260 + "$ref": "#/$defs/non_empty_string"

261 + }

262 + },

263 + "unresolved_reference": {

264 + "type": "object",

265 + "additionalProperties": false,

266 + "properties": {

267 + "reason": {

268 + "$ref": "#/$defs/non_empty_string"

269 + },

270 + "owner": {

271 + "$ref": "#/$defs/non_empty_string"

272 + },

273 + "expires": {

274 + "type": "string",

275 + "pattern": "^\\d{4}-\\d{2}-\\d{2}$"

276 + }

277 + },

278 + "required": [

279 + "reason",

280 + "owner",

281 + "expires"

282 + ]

283 + },

284 + "literal_context_reference": {

285 + "type": "object",

286 + "additionalProperties": false,

287 + "properties": {

288 + "context": {

289 + "$ref": "#/$defs/non_empty_string"

290 + },

291 + "unresolved": {

292 + "$ref": "#/$defs/unresolved_reference"

293 + }

294 + },

295 + "required": [

296 + "context",

297 + "unresolved"

298 + ]

299 + },

300 + "selector_context_reference": {

301 + "type": "object",

302 + "additionalProperties": false,

303 + "properties": {

304 + "context_prefix": {

305 + "$ref": "#/$defs/string_list"

306 + },

307 + "context_prefix_exclude": {

308 + "$ref": "#/$defs/string_list"

309 + },

310 + "collect_plugin": {

311 + "$ref": "#/$defs/string_list"

312 + }

313 + },

314 + "oneOf": [

315 + {

316 + "required": [

317 + "context_prefix"

318 + ]

319 + },

320 + {

321 + "required": [

322 + "collect_plugin"

323 + ]

324 + }

325 + ]

326 + },

327 + "context_reference": {

328 + "oneOf": [

329 + {

330 + "$ref": "#/$defs/non_empty_string"

331 + },

332 + {

333 + "$ref": "#/$defs/literal_context_reference"

334 + },

335 + {

336 + "$ref": "#/$defs/selector_context_reference"

337 + }

338 + ]

339 + },

340 + "context_references": {

341 + "type": "array",

342 + "items": {

343 + "$ref": "#/$defs/context_reference"

344 + },

345 + "minItems": 1

346 + },

347 + "single_node": {

348 + "type": "object",

349 + "additionalProperties": false,

350 + "patternProperties": {

351 + "^x_": {}

352 + },

353 + "properties": {

354 + "title": {

355 + "$ref": "#/$defs/non_empty_string"

356 + },

357 + "short_name": {

358 + "$ref": "#/$defs/non_empty_string"

359 + },

360 + "icon": {

361 + "type": "string",

362 + "pattern": "^[a-zA-Z0-9_.-]+$"

363 + },

364 + "priority": {

365 + "type": "integer"

366 + },

367 + "families": {

368 + "$ref": "#/$defs/families"

369 + },

370 + "tooltip": {

371 + "$ref": "#/$defs/non_empty_string"

372 + },

373 + "menu_pattern": {

374 + "$ref": "#/$defs/non_empty_string"

375 + },

376 + "hide_sub_icon": {

377 + "type": "boolean"

378 + },

379 + "force_visibility": {

380 + "type": "boolean"

381 + },

382 + "fallback_icon": {

383 + "type": "string",

384 + "pattern": "^[a-zA-Z0-9_.-]+$"

385 + },

386 + "include_grand_parents": {

387 + "type": "boolean"

388 + },

389 + "properties": {

390 + "$ref": "#/$defs/properties"

391 + },

392 + "section_filters": {

393 + "type": "object"

394 + },

395 + "dyncfg": {

396 + "type": "object"

397 + },

398 + "contexts": {

399 + "$ref": "#/$defs/context_references"

400 + },

401 + "context_prefix": {

402 + "$ref": "#/$defs/string_list"

403 + },

404 + "context_prefix_exclude": {

405 + "$ref": "#/$defs/string_list"

406 + },

407 + "collect_plugin": {

408 + "$ref": "#/$defs/string_list"

409 + },

410 + "chart_library": {

411 + "$ref": "#/$defs/chart_library"

412 + },

413 + "group_by": {

414 + "$ref": "#/$defs/string_list"

415 + },

416 + "group_by_label": {

417 + "$ref": "#/$defs/string_list"

418 + },

419 + "aggregation_method": {

420 + "type": "string",

421 + "enum": [

422 + "avg",

423 + "max",

424 + "min",

425 + "sum"

426 + ]

427 + },

428 + "selected_dimensions": {

429 + "$ref": "#/$defs/string_list"

430 + },

431 + "dimensions_sort": {

432 + "$ref": "#/$defs/non_empty_string"

433 + },

434 + "colors": {

435 + "$ref": "#/$defs/string_list"

436 + },

437 + "layout": {

438 + "$ref": "#/$defs/layout"

439 + },

440 + "table_columns": {

441 + "$ref": "#/$defs/string_list"

442 + },

443 + "table_sort_by": {

444 + "type": "array",

445 + "items": {

446 + "$ref": "#/$defs/table_sort_item"

447 + },

448 + "minItems": 1

449 + },

450 + "labels": {

451 + "$ref": "#/$defs/labels"

452 + },

453 + "value_range": {

454 + "type": "array",

455 + "items": {

456 + "type": [

457 + "number",

458 + "null"

459 + ]

460 + },

461 + "minItems": 1

462 + },

463 + "eliminate_zero_dimensions": {

464 + "type": "boolean"

465 + },

466 + "context_items": {

467 + "type": "array",

468 + "items": {

469 + "$ref": "#/$defs/context_item"

470 + },

471 + "minItems": 1

472 + },

473 + "post_group_by": {

474 + "$ref": "#/$defs/string_list"

475 + },

476 + "show_post_aggregations": {

477 + "type": "boolean"

478 + },

479 + "grouping_method": {

480 + "$ref": "#/$defs/non_empty_string"

481 + },

482 + "sparkline": {

483 + "type": "boolean"

484 + },

485 + "renderer": {

486 + "$ref": "#/$defs/renderer"

487 + }

488 + }

489 + },

490 + "chart_library": {

491 + "type": "string",

492 + "enum": [

493 + "bars",

494 + "d3pie",

495 + "dygraph",

496 + "easypiechart",

497 + "gauge",

498 + "groupBoxes",

499 + "number",

500 + "table"

501 + ]

502 + },

503 + "placement": {

504 + "type": "object",

505 + "additionalProperties": false,

506 + "patternProperties": {

507 + "^x_": {}

508 + },

509 + "properties": {

510 + "id": {

511 + "$ref": "#/$defs/id"

512 + },

513 + "section_id": {

514 + "$ref": "#/$defs/id"

515 + },

516 + "title": {

517 + "$ref": "#/$defs/non_empty_string"

518 + },

519 + "short_name": {

520 + "$ref": "#/$defs/non_empty_string"

521 + },

522 + "icon": {

523 + "type": "string",

524 + "pattern": "^[a-zA-Z0-9_.-]+$"

525 + },

526 + "priority": {

527 + "type": "integer"

528 + },

529 + "families": {

530 + "$ref": "#/$defs/families"

531 + },

532 + "tooltip": {

533 + "$ref": "#/$defs/non_empty_string"

534 + },

535 + "menu_pattern": {

536 + "$ref": "#/$defs/non_empty_string"

537 + },

538 + "hide_sub_icon": {

539 + "type": "boolean"

540 + },

541 + "force_visibility": {

542 + "type": "boolean"

543 + },

544 + "fallback_icon": {

545 + "type": "string",

546 + "pattern": "^[a-zA-Z0-9_.-]+$"

547 + },

548 + "include_grand_parents": {

549 + "type": "boolean"

550 + },

551 + "properties": {

552 + "$ref": "#/$defs/properties"

553 + },

554 + "items": {

555 + "type": "array",

556 + "items": {

557 + "$ref": "#/$defs/structural_item"

558 + },

559 + "minItems": 1

560 + },

561 + "single_node": {

562 + "$ref": "#/$defs/single_node"

563 + }

564 + },

565 + "required": [

566 + "id",

567 + "section_id",

568 + "title",

569 + "items"

570 + ]

571 + },

572 + "structural_item": {

573 + "oneOf": [

574 + {

575 + "$ref": "#/$defs/non_empty_string"

576 + },

577 + {

578 + "$ref": "#/$defs/owned_context"

579 + },

580 + {

581 + "$ref": "#/$defs/group"

582 + },

583 + {

584 + "$ref": "#/$defs/flatten"

585 + },

586 + {

587 + "$ref": "#/$defs/selector"

588 + },

589 + {

590 + "$ref": "#/$defs/context_widget"

591 + },

592 + {

593 + "$ref": "#/$defs/grid"

594 + },

595 + {

596 + "$ref": "#/$defs/first_available"

597 + },

598 + {

599 + "$ref": "#/$defs/view_switch"

600 + }

601 + ]

602 + },

603 + "non_flatten_structural_item": {

604 + "oneOf": [

605 + {

606 + "$ref": "#/$defs/non_empty_string"

607 + },

608 + {

609 + "$ref": "#/$defs/owned_context"

610 + },

611 + {

612 + "$ref": "#/$defs/group"

613 + },

614 + {

615 + "$ref": "#/$defs/selector"

616 + },

617 + {

618 + "$ref": "#/$defs/context_widget"

619 + },

620 + {

621 + "$ref": "#/$defs/grid"

622 + },

623 + {

624 + "$ref": "#/$defs/first_available"

625 + },

626 + {

627 + "$ref": "#/$defs/view_switch"

628 + }

629 + ]

630 + },

631 + "grid_item": {

632 + "oneOf": [

633 + {

634 + "$ref": "#/$defs/context_widget"

635 + },

636 + {

637 + "$ref": "#/$defs/first_available"

638 + },

639 + {

640 + "$ref": "#/$defs/view_switch_display"

641 + }

642 + ]

643 + },

644 + "first_available_item": {

645 + "oneOf": [

646 + {

647 + "$ref": "#/$defs/context_widget"

648 + },

649 + {

650 + "$ref": "#/$defs/grid"

651 + },

652 + {

653 + "$ref": "#/$defs/view_switch_display"

654 + }

655 + ]

656 + },

657 + "view_switch_branch_item": {

658 + "oneOf": [

659 + {

660 + "$ref": "#/$defs/owned_context"

661 + },

662 + {

663 + "$ref": "#/$defs/group"

664 + },

665 + {

666 + "$ref": "#/$defs/selector"

667 + },

668 + {

669 + "$ref": "#/$defs/context_widget"

670 + },

671 + {

672 + "$ref": "#/$defs/grid"

673 + },

674 + {

675 + "$ref": "#/$defs/first_available"

676 + }

677 + ]

678 + },

679 + "view_switch_display_branch_item": {

680 + "oneOf": [

681 + {

682 + "$ref": "#/$defs/context_widget"

683 + },

684 + {

685 + "$ref": "#/$defs/grid"

686 + },

687 + {

688 + "$ref": "#/$defs/first_available"

689 + }

690 + ]

691 + },

692 + "owned_context": {

693 + "type": "object",

694 + "additionalProperties": false,

695 + "patternProperties": {

696 + "^x_": {}

697 + },

698 + "properties": {

699 + "type": {

700 + "const": "owned_context"

701 + },

702 + "context": {

703 + "$ref": "#/$defs/non_empty_string"

704 + },

705 + "title": {

706 + "$ref": "#/$defs/non_empty_string"

707 + },

708 + "short_name": {

709 + "$ref": "#/$defs/non_empty_string"

710 + },

711 + "icon": {

712 + "type": "string",

713 + "pattern": "^[a-zA-Z0-9_.-]+$"

714 + },

715 + "priority": {

716 + "type": "integer"

717 + },

718 + "families": {

719 + "$ref": "#/$defs/families"

720 + },

721 + "tooltip": {

722 + "$ref": "#/$defs/non_empty_string"

723 + },

724 + "menu_pattern": {

725 + "$ref": "#/$defs/non_empty_string"

726 + },

727 + "hide_sub_icon": {

728 + "type": "boolean"

729 + },

730 + "force_visibility": {

731 + "type": "boolean"

732 + },

733 + "fallback_icon": {

734 + "type": "string",

735 + "pattern": "^[a-zA-Z0-9_.-]+$"

736 + },

737 + "include_grand_parents": {

738 + "type": "boolean"

739 + },

740 + "properties": {

741 + "$ref": "#/$defs/properties"

742 + },

743 + "single_node": {

744 + "$ref": "#/$defs/single_node"

745 + }

746 + },

747 + "required": [

748 + "type",

749 + "context"

750 + ]

751 + },

752 + "group": {

753 + "type": "object",

754 + "additionalProperties": false,

755 + "patternProperties": {

756 + "^x_": {}

757 + },

758 + "properties": {

759 + "type": {

760 + "const": "group"

761 + },

762 + "id": {

763 + "$ref": "#/$defs/id"

764 + },

765 + "title": {

766 + "$ref": "#/$defs/non_empty_string"

767 + },

768 + "short_name": {

769 + "$ref": "#/$defs/non_empty_string"

770 + },

771 + "icon": {

772 + "type": "string",

773 + "pattern": "^[a-zA-Z0-9_.-]+$"

774 + },

775 + "priority": {

776 + "type": "integer"

777 + },

778 + "families": {

779 + "$ref": "#/$defs/families"

780 + },

781 + "tooltip": {

782 + "$ref": "#/$defs/non_empty_string"

783 + },

784 + "menu_pattern": {

785 + "$ref": "#/$defs/non_empty_string"

786 + },

787 + "hide_sub_icon": {

788 + "type": "boolean"

789 + },

790 + "force_visibility": {

791 + "type": "boolean"

792 + },

793 + "fallback_icon": {

794 + "type": "string",

795 + "pattern": "^[a-zA-Z0-9_.-]+$"

796 + },

797 + "include_grand_parents": {

798 + "type": "boolean"

799 + },

800 + "properties": {

801 + "$ref": "#/$defs/properties"

802 + },

803 + "section_filters": {

804 + "type": "object"

805 + },

806 + "dyncfg": {

807 + "type": "object"

808 + },

809 + "items": {

810 + "type": "array",

811 + "items": {

812 + "$ref": "#/$defs/structural_item"

813 + },

814 + "minItems": 1

815 + },

816 + "single_node": {

817 + "$ref": "#/$defs/single_node"

818 + }

819 + },

820 + "required": [

821 + "type",

822 + "id",

823 + "title",

824 + "items"

825 + ]

826 + },

827 + "flatten": {

828 + "type": "object",

829 + "additionalProperties": false,

830 + "patternProperties": {

831 + "^x_": {}

832 + },

833 + "properties": {

834 + "type": {

835 + "const": "flatten"

836 + },

837 + "id": {

838 + "$ref": "#/$defs/id"

839 + },

840 + "title": {

841 + "$ref": "#/$defs/non_empty_string"

842 + },

843 + "short_name": {

844 + "$ref": "#/$defs/non_empty_string"

845 + },

846 + "icon": {

847 + "type": "string",

848 + "pattern": "^[a-zA-Z0-9_.-]+$"

849 + },

850 + "priority": {

851 + "type": "integer"

852 + },

853 + "families": {

854 + "$ref": "#/$defs/families"

855 + },

856 + "tooltip": {

857 + "$ref": "#/$defs/non_empty_string"

858 + },

859 + "menu_pattern": {

860 + "$ref": "#/$defs/non_empty_string"

861 + },

862 + "hide_sub_icon": {

863 + "type": "boolean"

864 + },

865 + "force_visibility": {

866 + "type": "boolean"

867 + },

868 + "fallback_icon": {

869 + "type": "string",

870 + "pattern": "^[a-zA-Z0-9_.-]+$"

871 + },

872 + "include_grand_parents": {

873 + "type": "boolean"

874 + },

875 + "properties": {

876 + "$ref": "#/$defs/properties"

877 + },

878 + "items": {

879 + "type": "array",

880 + "items": {

881 + "$ref": "#/$defs/non_flatten_structural_item"

882 + },

883 + "minItems": 1

884 + },

885 + "single_node": {

886 + "$ref": "#/$defs/single_node"

887 + }

888 + },

889 + "required": [

890 + "type",

891 + "id",

892 + "title",

893 + "items"

894 + ]

895 + },

896 + "selector": {

897 + "type": "object",

898 + "additionalProperties": false,

899 + "patternProperties": {

900 + "^x_": {}

901 + },

902 + "properties": {

903 + "type": {

904 + "const": "selector"

905 + },

906 + "id": {

907 + "$ref": "#/$defs/id"

908 + },

909 + "title": {

910 + "$ref": "#/$defs/non_empty_string"

911 + },

912 + "short_name": {

913 + "$ref": "#/$defs/non_empty_string"

914 + },

915 + "icon": {

916 + "type": "string",

917 + "pattern": "^[a-zA-Z0-9_.-]+$"

918 + },

919 + "priority": {

920 + "type": "integer"

921 + },

922 + "families": {

923 + "$ref": "#/$defs/families"

924 + },

925 + "tooltip": {

926 + "$ref": "#/$defs/non_empty_string"

927 + },

928 + "menu_pattern": {

929 + "$ref": "#/$defs/non_empty_string"

930 + },

931 + "hide_sub_icon": {

932 + "type": "boolean"

933 + },

934 + "force_visibility": {

935 + "type": "boolean"

936 + },

937 + "fallback_icon": {

938 + "type": "string",

939 + "pattern": "^[a-zA-Z0-9_.-]+$"

940 + },

941 + "include_grand_parents": {

942 + "type": "boolean"

943 + },

944 + "context_prefix": {

945 + "$ref": "#/$defs/string_list"

946 + },

947 + "context_prefix_exclude": {

948 + "$ref": "#/$defs/string_list"

949 + },

950 + "collect_plugin": {

951 + "$ref": "#/$defs/string_list"

952 + },

953 + "single_node": {

954 + "$ref": "#/$defs/single_node"

955 + }

956 + },

957 + "required": [

958 + "type",

959 + "id",

960 + "title"

961 + ],

962 + "oneOf": [

963 + {

964 + "required": [

965 + "context_prefix"

966 + ]

967 + },

968 + {

969 + "required": [

970 + "collect_plugin"

971 + ]

972 + }

973 + ]

974 + },

975 + "context_widget": {

976 + "type": "object",

977 + "additionalProperties": false,

978 + "patternProperties": {

979 + "^x_": {}

980 + },

981 + "properties": {

982 + "type": {

983 + "const": "context"

984 + },

985 + "id": {

986 + "$ref": "#/$defs/id"

987 + },

988 + "title": {

989 + "$ref": "#/$defs/non_empty_string"

990 + },

991 + "short_name": {

992 + "$ref": "#/$defs/non_empty_string"

993 + },

994 + "icon": {

995 + "type": "string",

996 + "pattern": "^[a-zA-Z0-9_.-]+$"

997 + },

998 + "priority": {

999 + "type": "integer"

1000 + },

1001 + "families": {

1002 + "$ref": "#/$defs/families"

1003 + },

1004 + "tooltip": {

1005 + "$ref": "#/$defs/non_empty_string"

1006 + },

1007 + "menu_pattern": {

1008 + "$ref": "#/$defs/non_empty_string"

1009 + },

1010 + "hide_sub_icon": {

1011 + "type": "boolean"

1012 + },

1013 + "force_visibility": {

1014 + "type": "boolean"

1015 + },

1016 + "fallback_icon": {

1017 + "type": "string",

1018 + "pattern": "^[a-zA-Z0-9_.-]+$"

1019 + },

1020 + "include_grand_parents": {

1021 + "type": "boolean"

1022 + },

1023 + "properties": {

1024 + "$ref": "#/$defs/properties"

1025 + },

1026 + "contexts": {

1027 + "$ref": "#/$defs/context_references"

1028 + },

1029 + "chart_library": {

1030 + "$ref": "#/$defs/chart_library"

1031 + },

1032 + "group_by": {

1033 + "$ref": "#/$defs/string_list"

1034 + },

1035 + "group_by_label": {

1036 + "$ref": "#/$defs/string_list"

1037 + },

1038 + "aggregation_method": {

1039 + "type": "string",

1040 + "enum": [

1041 + "avg",

1042 + "max",

1043 + "min",

1044 + "sum"

1045 + ]

1046 + },

1047 + "selected_dimensions": {

1048 + "$ref": "#/$defs/string_list"

1049 + },

1050 + "dimensions_sort": {

1051 + "$ref": "#/$defs/non_empty_string"

1052 + },

1053 + "colors": {

1054 + "$ref": "#/$defs/string_list"

1055 + },

1056 + "layout": {

1057 + "$ref": "#/$defs/layout"

1058 + },

1059 + "table_columns": {

1060 + "$ref": "#/$defs/string_list"

1061 + },

1062 + "table_sort_by": {

1063 + "type": "array",

1064 + "items": {

1065 + "$ref": "#/$defs/table_sort_item"

1066 + },

1067 + "minItems": 1

1068 + },

1069 + "labels": {

1070 + "$ref": "#/$defs/labels"

1071 + },

1072 + "value_range": {

1073 + "type": "array",

1074 + "items": {

1075 + "type": [

1076 + "number",

1077 + "null"

1078 + ]

1079 + },

1080 + "minItems": 1

1081 + },

1082 + "eliminate_zero_dimensions": {

1083 + "type": "boolean"

1084 + },

1085 + "context_items": {

1086 + "type": "array",

1087 + "items": {

1088 + "$ref": "#/$defs/context_item"

1089 + },

1090 + "minItems": 1

1091 + },

1092 + "post_group_by": {

1093 + "$ref": "#/$defs/string_list"

1094 + },

1095 + "show_post_aggregations": {

1096 + "type": "boolean"

1097 + },

1098 + "grouping_method": {

1099 + "$ref": "#/$defs/non_empty_string"

1100 + },

1101 + "sparkline": {

1102 + "type": "boolean"

1103 + },

1104 + "renderer": {

1105 + "$ref": "#/$defs/renderer"

1106 + },

1107 + "single_node": {

1108 + "$ref": "#/$defs/single_node"

1109 + }

1110 + },

1111 + "required": [

1112 + "type",

1113 + "contexts",

1114 + "chart_library"

1115 + ]

1116 + },

1117 + "grid": {

1118 + "type": "object",

1119 + "additionalProperties": false,

1120 + "patternProperties": {

1121 + "^x_": {}

1122 + },

1123 + "properties": {

1124 + "type": {

1125 + "const": "grid"

1126 + },

1127 + "id": {

1128 + "$ref": "#/$defs/id"

1129 + },

1130 + "title": {

1131 + "$ref": "#/$defs/non_empty_string"

1132 + },

1133 + "short_name": {

1134 + "$ref": "#/$defs/non_empty_string"

1135 + },

1136 + "icon": {

1137 + "type": "string",

1138 + "pattern": "^[a-zA-Z0-9_.-]+$"

1139 + },

1140 + "priority": {

1141 + "type": "integer"

1142 + },

1143 + "families": {

1144 + "$ref": "#/$defs/families"

1145 + },

1146 + "tooltip": {

1147 + "$ref": "#/$defs/non_empty_string"

1148 + },

1149 + "menu_pattern": {

1150 + "$ref": "#/$defs/non_empty_string"

1151 + },

1152 + "hide_sub_icon": {

1153 + "type": "boolean"

1154 + },

1155 + "force_visibility": {

1156 + "type": "boolean"

1157 + },

1158 + "fallback_icon": {

1159 + "type": "string",

1160 + "pattern": "^[a-zA-Z0-9_.-]+$"

1161 + },

1162 + "include_grand_parents": {

1163 + "type": "boolean"

1164 + },

1165 + "renderer": {

1166 + "$ref": "#/$defs/renderer"

1167 + },

1168 + "items": {

1169 + "type": "array",

1170 + "items": {

1171 + "$ref": "#/$defs/grid_item"

1172 + },

1173 + "minItems": 1

1174 + },

1175 + "single_node": {

1176 + "$ref": "#/$defs/single_node"

1177 + }

1178 + },

1179 + "required": [

1180 + "type",

1181 + "id",

1182 + "items"

1183 + ]

1184 + },

1185 + "first_available": {

1186 + "type": "object",

1187 + "additionalProperties": false,

1188 + "patternProperties": {

1189 + "^x_": {}

1190 + },

1191 + "properties": {

1192 + "type": {

1193 + "const": "first_available"

1194 + },

1195 + "id": {

1196 + "$ref": "#/$defs/id"

1197 + },

1198 + "title": {

1199 + "$ref": "#/$defs/non_empty_string"

1200 + },

1201 + "short_name": {

1202 + "$ref": "#/$defs/non_empty_string"

1203 + },

1204 + "icon": {

1205 + "type": "string",

1206 + "pattern": "^[a-zA-Z0-9_.-]+$"

1207 + },

1208 + "priority": {

1209 + "type": "integer"

1210 + },

1211 + "families": {

1212 + "$ref": "#/$defs/families"

1213 + },

1214 + "tooltip": {

1215 + "$ref": "#/$defs/non_empty_string"

1216 + },

1217 + "menu_pattern": {

1218 + "$ref": "#/$defs/non_empty_string"

1219 + },

1220 + "hide_sub_icon": {

1221 + "type": "boolean"

1222 + },

1223 + "force_visibility": {

1224 + "type": "boolean"

1225 + },

1226 + "fallback_icon": {

1227 + "type": "string",

1228 + "pattern": "^[a-zA-Z0-9_.-]+$"

1229 + },

1230 + "include_grand_parents": {

1231 + "type": "boolean"

1232 + },

1233 + "items": {

1234 + "type": "array",

1235 + "items": {

1236 + "$ref": "#/$defs/first_available_item"

1237 + },

1238 + "minItems": 1

1239 + },

1240 + "single_node": {

1241 + "$ref": "#/$defs/single_node"

1242 + }

1243 + },

1244 + "required": [

1245 + "type",

1246 + "items"

1247 + ]

1248 + },

1249 + "view_switch": {

1250 + "type": "object",

1251 + "additionalProperties": false,

1252 + "patternProperties": {

1253 + "^x_": {}

1254 + },

1255 + "properties": {

1256 + "type": {

1257 + "const": "view_switch"

1258 + },

1259 + "id": {

1260 + "$ref": "#/$defs/id"

1261 + },

1262 + "multi_node": {

1263 + "$ref": "#/$defs/view_switch_branch_item"

1264 + },

1265 + "single_node": {

1266 + "$ref": "#/$defs/view_switch_branch_item"

1267 + }

1268 + },

1269 + "required": [

1270 + "type",

1271 + "multi_node",

1272 + "single_node"

1273 + ]

1274 + },

1275 + "view_switch_display": {

1276 + "type": "object",

1277 + "additionalProperties": false,

1278 + "patternProperties": {

1279 + "^x_": {}

1280 + },

1281 + "properties": {

1282 + "type": {

1283 + "const": "view_switch"

1284 + },

1285 + "id": {

1286 + "$ref": "#/$defs/id"

1287 + },

1288 + "multi_node": {

1289 + "$ref": "#/$defs/view_switch_display_branch_item"

1290 + },

1291 + "single_node": {

1292 + "$ref": "#/$defs/view_switch_display_branch_item"

1293 + }

1294 + },

1295 + "required": [

1296 + "type",

1297 + "multi_node",

1298 + "single_node"

1299 + ]

1300 + }

1301 + }

1302 +}

integrations/schemas/taxonomy_output.json new

+487

@@ -0,0 +1,487 @@

1 +{

2 + "$schema": "http://json-schema.org/draft-07/schema#",

3 + "type": "object",

4 + "title": "Generated Netdata collector taxonomy artifact.",

5 + "additionalProperties": false,

6 + "properties": {

7 + "taxonomy_schema_version": {

8 + "type": "integer",

9 + "const": 1

10 + },

11 + "source": {

12 + "type": "object",

13 + "additionalProperties": false,

14 + "properties": {

15 + "netdata_commit": {

16 + "type": "string",

17 + "minLength": 1

18 + },

19 + "generated_at": {

20 + "type": "string",

21 + "minLength": 1

22 + }

23 + },

24 + "required": [

25 + "netdata_commit",

26 + "generated_at"

27 + ]

28 + },

29 + "sections": {

30 + "type": "array",

31 + "items": {

32 + "$ref": "#/$defs/section"

33 + }

34 + },

35 + "placements": {

36 + "type": "array",

37 + "items": {

38 + "$ref": "#/$defs/placement"

39 + }

40 + },

41 + "opted_out_collectors": {

42 + "type": "array",

43 + "items": {

44 + "$ref": "#/$defs/optout"

45 + }

46 + }

47 + },

48 + "required": [

49 + "taxonomy_schema_version",

50 + "source",

51 + "sections",

52 + "placements",

53 + "opted_out_collectors"

54 + ],

55 + "$defs": {

56 + "section": {

57 + "type": "object",

58 + "additionalProperties": false,

59 + "properties": {

60 + "id": {

61 + "type": "string"

62 + },

63 + "parent_id": {

64 + "type": "string"

65 + },

66 + "path": {

67 + "type": "string"

68 + },

69 + "title": {

70 + "type": "string"

71 + },

72 + "short_name": {

73 + "type": "string"

74 + },

75 + "icon": {

76 + "type": "string"

77 + },

78 + "section_order": {

79 + "type": "integer"

80 + },

81 + "status": {

82 + "type": "string",

83 + "enum": [

84 + "active",

85 + "deprecated"

86 + ]

87 + },

88 + "deprecation": {

89 + "type": "object"

90 + },

91 + "_extra": {

92 + "type": "object"

93 + }

94 + },

95 + "required": [

96 + "id",

97 + "path",

98 + "title",

99 + "section_order",

100 + "status"

101 + ]

102 + },

103 + "snapshot": {

104 + "type": "array",

105 + "items": {

106 + "type": "string"

107 + },

108 + "uniqueItems": true

109 + },

110 + "unresolved_reference": {

111 + "type": "object",

112 + "additionalProperties": false,

113 + "properties": {

114 + "context": {

115 + "type": "string"

116 + },

117 + "reason": {

118 + "type": "string"

119 + },

120 + "owner": {

121 + "type": "string"

122 + },

123 + "expires": {

124 + "type": "string",

125 + "pattern": "^\\d{4}-\\d{2}-\\d{2}$"

126 + },

127 + "item_path": {

128 + "type": "string"

129 + }

130 + },

131 + "required": [

132 + "context",

133 + "reason",

134 + "owner",

135 + "expires",

136 + "item_path"

137 + ]

138 + },

139 + "unresolved_references": {

140 + "type": "array",

141 + "items": {

142 + "$ref": "#/$defs/unresolved_reference"

143 + }

144 + },

145 + "node": {

146 + "type": "object",

147 + "additionalProperties": false,

148 + "properties": {

149 + "type": {

150 + "type": "string",

151 + "enum": [

152 + "owned_context",

153 + "group",

154 + "flatten",

155 + "selector",

156 + "context",

157 + "grid",

158 + "first_available",

159 + "view_switch"

160 + ]

161 + },

162 + "id": {

163 + "type": "string"

164 + },

165 + "title": {

166 + "type": "string"

167 + },

168 + "short_name": {

169 + "type": "string"

170 + },

171 + "icon": {

172 + "type": "string"

173 + },

174 + "priority": {

175 + "type": "integer"

176 + },

177 + "families": {},

178 + "tooltip": {

179 + "type": "string"

180 + },

181 + "menu_pattern": {

182 + "type": "string"

183 + },

184 + "hide_sub_icon": {

185 + "type": "boolean"

186 + },

187 + "force_visibility": {

188 + "type": "boolean"

189 + },

190 + "fallback_icon": {

191 + "type": "string"

192 + },

193 + "include_grand_parents": {

194 + "type": "boolean"

195 + },

196 + "properties": {

197 + "type": "object"

198 + },

199 + "section_filters": {

200 + "type": "object"

201 + },

202 + "dyncfg": {

203 + "type": "object"

204 + },

205 + "context": {

206 + "type": "string"

207 + },

208 + "contexts": {

209 + "type": "array",

210 + "items": {}

211 + },

212 + "context_prefix": {

213 + "type": "array",

214 + "items": {

215 + "type": "string"

216 + }

217 + },

218 + "context_prefix_exclude": {

219 + "type": "array",

220 + "items": {

221 + "type": "string"

222 + }

223 + },

224 + "collect_plugin": {

225 + "type": "array",

226 + "items": {

227 + "type": "string"

228 + }

229 + },

230 + "chart_library": {

231 + "type": "string"

232 + },

233 + "group_by": {

234 + "type": "array",

235 + "items": {

236 + "type": "string"

237 + }

238 + },

239 + "group_by_label": {

240 + "type": "array",

241 + "items": {

242 + "type": "string"

243 + }

244 + },

245 + "aggregation_method": {

246 + "type": "string"

247 + },

248 + "selected_dimensions": {

249 + "type": "array",

250 + "items": {

251 + "type": "string"

252 + }

253 + },

254 + "dimensions_sort": {

255 + "type": "string"

256 + },

257 + "colors": {

258 + "type": "array",

259 + "items": {

260 + "type": "string"

261 + }

262 + },

263 + "layout": {

264 + "type": "object"

265 + },

266 + "table_columns": {

267 + "type": "array",

268 + "items": {

269 + "type": "string"

270 + }

271 + },

272 + "table_sort_by": {

273 + "type": "array",

274 + "items": {

275 + "type": "object"

276 + }

277 + },

278 + "labels": {

279 + "type": "object"

280 + },

281 + "value_range": {

282 + "type": "array"

283 + },

284 + "eliminate_zero_dimensions": {

285 + "type": "boolean"

286 + },

287 + "context_items": {

288 + "type": "array",

289 + "items": {

290 + "type": "object"

291 + }

292 + },

293 + "post_group_by": {

294 + "type": "array",

295 + "items": {

296 + "type": "string"

297 + }

298 + },

299 + "show_post_aggregations": {

300 + "type": "boolean"

301 + },

302 + "grouping_method": {

303 + "type": "string"

304 + },

305 + "sparkline": {

306 + "type": "boolean"

307 + },

308 + "renderer": {

309 + "type": "object"

310 + },

311 + "single_node": {

312 + "anyOf": [

313 + {

314 + "type": "object"

315 + },

316 + {

317 + "$ref": "#/$defs/node"

318 + }

319 + ]

320 + },

321 + "multi_node": {

322 + "$ref": "#/$defs/node"

323 + },

324 + "items": {

325 + "type": "array",

326 + "items": {

327 + "$ref": "#/$defs/node"

328 + }

329 + },

330 + "resolved_contexts": {

331 + "$ref": "#/$defs/snapshot",

332 + "description": "Literal metric contexts owned by this node after selector expansion and child aggregation. FE consumers use this as the node's owned/renderable context snapshot."

333 + },

334 + "referenced_contexts": {

335 + "$ref": "#/$defs/snapshot",

336 + "description": "Metric contexts referenced by display widgets under this node but owned elsewhere in the structural tree. FE consumers use this for widget context wiring, not ownership."

337 + },

338 + "unresolved_references": {

339 + "$ref": "#/$defs/unresolved_references",

340 + "description": "Explicit staged widget references that do not resolve yet, with owner/reason/expiry metadata preserved for downstream consumers."

341 + },

342 + "_extra": {

343 + "type": "object"

344 + }

345 + },

346 + "required": [

347 + "type",

348 + "resolved_contexts",

349 + "referenced_contexts",

350 + "unresolved_references"

351 + ]

352 + },

353 + "placement": {

354 + "type": "object",

355 + "additionalProperties": false,

356 + "properties": {

357 + "collector_ids": {

358 + "type": "array",

359 + "items": {

360 + "type": "string"

361 + }

362 + },

363 + "plugin_name": {

364 + "type": "string"

365 + },

366 + "module_name": {

367 + "type": "string"

368 + },

369 + "source_path": {

370 + "type": "string"

371 + },

372 + "id": {

373 + "type": "string"

374 + },

375 + "section_id": {

376 + "type": "string"

377 + },

378 + "section_path": {

379 + "type": "string"

380 + },

381 + "title": {

382 + "type": "string"

383 + },

384 + "short_name": {

385 + "type": "string"

386 + },

387 + "icon": {

388 + "type": "string"

389 + },

390 + "priority": {

391 + "type": "integer"

392 + },

393 + "families": {},

394 + "tooltip": {

395 + "type": "string"

396 + },

397 + "menu_pattern": {

398 + "type": "string"

399 + },

400 + "hide_sub_icon": {

401 + "type": "boolean"

402 + },

403 + "force_visibility": {

404 + "type": "boolean"

405 + },

406 + "fallback_icon": {

407 + "type": "string"

408 + },

409 + "include_grand_parents": {

410 + "type": "boolean"

411 + },

412 + "properties": {

413 + "type": "object"

414 + },

415 + "items": {

416 + "type": "array",

417 + "items": {

418 + "$ref": "#/$defs/node"

419 + }

420 + },

421 + "resolved_contexts": {

422 + "$ref": "#/$defs/snapshot",

423 + "description": "Literal metric contexts owned by this placement after selector expansion and item aggregation. FE consumers use this as the placement's owned/renderable context snapshot."

424 + },

425 + "referenced_contexts": {

426 + "$ref": "#/$defs/snapshot",

427 + "description": "Metric contexts referenced by display widgets under this placement but owned elsewhere in the structural tree. FE consumers use this for widget context wiring, not ownership."

428 + },

429 + "unresolved_references": {

430 + "$ref": "#/$defs/unresolved_references",

431 + "description": "Explicit staged widget references in this placement that do not resolve yet, with owner/reason/expiry metadata preserved for downstream consumers."

432 + },

433 + "single_node": {

434 + "type": "object"

435 + },

436 + "_extra": {

437 + "type": "object"

438 + }

439 + },

440 + "required": [

441 + "collector_ids",

442 + "plugin_name",

443 + "module_name",

444 + "source_path",

445 + "id",

446 + "section_id",

447 + "section_path",

448 + "title",

449 + "items",

450 + "resolved_contexts",

451 + "referenced_contexts",

452 + "unresolved_references"

453 + ]

454 + },

455 + "optout": {

456 + "type": "object",

457 + "additionalProperties": false,

458 + "properties": {

459 + "collector_ids": {

460 + "type": "array",

461 + "items": {

462 + "type": "string"

463 + }

464 + },

465 + "plugin_name": {

466 + "type": "string"

467 + },

468 + "module_name": {

469 + "type": "string"

470 + },

471 + "source_path": {

472 + "type": "string"

473 + },

474 + "reason": {

475 + "type": "string"

476 + }

477 + },

478 + "required": [

479 + "collector_ids",

480 + "plugin_name",

481 + "module_name",

482 + "source_path",

483 + "reason"

484 + ]

485 + }

486 + }

487 +}

integrations/schemas/taxonomy_sections.json new

+94

@@ -0,0 +1,94 @@

1 +{

2 + "$schema": "http://json-schema.org/draft-07/schema#",

3 + "type": "object",

4 + "title": "Netdata collector taxonomy section registry.",

5 + "additionalProperties": false,

6 + "patternProperties": {

7 + "^x_": {}

8 + },

9 + "properties": {

10 + "taxonomy_version": {

11 + "type": "integer",

12 + "const": 1

13 + },

14 + "sections": {

15 + "type": "array",

16 + "items": {

17 + "$ref": "#/$defs/section"

18 + },

19 + "minItems": 1

20 + }

21 + },

22 + "required": [

23 + "taxonomy_version",

24 + "sections"

25 + ],

26 + "$defs": {

27 + "section": {

28 + "type": "object",

29 + "additionalProperties": false,

30 + "patternProperties": {

31 + "^x_": {}

32 + },

33 + "properties": {

34 + "id": {

35 + "type": "string",

36 + "pattern": "^[a-z0-9][a-z0-9_.-]*$"

37 + },

38 + "parent_id": {

39 + "type": "string",

40 + "pattern": "^[a-z0-9][a-z0-9_.-]*$"

41 + },

42 + "title": {

43 + "type": "string",

44 + "minLength": 1

45 + },

46 + "short_name": {

47 + "type": "string",

48 + "minLength": 1

49 + },

50 + "icon": {

51 + "type": "string",

52 + "pattern": "^[a-zA-Z0-9_.-]+$"

53 + },

54 + "section_order": {

55 + "type": "integer"

56 + },

57 + "status": {

58 + "type": "string",

59 + "enum": [

60 + "active",

61 + "deprecated"

62 + ]

63 + },

64 + "deprecation": {

65 + "type": "object",

66 + "additionalProperties": false,

67 + "properties": {

68 + "replacement_id": {

69 + "type": "string",

70 + "pattern": "^[a-z0-9][a-z0-9_.-]*$"

71 + },

72 + "since": {

73 + "type": "string",

74 + "minLength": 1

75 + },

76 + "removal_in": {

77 + "type": "string",

78 + "minLength": 1

79 + }

80 + },

81 + "required": [

82 + "since"

83 + ]

84 + }

85 + },

86 + "required": [

87 + "id",

88 + "title",

89 + "section_order",

90 + "status"

91 + ]

92 + }

93 + }

94 +}

integrations/taxonomy/icons.yaml new

+17

@@ -0,0 +1,17 @@

1 +taxonomy_version: 1

2 +icons:

3 + - apache

4 + - apps

5 + - azure

6 + - cgroup

7 + - compute

8 + - hardware

9 + - k8s

10 + - mysql

11 + - netdata

12 + - none

13 + - otel

14 + - postgres

15 + - snmp

16 + - synthetics

17 + - system

integrations/taxonomy/sections.yaml new

+104

@@ -0,0 +1,104 @@

1 +taxonomy_version: 1

2 +sections:

3 + - id: system

4 + title: System

5 + icon: system

6 + section_order: 10

7 + status: active

8 +

9 + - id: kubernetes

10 + title: Kubernetes

11 + icon: k8s

12 + section_order: 20

13 + status: active

14 +

15 + - id: containers-vms

16 + title: Containers and VMs

17 + icon: cgroup

18 + section_order: 30

19 + status: active

20 +

21 + - id: synthetic-checks

22 + title: Synthetic Checks

23 + icon: synthetics

24 + section_order: 40

25 + status: active

26 +

27 + - id: remote-devices

28 + title: Remote Devices

29 + icon: snmp

30 + section_order: 50

31 + status: active

32 +

33 + - id: otel

34 + title: OpenTelemetry

35 + icon: otel

36 + section_order: 60

37 + status: active

38 +

39 + - id: azure-monitor

40 + title: Azure Monitor

41 + icon: azure

42 + section_order: 70

43 + status: active

44 +

45 + - id: applications

46 + title: Applications

47 + icon: apps

48 + section_order: 80

49 + status: active

50 +

51 + - id: netdata

52 + title: Netdata

53 + icon: netdata

54 + section_order: 90

55 + status: active

56 +

57 + - id: applications.apache

58 + parent_id: applications

59 + title: Apache

60 + icon: apache

61 + section_order: 100

62 + status: active

63 +

64 + - id: applications.mysql

65 + parent_id: applications

66 + title: MySQL

67 + icon: mysql

68 + section_order: 110

69 + status: active

70 +

71 + - id: applications.postgres

72 + parent_id: applications

73 + title: PostgreSQL

74 + icon: postgres

75 + section_order: 120

76 + status: active

77 +

78 + - id: system.hardware

79 + parent_id: system

80 + title: Hardware

81 + icon: hardware

82 + section_order: 130

83 + status: active

84 +

85 + - id: system.hardware.gpus

86 + parent_id: system.hardware

87 + title: GPUs

88 + icon: compute

89 + section_order: 140

90 + status: active

91 +

92 + - id: system.hardware.gpus.nvidia

93 + parent_id: system.hardware.gpus

94 + title: NVIDIA GPUs

95 + icon: compute

96 + section_order: 150

97 + status: active

98 +

99 + - id: remote-devices.snmp

100 + parent_id: remote-devices

101 + title: SNMP-enabled Devices

102 + icon: snmp

103 + section_order: 160

104 + status: active

integrations/tests/__init__.py new

@@ -0,0 +1 @@

1 +"""Tests for integrations tooling."""

integrations/tests/test_taxonomy.py new

+647

@@ -0,0 +1,647 @@

1 +#!/usr/bin/env python3

2 +

3 +import sys

4 +import tempfile

5 +import unittest

6 +import json

7 +from pathlib import Path

8 +from unittest.mock import patch

9 +

10 +INTEGRATIONS_DIR = Path(__file__).resolve().parents[1]

11 +sys.path.insert(0, str(INTEGRATIONS_DIR))

12 +

13 +import check_collector_taxonomy

14 +import gen_taxonomy

15 +

16 +

17 +class TaxonomySchemaTest(unittest.TestCase):

18 + def valid_taxonomy(self):

19 + return {

20 + 'taxonomy_version': 1,

21 + 'plugin_name': 'go.d.plugin',

22 + 'module_name': 'apache',

23 + 'placements': [

24 + {

25 + 'id': 'apache',

26 + 'section_id': 'applications.apache',

27 + 'title': 'Apache',

28 + 'items': ['apache.connections'],

29 + },

30 + ],

31 + }

32 +

33 + def test_valid_authoring_schema(self):

34 + errors = list(gen_taxonomy.COLLECTOR_TAXONOMY_VALIDATOR.iter_errors(self.valid_taxonomy()))

35 + self.assertEqual(errors, [])

36 +

37 + def test_section_path_authoring_is_rejected(self):

38 + data = self.valid_taxonomy()

39 + data['placements'][0]['section_path'] = ['applications', 'apache']

40 + errors = list(gen_taxonomy.COLLECTOR_TAXONOMY_VALIDATOR.iter_errors(data))

41 + self.assertTrue(errors)

42 +

43 + def test_old_contexts_authoring_is_rejected(self):

44 + data = self.valid_taxonomy()

45 + data['placements'][0]['contexts'] = data['placements'][0].pop('items')

46 + errors = list(gen_taxonomy.COLLECTOR_TAXONOMY_VALIDATOR.iter_errors(data))

47 + self.assertTrue(errors)

48 +

49 + def test_grid_rejects_string_shorthand(self):

50 + data = self.valid_taxonomy()

51 + data['placements'][0]['items'] = [

52 + {

53 + 'type': 'grid',

54 + 'id': 'apache-heads',

55 + 'items': ['apache.connections'],

56 + },

57 + ]

58 + errors = list(gen_taxonomy.COLLECTOR_TAXONOMY_VALIDATOR.iter_errors(data))

59 + self.assertTrue(errors)

60 +

61 + def assert_schema_accepts_item(self, item):

62 + data = self.valid_taxonomy()

63 + data['placements'][0]['items'] = [item]

64 + errors = list(gen_taxonomy.COLLECTOR_TAXONOMY_VALIDATOR.iter_errors(data))

65 + self.assertEqual(errors, [])

66 +

67 + def assert_schema_rejects_item(self, item):

68 + data = self.valid_taxonomy()

69 + data['placements'][0]['items'] = [item]

70 + errors = list(gen_taxonomy.COLLECTOR_TAXONOMY_VALIDATOR.iter_errors(data))

71 + self.assertTrue(errors)

72 +

73 + def test_explicit_owned_context_is_accepted(self):

74 + self.assert_schema_accepts_item({

75 + 'type': 'owned_context',

76 + 'context': 'apache.connections',

77 + })

78 +

79 + def test_group_is_accepted(self):

80 + self.assert_schema_accepts_item({

81 + 'type': 'group',

82 + 'id': 'requests',

83 + 'title': 'Requests',

84 + 'items': ['apache.requests'],

85 + })

86 +

87 + def test_flatten_is_accepted(self):

88 + self.assert_schema_accepts_item({

89 + 'type': 'flatten',

90 + 'id': 'apache-flat',

91 + 'title': 'Apache',

92 + 'items': ['apache.connections'],

93 + })

94 +

95 + def test_first_available_is_accepted(self):

96 + self.assert_schema_accepts_item({

97 + 'type': 'first_available',

98 + 'items': [

99 + {

100 + 'type': 'context',

101 + 'contexts': ['apache.requests'],

102 + 'chart_library': 'number',

103 + },

104 + ],

105 + })

106 +

107 + def test_view_switch_is_accepted(self):

108 + self.assert_schema_accepts_item({

109 + 'type': 'view_switch',

110 + 'multi_node': {

111 + 'type': 'context',

112 + 'contexts': ['apache.requests'],

113 + 'chart_library': 'bars',

114 + },

115 + 'single_node': {

116 + 'type': 'context',

117 + 'contexts': ['apache.requests'],

118 + 'chart_library': 'dygraph',

119 + },

120 + })

121 +

122 + def test_grid_rejects_owned_context(self):

123 + self.assert_schema_rejects_item({

124 + 'type': 'grid',

125 + 'id': 'apache-heads',

126 + 'items': [

127 + {

128 + 'type': 'owned_context',

129 + 'context': 'apache.connections',

130 + },

131 + ],

132 + })

133 +

134 + def test_grid_rejects_selector(self):

135 + self.assert_schema_rejects_item({

136 + 'type': 'grid',

137 + 'id': 'apache-heads',

138 + 'items': [

139 + {

140 + 'type': 'selector',

141 + 'id': 'apache-prefix',

142 + 'title': 'Apache prefix',

143 + 'context_prefix': ['apache.'],

144 + },

145 + ],

146 + })

147 +

148 + def test_flatten_rejects_nested_flatten(self):

149 + self.assert_schema_rejects_item({

150 + 'type': 'flatten',

151 + 'id': 'outer',

152 + 'title': 'Outer',

153 + 'items': [

154 + {

155 + 'type': 'flatten',

156 + 'id': 'inner',

157 + 'title': 'Inner',

158 + 'items': ['apache.connections'],

159 + },

160 + ],

161 + })

162 +

163 + def test_first_available_rejects_string_shorthand(self):

164 + self.assert_schema_rejects_item({

165 + 'type': 'first_available',

166 + 'items': ['apache.connections'],

167 + })

168 +

169 + def test_view_switch_rejects_string_branch(self):

170 + self.assert_schema_rejects_item({

171 + 'type': 'view_switch',

172 + 'multi_node': 'apache.connections',

173 + 'single_node': {

174 + 'type': 'context',

175 + 'contexts': ['apache.connections'],

176 + 'chart_library': 'number',

177 + },

178 + })

179 +

180 + def test_view_switch_rejects_flatten_branch(self):

181 + self.assert_schema_rejects_item({

182 + 'type': 'view_switch',

183 + 'multi_node': {

184 + 'type': 'flatten',

185 + 'id': 'flat',

186 + 'title': 'Flat',

187 + 'items': ['apache.connections'],

188 + },

189 + 'single_node': {

190 + 'type': 'context',

191 + 'contexts': ['apache.connections'],

192 + 'chart_library': 'number',

193 + },

194 + })

195 +

196 + def test_view_switch_rejects_nested_view_switch(self):

197 + self.assert_schema_rejects_item({

198 + 'type': 'view_switch',

199 + 'multi_node': {

200 + 'type': 'view_switch',

201 + 'multi_node': {

202 + 'type': 'context',

203 + 'contexts': ['apache.connections'],

204 + 'chart_library': 'number',

205 + },

206 + 'single_node': {

207 + 'type': 'context',

208 + 'contexts': ['apache.connections'],

209 + 'chart_library': 'number',

210 + },

211 + },

212 + 'single_node': {

213 + 'type': 'context',

214 + 'contexts': ['apache.connections'],

215 + 'chart_library': 'number',

216 + },

217 + })

218 +

219 + def test_renderer_allows_x_extension(self):

220 + self.assert_schema_accepts_item({

221 + 'type': 'context',

222 + 'contexts': ['apache.connections'],

223 + 'chart_library': 'number',

224 + 'renderer': {

225 + 'x_future_renderer_option': True,

226 + },

227 + })

228 +

229 + def test_renderer_rejects_unknown_non_extension_key(self):

230 + self.assert_schema_rejects_item({

231 + 'type': 'context',

232 + 'contexts': ['apache.connections'],

233 + 'chart_library': 'number',

234 + 'renderer': {

235 + 'latetValue': 5,

236 + },

237 + })

238 +

239 + def test_renderer_fields_are_rejected_as_item_body_siblings(self):

240 + self.assert_schema_rejects_item({

241 + 'type': 'context',

242 + 'contexts': ['apache.connections'],

243 + 'chart_library': 'number',

244 + 'toolbox_elements': [],

245 + })

246 +

247 + def test_prescan_rejects_multi_node(self):

248 + findings = []

249 + gen_taxonomy.prescan_removed_shapes({'multi_node': {'title': 'Bad'}}, Path('taxonomy.yaml'), findings)

250 + self.assertEqual([finding.code for finding in findings], ['TAX022'])

251 +

252 + def test_optout_does_not_require_metadata(self):

253 + text = """taxonomy_version: 1

254 +plugin_name: statsd.plugin

255 +module_name: statsd

256 +taxonomy_optout:

257 + reason: Operator-defined statsd synthetic charts have no static collector taxonomy.

258 +"""

259 + with tempfile.TemporaryDirectory() as tmp:

260 + path = Path(tmp) / 'taxonomy.yaml'

261 + path.write_text(text)

262 + findings = []

263 + placements, optouts = gen_taxonomy.process_taxonomy_file(

264 + path,

265 + sections={},

266 + icons=set(),

267 + metadata_indexes={

268 + 'by_path_module': {},

269 + 'all_contexts': [],

270 + 'contexts_by_plugin': {},

271 + },

272 + ownership={},

273 + findings=findings,

274 + )

275 + self.assertEqual(placements, [])

276 + self.assertEqual(optouts[0]['collector_ids'], ['statsd.plugin-statsd'])

277 + self.assertEqual([finding.code for finding in findings], [])

278 +

279 +

280 +class TaxonomyResolverTest(unittest.TestCase):

281 + def test_path_segment_uses_last_id_component(self):

282 + self.assertEqual(gen_taxonomy.path_segment({'id': 'applications.postgres'}), 'postgres')

283 +

284 + def test_resolve_prefix_uses_sorted_contexts(self):

285 + contexts = ['apache.requests', 'snmp.ifaces.in', 'snmp.ifaces.out', 'zfs.pool']

286 + self.assertEqual(

287 + gen_taxonomy.resolve_prefix('snmp.', contexts),

288 + ['snmp.ifaces.in', 'snmp.ifaces.out'],

289 + )

290 +

291 + def test_context_prefix_can_narrow_declared_dynamic_namespace(self):

292 + findings = []

293 + contexts = ['snmp.device_prof_ifTraffic', 'snmp.license.state']

294 + resolved = gen_taxonomy.resolve_node_contexts(

295 + {'context_prefix': ['snmp.device_prof_']},

296 + known_contexts=set(),

297 + allowed_prefixes={'snmp.'},

298 + allowed_plugins=set(),

299 + metadata_indexes={'all_contexts': contexts, 'contexts_by_plugin': {}},

300 + path=Path('taxonomy.yaml'),

301 + findings=findings,

302 + )

303 + self.assertEqual(resolved, ['snmp.device_prof_ifTraffic'])

304 + self.assertEqual([finding.code for finding in findings], [])

305 +

306 + def test_context_prefix_exclude_requires_prefix(self):

307 + findings = []

308 + gen_taxonomy.resolve_node_contexts(

309 + {'context_prefix_exclude': ['snmp.license.']},

310 + known_contexts=set(),

311 + allowed_prefixes=set(),

312 + allowed_plugins=set(),

313 + metadata_indexes={'all_contexts': [], 'contexts_by_plugin': {}},

314 + path=Path('taxonomy.yaml'),

315 + findings=findings,

316 + )

317 + self.assertEqual([finding.code for finding in findings], ['TAX029'])

318 +

319 + def test_metadata_loader_warnings_are_taxonomy_findings(self):

320 + original_len = len(gen_taxonomy.WARNINGS)

321 +

322 + def fake_load_collectors():

323 + gen_taxonomy.WARNINGS.append(('metadata.yaml', 'invalid metadata'))

324 + return []

325 +

326 + try:

327 + findings = []

328 + with patch.object(gen_taxonomy, 'load_collectors', side_effect=fake_load_collectors):

329 + indexes = gen_taxonomy.build_metadata_indexes(findings)

330 + self.assertEqual(indexes['modules'], [])

331 + self.assertEqual([finding.code for finding in findings], ['TAX001'])

332 + self.assertEqual(findings[0].message, 'invalid metadata')

333 + finally:

334 + del gen_taxonomy.WARNINGS[original_len:]

335 +

336 +

337 +class TaxonomyOwnershipTest(unittest.TestCase):

338 + def metadata_indexes(self, tmp, context='apache.connections', contexts=None, dynamic_prefixes=None):

339 + metadata_path = Path(tmp) / 'metadata.yaml'

340 + if contexts is None:

341 + contexts = [context]

342 + metrics = {

343 + 'scopes': [

344 + {

345 + 'metrics': [

346 + {'name': name}

347 + for name in contexts

348 + ],

349 + },

350 + ],

351 + }

352 + if dynamic_prefixes:

353 + metrics['dynamic_context_prefixes'] = [

354 + {

355 + 'prefix': prefix,

356 + 'reason': 'test dynamic contexts',

357 + }

358 + for prefix in dynamic_prefixes

359 + ]

360 + module = {

361 + '_src_path': str(metadata_path),

362 + 'meta': {

363 + 'plugin_name': 'go.d.plugin',

364 + 'module_name': 'apache',

365 + },

366 + 'metrics': metrics,

367 + }

368 + return {

369 + 'by_path_module': {

370 + (metadata_path, 'go.d.plugin', 'apache'): [module],

371 + },

372 + 'all_contexts': sorted(contexts),

373 + 'contexts_by_plugin': {},

374 + }

375 +

376 + def write_taxonomy(self, tmp, item):

377 + path = Path(tmp) / 'taxonomy.yaml'

378 + path.write_text("""taxonomy_version: 1

379 +plugin_name: go.d.plugin

380 +module_name: apache

381 +placements:

382 + - id: apache

383 + section_id: applications.apache

384 + title: Apache

385 + items:

386 +""")

387 + with path.open('a') as fp:

388 + fp.write(item)

389 + return path

390 +

391 + def test_referenced_context_without_owner_is_fatal(self):

392 + with tempfile.TemporaryDirectory() as tmp:

393 + path = self.write_taxonomy(tmp, """ - type: context

394 + contexts: [apache.connections]

395 + chart_library: number

396 +""")

397 + ownership = {}

398 + referenced_literals = []

399 + findings = []

400 + gen_taxonomy.process_taxonomy_file(

401 + path,

402 + sections={'applications.apache': ({'status': 'active', 'section_order': 1}, 'applications.apache')},

403 + icons=set(),

404 + metadata_indexes=self.metadata_indexes(tmp),

405 + ownership=ownership,

406 + findings=findings,

407 + referenced_literals=referenced_literals,

408 + )

409 + gen_taxonomy.emit_referenced_only_findings(referenced_literals, ownership, findings)

410 + self.assertEqual([finding.code for finding in findings], ['TAX037'])

411 +

412 + def test_referenced_context_owned_elsewhere_is_allowed(self):

413 + with tempfile.TemporaryDirectory() as tmp:

414 + path = self.write_taxonomy(tmp, """ - apache.connections

415 + - type: context

416 + contexts: [apache.connections]

417 + chart_library: number

418 +""")

419 + ownership = {}

420 + referenced_literals = []

421 + findings = []

422 + gen_taxonomy.process_taxonomy_file(

423 + path,

424 + sections={'applications.apache': ({'status': 'active', 'section_order': 1}, 'applications.apache')},

425 + icons=set(),

426 + metadata_indexes=self.metadata_indexes(tmp),

427 + ownership=ownership,

428 + findings=findings,

429 + referenced_literals=referenced_literals,

430 + )

431 + gen_taxonomy.emit_referenced_only_findings(referenced_literals, ownership, findings)

432 + self.assertEqual([finding.code for finding in findings], [])

433 +

434 + def test_unknown_literal_context_is_tax003(self):

435 + with tempfile.TemporaryDirectory() as tmp:

436 + path = self.write_taxonomy(tmp, """ - apache.unknown

437 +""")

438 + findings = []

439 + gen_taxonomy.process_taxonomy_file(

440 + path,

441 + sections={'applications.apache': ({'status': 'active', 'section_order': 1}, 'applications.apache')},

442 + icons=set(),

443 + metadata_indexes=self.metadata_indexes(tmp),

444 + ownership={},

445 + findings=findings,

446 + )

447 + self.assertEqual([finding.code for finding in findings], ['TAX003'])

448 +

449 + def test_selector_overlap_uses_tax036(self):

450 + with tempfile.TemporaryDirectory() as tmp:

451 + path = self.write_taxonomy(tmp, """ - apache.connections

452 + - type: selector

453 + id: apache-prefix

454 + title: Apache prefix

455 + context_prefix: [apache.]

456 +""")

457 + ownership = {}

458 + ownership_conflicts = {}

459 + referenced_literals = []

460 + findings = []

461 + gen_taxonomy.process_taxonomy_file(

462 + path,

463 + sections={'applications.apache': ({'status': 'active', 'section_order': 1}, 'applications.apache')},

464 + icons=set(),

465 + metadata_indexes=self.metadata_indexes(tmp, dynamic_prefixes=['apache.']),

466 + ownership=ownership,

467 + findings=findings,

468 + referenced_literals=referenced_literals,

469 + ownership_conflicts=ownership_conflicts,

470 + )

471 + gen_taxonomy.emit_ownership_conflicts(ownership_conflicts, findings)

472 + self.assertEqual([finding.code for finding in findings], ['TAX036'])

473 +

474 + def test_duplicate_literal_ownership_uses_tax033_once(self):

475 + with tempfile.TemporaryDirectory() as tmp:

476 + path = self.write_taxonomy(tmp, """ - apache.connections

477 + - type: group

478 + id: duplicate

479 + title: Duplicate

480 + items:

481 + - apache.connections

482 +""")

483 + ownership = {}

484 + ownership_conflicts = {}

485 + findings = []

486 + gen_taxonomy.process_taxonomy_file(

487 + path,

488 + sections={'applications.apache': ({'status': 'active', 'section_order': 1}, 'applications.apache')},

489 + icons=set(),

490 + metadata_indexes=self.metadata_indexes(tmp),

491 + ownership=ownership,

492 + findings=findings,

493 + ownership_conflicts=ownership_conflicts,

494 + )

495 + gen_taxonomy.emit_ownership_conflicts(ownership_conflicts, findings)

496 + self.assertEqual([finding.code for finding in findings], ['TAX033'])

497 +

498 + def test_stale_unresolved_reference_warns(self):

499 + with tempfile.TemporaryDirectory() as tmp:

500 + path = self.write_taxonomy(tmp, """ - type: context

501 + contexts:

502 + - context: apache.connections

503 + unresolved:

504 + reason: staged rename

505 + owner: cloud-frontend

506 + expires: "2026-08-01"

507 + chart_library: number

508 +""")

509 + ownership = {}

510 + referenced_literals = []

511 + findings = []

512 + gen_taxonomy.process_taxonomy_file(

513 + path,

514 + sections={'applications.apache': ({'status': 'active', 'section_order': 1}, 'applications.apache')},

515 + icons=set(),

516 + metadata_indexes=self.metadata_indexes(tmp),

517 + ownership=ownership,

518 + findings=findings,

519 + referenced_literals=referenced_literals,

520 + )

521 + gen_taxonomy.emit_referenced_only_findings(referenced_literals, ownership, findings)

522 + self.assertEqual([finding.code for finding in findings], ['TAX038'])

523 +

524 + def test_unresolved_reference_payload_is_emitted(self):

525 + with tempfile.TemporaryDirectory() as tmp:

526 + path = self.write_taxonomy(tmp, """ - type: context

527 + contexts:

528 + - context: apache.future

529 + unresolved:

530 + reason: staged rename

531 + owner: cloud-frontend

532 + expires: "2026-08-01"

533 + chart_library: number

534 +""")

535 + findings = []

536 + placements, _ = gen_taxonomy.process_taxonomy_file(

537 + path,

538 + sections={'applications.apache': ({'status': 'active', 'section_order': 1}, 'applications.apache')},

539 + icons=set(),

540 + metadata_indexes=self.metadata_indexes(tmp),

541 + ownership={},

542 + findings=findings,

543 + )

544 + self.assertEqual([finding.code for finding in findings], [])

545 + unresolved = placements[0]['unresolved_references']

546 + self.assertEqual(unresolved, [

547 + {

548 + 'context': 'apache.future',

549 + 'reason': 'staged rename',

550 + 'owner': 'cloud-frontend',

551 + 'expires': '2026-08-01',

552 + 'item_path': 'apache.0',

553 + },

554 + ])

555 + self.assertEqual(placements[0]['items'][0]['unresolved_references'], unresolved)

556 +

557 +

558 +class TouchedCollectorGateTest(unittest.TestCase):

559 + def test_metadata_metrics_spans_ignore_overview_blocks(self):

560 + text = """plugin_name: go.d.plugin

561 +modules:

562 + - meta:

563 + module_name: demo

564 + overview:

565 + data_collection:

566 + metrics_description: demo

567 + metrics:

568 + folding:

569 + title: Metrics

570 + scopes: []

571 + setup:

572 + configuration: {}

573 +"""

574 + with tempfile.TemporaryDirectory() as tmp:

575 + path = Path(tmp) / 'metadata.yaml'

576 + path.write_text(text)

577 + self.assertEqual(check_collector_taxonomy.metadata_metrics_spans(path), [(8, 11)])

578 +

579 + def test_metadata_metrics_spans_do_not_depend_on_four_space_indent(self):

580 + text = """plugin_name: go.d.plugin

581 +modules:

582 +- meta:

583 + module_name: demo

584 + metrics:

585 + folding:

586 + title: Metrics

587 + scopes: []

588 + setup:

589 + configuration: {}

590 +"""

591 + with tempfile.TemporaryDirectory() as tmp:

592 + path = Path(tmp) / 'metadata.yaml'

593 + path.write_text(text)

594 + self.assertEqual(check_collector_taxonomy.metadata_metrics_spans(path), [(5, 8)])

595 +

596 + def test_range_intersection(self):

597 + spans = [(7, 10)]

598 + self.assertFalse(check_collector_taxonomy.range_intersects_spans(3, 1, spans))

599 + self.assertTrue(check_collector_taxonomy.range_intersects_spans(8, 1, spans))

600 +

601 + def test_missing_metrics_block_with_diff_is_touched(self):

602 + text = """plugin_name: go.d.plugin

603 +modules:

604 + - meta:

605 + module_name: demo

606 + setup:

607 + configuration: {}

608 +"""

609 + with tempfile.TemporaryDirectory() as tmp:

610 + path = Path(tmp) / 'metadata.yaml'

611 + path.write_text(text)

612 + with patch.object(check_collector_taxonomy, 'run_git', return_value='@@ -8,4 +0,0 @@\n- metrics:\n'):

613 + self.assertTrue(check_collector_taxonomy.metadata_metrics_touched('base...head', path))

614 +

615 + def test_missing_metrics_block_without_diff_is_not_touched(self):

616 + text = """plugin_name: go.d.plugin

617 +modules:

618 + - meta:

619 + module_name: demo

620 +"""

621 + with tempfile.TemporaryDirectory() as tmp:

622 + path = Path(tmp) / 'metadata.yaml'

623 + path.write_text(text)

624 + with patch.object(check_collector_taxonomy, 'run_git', return_value=''):

625 + self.assertFalse(check_collector_taxonomy.metadata_metrics_touched('base...head', path))

626 +

627 + def test_deleted_collector_does_not_require_taxonomy(self):

628 + with tempfile.TemporaryDirectory() as tmp:

629 + collector_dir = Path(tmp) / 'demo'

630 + collector_dir.mkdir()

631 + with patch.object(check_collector_taxonomy, 'touched_collectors', return_value=[collector_dir]):

632 + findings = check_collector_taxonomy.check_touched_coverage('base...head')

633 + self.assertEqual(findings, [])

634 +

635 +

636 +class TaxonomyDeterminismTest(unittest.TestCase):

637 + def test_build_taxonomy_is_byte_identical_for_ten_runs(self):

638 + outputs = []

639 + for _ in range(10):

640 + taxonomy, findings = gen_taxonomy.build_taxonomy()

641 + self.assertEqual([finding for finding in findings if finding.severity == gen_taxonomy.FATAL], [])

642 + outputs.append(json.dumps(taxonomy, indent=2, sort_keys=True) + '\n')

643 + self.assertEqual(len(set(outputs)), 1)

644 +

645 +

646 +if __name__ == '__main__':

647 + unittest.main()

src/go/plugin/go.d/collector/apache/taxonomy.yaml new

+103

@@ -0,0 +1,103 @@

1 +taxonomy_version: 1

2 +plugin_name: go.d.plugin

3 +module_name: apache

4 +placements:

5 + - id: apache

6 + section_id: applications.apache

7 + title: Apache

8 + icon: apache

9 + families: true

10 + items:

11 + - type: grid

12 + id: apache-heads

13 + items:

14 + - type: context

15 + title: Connections

16 + contexts:

17 + - apache.connections

18 + chart_library: easypiechart

19 + group_by:

20 + - selected

21 + value_range: [0, null]

22 + colors:

23 + - "#990099"

24 + - "#0099C6"

25 + layout: { left: 0, top: 0, width: 3, height: 4 }

26 + - type: context

27 + title: Requests

28 + contexts:

29 + - apache.requests

30 + chart_library: easypiechart

31 + show_post_aggregations: true

32 + post_group_by:

33 + - selected

34 + value_range: [0, null]

35 + colors:

36 + - "#994499"

37 + - "#0099C6"

38 + layout: { left: 3, top: 0, width: 3, height: 4 }

39 + - type: context

40 + title: Traffic

41 + contexts:

42 + - apache.net

43 + chart_library: gauge

44 + group_by:

45 + - selected

46 + value_range: [0, null]

47 + colors:

48 + - "#994499"

49 + - "#0099C6"

50 + layout: { left: 6, top: 0, width: 3, height: 4 }

51 + - type: context

52 + title: Max Workers %

53 + contexts:

54 + - apache.workers

55 + chart_library: gauge

56 + group_by:

57 + - percentage-of-instance

58 + show_post_aggregations: true

59 + post_group_by:

60 + - selected

61 + aggregation_method: max

62 + selected_dimensions:

63 + - busy

64 + value_range: [0, 100]

65 + colors:

66 + - "#994499"

67 + - "#0099C6"

68 + layout: { left: 9, top: 0, width: 3, height: 4 }

69 + - type: context

70 + title: List Apache Jobs

71 + contexts:

72 + - apache.connections

73 + - apache.requests

74 + - apache.net

75 + - apache.workers

76 + - apache.uptime

77 + chart_library: table

78 + group_by:

79 + - dimension

80 + - label

81 + - node

82 + - context

83 + group_by_label:

84 + - _collect_job

85 + table_columns:

86 + - context

87 + - dimension

88 + labels:

89 + apache.connections: Connections

90 + apache.requests: Requests

91 + apache.net: Traffic

92 + apache.workers: Workers

93 + apache.uptime: Uptime

94 + - apache.connections

95 + - apache.conns_async

96 + - apache.workers

97 + - apache.scoreboard

98 + - apache.requests

99 + - apache.net

100 + - apache.reqpersec

101 + - apache.bytespersec

102 + - apache.bytesperreq

103 + - apache.uptime

src/go/plugin/go.d/collector/mysql/taxonomy.yaml new

+303

@@ -0,0 +1,303 @@

1 +taxonomy_version: 1

2 +plugin_name: go.d.plugin

3 +module_name: mysql

4 +placements:

5 + - id: mysql

6 + section_id: applications.mysql

7 + title: MySQL

8 + icon: mysql

9 + items:

10 + - type: grid

11 + id: mysql-heads

12 + title: mysql-heads

13 + items:

14 + - type: context

15 + title: No. of Slow Queries

16 + contexts: [mysql.queries]

17 + chart_library: number

18 + group_by: [selected]

19 + selected_dimensions: [slow_queries]

20 + colors: ["#3366CC", "#66AA00"]

21 + layout: { left: 0, top: 0, width: 3, height: 4 }

22 + - type: context

23 + title: Total No. of Deadlocks

24 + contexts: [mysql.innodb_deadlocks]

25 + chart_library: number

26 + group_by: [selected]

27 + colors: ["#DC3912", "#FE3912"]

28 + layout: { left: 3, top: 0, width: 3, height: 4 }

29 + - type: context

30 + title: Galera Cluster Status

31 + contexts: [mysql.galera_cluster_status]

32 + chart_library: bars

33 + dimensions_sort: valueDesc

34 + colors: ["#109618", "#3366CC"]

35 + layout: { left: 6, top: 0, width: 3, height: 4 }

36 + - type: context

37 + title: Total Open Transactions

38 + contexts: [mysql.galera_open_transactions]

39 + chart_library: easypiechart

40 + group_by: [selected]

41 + colors: ["#FF9900", "#D66300"]

42 + layout: { left: 9, top: 0, width: 3, height: 4 }

43 + - type: context

44 + title: MySQL Join Operation Issues

45 + contexts: [mysql.join_issues]

46 + chart_library: bars

47 + dimensions_sort: valueDesc

48 + layout: { left: 0, top: 4, width: 3, height: 4 }

49 + - type: context

50 + title: Number of MySQL threads

51 + contexts: [mysql.threads_created]

52 + chart_library: number

53 + group_by: [selected]

54 + colors: ["#990099", "#0099C6"]

55 + layout: { left: 3, top: 4, width: 3, height: 4 }

56 + - type: context

57 + title: InnoDB Buffer Pool Read Ahead

58 + contexts: [mysql.innodb_buffer_pool_read_ahead]

59 + chart_library: bars

60 + dimensions_sort: valueDesc

61 + colors: ["#3B3EAC", "#5054e6"]

62 + layout: { left: 6, top: 4, width: 3, height: 4 }

63 + - type: context

64 + title: Longest Query Duration

65 + contexts: [mysql.process_list_longest_query_duration]

66 + chart_library: number

67 + group_by: [selected]

68 + aggregation_method: max

69 + colors: ["#66AA00", "#EE9911"]

70 + layout: { left: 9, top: 4, width: 3, height: 4 }

71 + - type: context

72 + id: mysql-servers

73 + title: List MySQL Servers

74 + contexts:

75 + - mysql.connections_active

76 + - mysql.queries_type

77 + - mysql.net

78 + chart_library: table

79 + group_by: [dimension, label, node, context]

80 + group_by_label: [_collect_job]

81 + table_columns: [context, dimension]

82 + labels:

83 + mysql.connections_active: Connections

84 + mysql.queries_type: Queries

85 + mysql.net: Network

86 + - type: group

87 + id: queries

88 + title: Queries

89 + properties: { grouping: true }

90 + items:

91 + - type: group

92 + id: statistics

93 + title: Statistics

94 + items:

95 + - mysql.queries

96 + - mysql.queries_type

97 + - mysql.process_list_queries_count

98 + - mysql.process_list_longest_query_duration

99 + - mysql.process_list_fetch_query_duration

100 + - type: group

101 + id: cache

102 + title: Cache

103 + items:

104 + - mysql.qcache_ops

105 + - mysql.qcache

106 + - mysql.qcache_freemem

107 + - mysql.qcache_memblocks

108 + - type: group

109 + id: issues

110 + title: Issues

111 + items:

112 + - mysql.join_issues

113 + - mysql.sort_issues

114 + - mysql.tmp

115 + - type: group

116 + id: resources

117 + title: Resources

118 + properties: { grouping: true }

119 + items:

120 + - type: group

121 + id: handlers

122 + title: Handlers

123 + items:

124 + - mysql.handlers

125 + - type: group

126 + id: connections-and-threads

127 + title: Connections & Threads

128 + items:

129 + - mysql.connections

130 + - mysql.connections_active

131 + - mysql.connection_errors

132 + - mysql.threads

133 + - mysql.threads_created

134 + - mysql.thread_cache_misses

135 + - type: group

136 + id: network

137 + title: Network

138 + items:

139 + - mysql.net

140 + - type: group

141 + id: files-and-tables

142 + title: Files & Tables

143 + items:

144 + - mysql.files

145 + - mysql.files_rate

146 + - mysql.open_tables

147 + - mysql.opened_tables

148 + - mysql.table_open_cache_overflows

149 + - mysql.table_locks

150 + - type: group

151 + id: storage-engine

152 + title: Storage Engine

153 + properties: { grouping: true }

154 + items:

155 + - type: group

156 + id: innodb

157 + title: InnoDB

158 + properties: { grouping: true }

159 + items:

160 + - type: group

161 + id: buffer-pool

162 + title: Buffer Pool

163 + items:

164 + - mysql.innodb_buffer_pool_pages

165 + - mysql.innodb_buffer_pool_bytes

166 + - mysql.innodb_buffer_pool_ops

167 + - mysql.innodb_buffer_pool_read_ahead

168 + - mysql.innodb_buffer_pool_pages_flushed

169 + - mysql.innodb_buffer_pool_read_ahead_rnd

170 + - type: group

171 + id: io

172 + title: I/O

173 + items:

174 + - mysql.innodb_io

175 + - mysql.innodb_io_ops

176 + - mysql.innodb_io_pending_ops

177 + - type: group

178 + id: transactions-and-locks

179 + title: Transactions & Locks

180 + items:

181 + - mysql.innodb_rows

182 + - mysql.innodb_cur_row_lock

183 + - mysql.innodb_deadlocks

184 + - type: group

185 + id: logging

186 + title: Logging

187 + items:

188 + - mysql.innodb_log

189 + - mysql.innodb_os_log

190 + - mysql.innodb_os_log_fsync_writes

191 + - mysql.innodb_os_log_io

192 + - mysql.innodb_redo_log_activity

193 + - mysql.innodb_redo_log_checkpoint_age

194 + - mysql.innodb_redo_log_occupancy

195 + - type: group

196 + id: myisam

197 + title: MyISAM

198 + items:

199 + - mysql.key_blocks

200 + - mysql.key_requests

201 + - mysql.key_disk_ops

202 + - type: group

203 + id: replication

204 + title: Replication

205 + properties: { grouping: true }

206 + items:

207 + - type: group

208 + id: slave-status

209 + title: Slave Status

210 + items:

211 + - mysql.slave_status

212 + - mysql.slave_behind

213 + - type: group

214 + id: binary-logging

215 + title: Binary Logging

216 + items:

217 + - mysql.binlog_cache

218 + - mysql.binlog_stmt_cache

219 + - type: group

220 + id: galera-cluster

221 + title: Galera Cluster

222 + properties: { grouping: true }

223 + items:

224 + - type: group

225 + id: cluster-status

226 + title: Cluster Status

227 + items:

228 + - mysql.galera_cluster_status

229 + - mysql.galera_cluster_state

230 + - mysql.galera_cluster_size

231 + - mysql.galera_cluster_weight

232 + - mysql.galera_connected

233 + - mysql.galera_ready

234 + - type: group

235 + id: galera-replication

236 + title: Replication

237 + items:

238 + - mysql.galera_writesets

239 + - mysql.galera_bytes

240 + - mysql.galera_queue

241 + - type: group

242 + id: conflicts-and-flow-control

243 + title: Conflicts & Flow Control

244 + items:

245 + - mysql.galera_conflicts

246 + - mysql.galera_flow_control

247 + - type: group

248 + id: galera-resources

249 + title: Resources

250 + items:

251 + - mysql.galera_open_transactions

252 + - mysql.galera_thread_count

253 + - type: group

254 + id: user-statistics

255 + title: User Statistics

256 + properties: { grouping: true }

257 + items:

258 + - type: context

259 + id: mysql-users

260 + title: List MySQL Users

261 + contexts:

262 + - mysql.userstats_cpu

263 + - mysql.userstats_rows

264 + - mysql.userstats_commands

265 + - mysql.userstats_created_transactions

266 + - mysql.userstats_empty_queries

267 + chart_library: table

268 + group_by: [dimension, label, node, context]

269 + group_by_label: [user]

270 + table_columns: [context, dimension]

271 + labels:

272 + mysql.userstats_cpu: Cpu

273 + mysql.userstats_rows: Rows

274 + mysql.userstats_commands: Commands

275 + mysql.userstats_created_transactions: Transactions

276 + mysql.userstats_empty_queries: Empty Queries

277 + - type: group

278 + id: resource-usage

279 + title: Resource Usage

280 + short_name: Rusage

281 + items:

282 + - mysql.userstats_cpu

283 + - mysql.userstats_rows

284 + - type: group

285 + id: commands

286 + title: Commands

287 + items:

288 + - mysql.userstats_commands

289 + - mysql.userstats_denied_commands

290 + - type: group

291 + id: user-connections

292 + title: Connections

293 + items:

294 + - mysql.userstats_connections

295 + - mysql.userstats_lost_connections

296 + - mysql.userstats_denied_connections

297 + - type: group

298 + id: transactions

299 + title: Transactions

300 + items:

301 + - mysql.userstats_created_transactions

302 + - mysql.userstats_binlog_written

303 + - mysql.userstats_empty_queries

src/go/plugin/go.d/collector/nvidia_smi/taxonomy.yaml new

+81

@@ -0,0 +1,81 @@

1 +taxonomy_version: 1

2 +plugin_name: go.d.plugin

3 +module_name: nvidia_smi

4 +placements:

5 + - id: nvidia-smi

6 + section_id: system.hardware.gpus.nvidia

7 + title: NVIDIA GPUs

8 + icon: compute

9 + families: true

10 + properties:

11 + important: false

12 + grouping: false

13 + items:

14 + - type: context

15 + title: List Nvidia GPUs

16 + contexts:

17 + - nvidia_smi.gpu_pcie_bandwidth_usage

18 + - nvidia_smi.gpu_fan_speed_perc

19 + - nvidia_smi.gpu_temperature

20 + - nvidia_smi.gpu_utilization

21 + - nvidia_smi.gpu_memory_utilization

22 + - nvidia_smi.gpu_decoder_utilization

23 + - nvidia_smi.gpu_encoder_utilization

24 + chart_library: table

25 + group_by:

26 + - dimension

27 + - label

28 + - node

29 + - context

30 + group_by_label:

31 + - index

32 + - product_name

33 + table_columns:

34 + - context

35 + - dimension

36 + labels:

37 + nvidia_smi.gpu_pcie_bandwidth_usage: PCIe

38 + nvidia_smi.gpu_fan_speed_perc: Fan

39 + nvidia_smi.gpu_temperature: Temperature

40 + nvidia_smi.gpu_utilization: GPU %

41 + nvidia_smi.gpu_memory_utilization: Mem %

42 + nvidia_smi.gpu_decoder_utilization: Dec %

43 + nvidia_smi.gpu_encoder_utilization: Enc %

44 + - type: group

45 + id: bus

46 + title: Bus

47 + items:

48 + - nvidia_smi.gpu_pcie_bandwidth_usage

49 + - nvidia_smi.gpu_pcie_bandwidth_utilization

50 + - type: group

51 + id: utilization

52 + title: Utilization

53 + items:

54 + - nvidia_smi.gpu_fan_speed_perc

55 + - nvidia_smi.gpu_utilization

56 + - nvidia_smi.gpu_memory_utilization

57 + - nvidia_smi.gpu_decoder_utilization

58 + - nvidia_smi.gpu_encoder_utilization

59 + - type: group

60 + id: memory

61 + title: Memory

62 + items:

63 + - nvidia_smi.gpu_frame_buffer_memory_usage

64 + - nvidia_smi.gpu_bar1_memory_usage

65 + - type: group

66 + id: sensors

67 + title: Sensors

68 + items:

69 + - nvidia_smi.gpu_temperature

70 + - nvidia_smi.gpu_voltage

71 + - nvidia_smi.gpu_clock_freq

72 + - nvidia_smi.gpu_power_draw

73 + - nvidia_smi.gpu_performance_state

74 + - type: group

75 + id: mig

76 + title: MIG

77 + items:

78 + - nvidia_smi.gpu_mig_mode_current_status

79 + - nvidia_smi.gpu_mig_devices_count

80 + - nvidia_smi.gpu_mig_frame_buffer_memory_usage

81 + - nvidia_smi.gpu_mig_bar1_memory_usage

src/go/plugin/go.d/collector/postgres/taxonomy.yaml new

+256

@@ -0,0 +1,256 @@

1 +taxonomy_version: 1

2 +plugin_name: go.d.plugin

3 +module_name: postgres

4 +placements:

5 + - id: postgres

6 + section_id: applications.postgres

7 + title: Postgres

8 + icon: postgres

9 + families: true

10 + items:

11 + - type: grid

12 + id: postgres-heads

13 + items:

14 + - type: context

15 + title: Average Connections Utilization

16 + contexts:

17 + - postgres.connections_utilization

18 + chart_library: gauge

19 + group_by:

20 + - selected

21 + aggregation_method: avg

22 + value_range: [0, 100]

23 + colors:

24 + - "#6633CC"

25 + - "#905bfd"

26 + layout: { left: 0, top: 0, width: 2.5, height: 5 }

27 + - type: context

28 + title: Avg Fetched Row Ratio

29 + contexts:

30 + - postgres.db_ops_fetched_rows_ratio

31 + chart_library: dygraph

32 + group_by:

33 + - selected

34 + aggregation_method: avg

35 + sparkline: true

36 + renderer:

37 + overlays:

38 + - type: latestValue

39 + colors:

40 + - "#AAAA11"

41 + - "#ef0aef"

42 + layout: { left: 2.5, top: 0, width: 2, height: 5 }

43 + - type: context

44 + title: Average Cache Miss Ratio

45 + contexts:

46 + - postgres.db_cache_io_ratio

47 + chart_library: gauge

48 + group_by:

49 + - selected

50 + aggregation_method: avg

51 + value_range: [0, 100]

52 + colors:

53 + - "#DC3912"

54 + - "#FE3912"

55 + layout: { left: 4.5, top: 0, width: 2.5, height: 5 }

56 + - type: context

57 + title: Connection State Count

58 + contexts:

59 + - postgres.connections_state_count

60 + chart_library: bars

61 + dimensions_sort: valueDesc

62 + layout: { left: 7, top: 0, width: 2.5, height: 5 }

63 + - type: context

64 + title: Minimum Uptime

65 + contexts:

66 + - postgres.uptime

67 + chart_library: number

68 + group_by:

69 + - selected

70 + aggregation_method: min

71 + layout: { left: 9.5, top: 0, width: 2.5, height: 1.66 }

72 + - type: context

73 + title: Total Database count

74 + contexts:

75 + - postgres.databases_count

76 + chart_library: number

77 + group_by:

78 + - selected

79 + layout: { left: 9.5, top: 1.66, width: 2.5, height: 1.66 }

80 + - type: context

81 + title: Total Database Size

82 + contexts:

83 + - postgres.db_size

84 + chart_library: dygraph

85 + sparkline: true

86 + renderer:

87 + overlays:

88 + - type: latestValue

89 + layout: { left: 9.5, top: 3.33, width: 2.5, height: 1.66 }

90 + - type: context

91 + title: Connections Utilization per Database

92 + contexts:

93 + - postgres.db_connections_utilization

94 + chart_library: bars

95 + group_by:

96 + - label

97 + group_by_label:

98 + - database

99 + dimensions_sort: valueDesc

100 + layout: { left: 0, top: 5, width: 2.5, height: 5 }

101 + - type: context

102 + title: Fetched Row Ratio per DB

103 + contexts:

104 + - postgres.db_ops_fetched_rows_ratio

105 + chart_library: bars

106 + group_by:

107 + - label

108 + group_by_label:

109 + - database

110 + dimensions_sort: valueDesc

111 + layout: { left: 2.5, top: 5, width: 2, height: 5 }

112 + - type: context

113 + title: Cache Miss Ratio per DB

114 + contexts:

115 + - postgres.db_cache_io_ratio

116 + chart_library: bars

117 + group_by:

118 + - label

119 + group_by_label:

120 + - database

121 + dimensions_sort: valueDesc

122 + layout: { left: 4.5, top: 5, width: 2.5, height: 5 }

123 + - type: context

124 + title: Rows Written per Database

125 + contexts:

126 + - postgres.db_ops_write_rows_rate

127 + chart_library: bars

128 + dimensions_sort: valueDesc

129 + layout: { left: 7, top: 5, width: 2.5, height: 5 }

130 + - type: context

131 + title: Database Sizes

132 + contexts:

133 + - postgres.db_size

134 + chart_library: bars

135 + group_by:

136 + - label

137 + group_by_label:

138 + - database

139 + dimensions_sort: valueDesc

140 + layout: { left: 9.5, top: 5, width: 2.5, height: 5 }

141 + - type: context

142 + title: Top Queries by Duration

143 + contexts:

144 + - postgres.queries_duration

145 + chart_library: bars

146 + dimensions_sort: valueDesc

147 + layout: { left: 0, top: 10, width: 4.5, height: 5 }

148 + - type: context

149 + title: Average Dead Row Ratio

150 + contexts:

151 + - postgres.table_rows_dead_ratio

152 + chart_library: gauge

153 + group_by:

154 + - selected

155 + aggregation_method: avg

156 + value_range: [0, 100]

157 + colors:

158 + - "#AAAA11"

159 + - "#ef0aef"

160 + layout: { left: 4.5, top: 10, width: 2.5, height: 5 }

161 + - type: context

162 + title: Average Table Bloat %

163 + contexts:

164 + - postgres.table_bloat_size_perc

165 + chart_library: gauge

166 + group_by:

167 + - selected

168 + aggregation_method: avg

169 + value_range: [0, 100]

170 + colors:

171 + - "#DC3912"

172 + - "#FE3912"

173 + layout: { left: 7, top: 10, width: 2.5, height: 5 }

174 + - type: context

175 + title: Average Index Bloat %

176 + contexts:

177 + - postgres.index_bloat_size_perc

178 + chart_library: gauge

179 + group_by:

180 + - selected

181 + aggregation_method: avg

182 + value_range: [0, 100]

183 + colors:

184 + - "#DC3912"

185 + - "#FE3912"

186 + layout: { left: 9.5, top: 10, width: 2.5, height: 5 }

187 + - postgres.connections_utilization

188 + - postgres.connections_usage

189 + - postgres.connections_state_count

190 + - postgres.transactions_duration

191 + - postgres.queries_duration

192 + - postgres.locks_utilization

193 + - postgres.checkpoints_rate

194 + - postgres.checkpoints_time

195 + - postgres.bgwriter_halts_rate

196 + - postgres.buffers_io_rate

197 + - postgres.buffers_backend_fsync_rate

198 + - postgres.buffers_allocated_rate

199 + - postgres.wal_io_rate

200 + - postgres.wal_files_count

201 + - postgres.wal_archiving_files_count

202 + - postgres.autovacuum_workers_count

203 + - postgres.txid_exhaustion_towards_autovacuum_perc

204 + - postgres.txid_exhaustion_perc

205 + - postgres.txid_exhaustion_oldest_txid_num

206 + - postgres.catalog_relations_count

207 + - postgres.catalog_relations_size

208 + - postgres.uptime

209 + - postgres.databases_count

210 + - postgres.replication_app_wal_lag_size

211 + - postgres.replication_app_wal_lag_time

212 + - postgres.replication_slot_files_count

213 + - postgres.db_transactions_ratio

214 + - postgres.db_transactions_rate

215 + - postgres.db_connections_utilization

216 + - postgres.db_connections_count

217 + - postgres.db_cache_io_ratio

218 + - postgres.db_io_rate

219 + - postgres.db_ops_fetched_rows_ratio

220 + - postgres.db_ops_read_rows_rate

221 + - postgres.db_ops_write_rows_rate

222 + - postgres.db_conflicts_rate

223 + - postgres.db_conflicts_reason_rate

224 + - postgres.db_deadlocks_rate

225 + - postgres.db_locks_held_count

226 + - postgres.db_locks_awaited_count

227 + - postgres.db_temp_files_created_rate

228 + - postgres.db_temp_files_io_rate

229 + - postgres.db_size

230 + - postgres.table_rows_dead_ratio

231 + - postgres.table_rows_count

232 + - postgres.table_ops_rows_rate

233 + - postgres.table_ops_rows_hot_ratio

234 + - postgres.table_ops_rows_hot_rate

235 + - postgres.table_cache_io_ratio

236 + - postgres.table_io_rate

237 + - postgres.table_index_cache_io_ratio

238 + - postgres.table_index_io_rate

239 + - postgres.table_toast_cache_io_ratio

240 + - postgres.table_toast_io_rate

241 + - postgres.table_toast_index_cache_io_ratio

242 + - postgres.table_toast_index_io_rate

243 + - postgres.table_scans_rate

244 + - postgres.table_scans_rows_rate

245 + - postgres.table_autovacuum_since_time

246 + - postgres.table_vacuum_since_time

247 + - postgres.table_autoanalyze_since_time

248 + - postgres.table_analyze_since_time

249 + - postgres.table_null_columns

250 + - postgres.table_size

251 + - postgres.table_bloat_size_perc

252 + - postgres.table_bloat_size

253 + - postgres.index_size

254 + - postgres.index_bloat_size_perc

255 + - postgres.index_bloat_size

256 + - postgres.index_usage_status

src/go/plugin/go.d/collector/snmp/metadata.yaml

@@ -1145,6 +1145,9 @@ modules:

1145 - `route_totals` contains cumulative counters where the vendor MIB only exposes totals.

1146 - When the source model is peer-family scoped, alerts and chart labels include AFI/SAFI so operators can distinguish otherwise similar peers.

1147 availability: []

1148 + dynamic_context_prefixes:

1149 + - prefix: snmp.

1150 + reason: SNMP profiles emit vendor- and device-specific chart contexts at runtime under the snmp namespace.

1151 scopes:

1152 - name: device licensing

1153 description: Shared device-level licensing health metrics emitted when the matched SNMP profile provides licensing telemetry. Supported profile coverage includes Check Point licensing state and per-blade expiry, Fortinet FortiGate contract/service/account expirations, Cisco traditional licensing end-date/remaining-time/state/usage telemetry, Cisco Smart Licensing authorization, certificate, evaluation, and state telemetry, Sophos Firewall subscription state and per-license expiry telemetry, Blue Coat ProxySG application/feature/component expiry, expire-type, and state telemetry, and basic MikroTik RouterOS upgrade-entitlement telemetry. MikroTik support is intentionally limited to the RouterOS upgrade-entitlement fields exposed by SNMP, and epoch-like placeholder `mtxrLicUpgrUntil` values are ignored.

src/go/plugin/go.d/collector/snmp/taxonomy.yaml new

+169

@@ -0,0 +1,169 @@

1 +taxonomy_version: 1

2 +plugin_name: go.d.plugin

3 +module_name: snmp

4 +placements:

5 + - id: snmp

6 + section_id: remote-devices.snmp

7 + title: SNMP-enabled devices

8 + short_name: SNMP

9 + icon: snmp

10 + families: true

11 + properties:

12 + important: false

13 + grouping: false

14 + single_node:

15 + properties:

16 + grouping: true

17 + items:

18 + - type: grid

19 + id: network-interfaces-heads

20 + items:

21 + - type: context

22 + title: Traffic In

23 + contexts:

24 + - context_prefix:

25 + - snmp.device_prof_ifTotalTraffic

26 + chart_library: gauge

27 + group_by:

28 + - selected

29 + selected_dimensions:

30 + - in

31 + value_range: [0, null]

32 + colors:

33 + - "#0099C6"

34 + - "#990099"

35 + layout: { left: 0, top: 0, width: 3, height: 4 }

36 + - type: context

37 + title: Traffic Out

38 + contexts:

39 + - context_prefix:

40 + - snmp.device_prof_ifTotalTraffic

41 + chart_library: gauge

42 + group_by:

43 + - selected

44 + selected_dimensions:

45 + - out

46 + value_range: [0, null]

47 + colors:

48 + - "#990099"

49 + - "#0099C6"

50 + layout: { left: 3, top: 0, width: 3, height: 4 }

51 + - type: context

52 + title: Errors

53 + contexts:

54 + - context_prefix:

55 + - snmp.device_prof_ifTotalErrors

56 + chart_library: gauge

57 + group_by:

58 + - selected

59 + colors:

60 + - "#990099"

61 + - "#0099C6"

62 + layout: { left: 6, top: 0, width: 3, height: 4 }

63 + - type: context

64 + title: Discards

65 + contexts:

66 + - context_prefix:

67 + - snmp.device_prof_ifTotalDiscards

68 + chart_library: gauge

69 + group_by:

70 + - selected

71 + colors:

72 + - "#990099"

73 + - "#0099C6"

74 + layout: { left: 9, top: 0, width: 3, height: 4 }

75 + - type: context

76 + title: Unicast

77 + contexts:

78 + - context_prefix:

79 + - snmp.device_prof_ifTotalPacketsUcast

80 + chart_library: bars

81 + layout: { left: 0, top: 4, width: 3, height: 2.5 }

82 + - type: context

83 + title: Multicast

84 + contexts:

85 + - context_prefix:

86 + - snmp.device_prof_ifTotalPacketsMulticast

87 + chart_library: bars

88 + layout: { left: 3, top: 4, width: 3, height: 2.5 }

89 + - type: context

90 + title: Broadcast

91 + contexts:

92 + - context_prefix:

93 + - snmp.device_prof_ifTotalPacketsBroadcast

94 + chart_library: bars

95 + layout: { left: 6, top: 4, width: 3, height: 2.5 }

96 + - type: context

97 + title: Uptime

98 + contexts:

99 + - context_prefix:

100 + - snmp.device_prof_systemUptime

101 + chart_library: number

102 + group_by:

103 + - selected

104 + aggregation_method: min

105 + layout: { left: 9, top: 4, width: 3, height: 2.5 }

106 + - type: view_switch

107 + id: snmp-devices-table

108 + multi_node:

109 + type: context

110 + title: List SNMP devices

111 + contexts:

112 + - context_prefix:

113 + - snmp.device_prof_ifTotalTraffic

114 + - context_prefix:

115 + - snmp.device_prof_ifTotalPacketsUcast

116 + - context_prefix:

117 + - snmp.device_prof_ifTotalErrors

118 + chart_library: table

119 + group_by:

120 + - dimension

121 + - label

122 + - node

123 + - context

124 + group_by_label:

125 + - _collect_job

126 + - vendor

127 + table_columns:

128 + - context

129 + - dimension

130 + table_sort_by:

131 + - id: snmp.device_prof_ifTotalTraffic

132 + desc: true

133 + labels:

134 + snmp.device_prof_ifTotalTraffic: Traffic

135 + snmp.device_prof_ifTotalPacketsUcast: Packets

136 + snmp.device_prof_ifTotalErrors: Errors

137 + single_node:

138 + type: context

139 + title: List SNMP devices

140 + contexts:

141 + - context_prefix:

142 + - snmp.device_prof_ifTraffic

143 + - context_prefix:

144 + - snmp.device_prof_ifPacketsUcast

145 + - context_prefix:

146 + - snmp.device_prof_ifErrors

147 + chart_library: table

148 + group_by:

149 + - dimension

150 + - label

151 + - context

152 + group_by_label:

153 + - interface

154 + - if_type

155 + table_columns:

156 + - context

157 + - dimension

158 + table_sort_by:

159 + - id: snmp.device_prof_ifTraffic

160 + desc: true

161 + labels:

162 + snmp.device_prof_ifTraffic: Traffic

163 + snmp.device_prof_ifPacketsUcast: Packets

164 + snmp.device_prof_ifErrors: Errors

165 + - type: selector

166 + id: snmp-contexts

167 + title: SNMP profile metrics

168 + context_prefix:

169 + - snmp.