| 1 | # How collector taxonomy becomes `integrations/taxonomy.json` |
| 2 | |
| 3 | Question answered: what is the general flow from collector |
| 4 | `metadata.yaml` and `taxonomy.yaml` to the generated dashboard |
| 5 | taxonomy artifact consumed by downstream frontend code? |
| 6 | |
| 7 | ## Short version |
| 8 | |
| 9 | `metadata.yaml` is the metric-context source of truth. Collector |
| 10 | `taxonomy.yaml` files organize those contexts into the dashboard table |
| 11 | of contents. `integrations/gen_taxonomy.py` validates both sides |
| 12 | against the taxonomy registries and schemas, then emits the gitignored |
| 13 | `integrations/taxonomy.json` cross-repo contract. |
| 14 | |
| 15 | The implementation details can evolve, but the durable model is: |
| 16 | |
| 17 | 1. metadata declares what metric contexts exist; |
| 18 | 2. taxonomy declares where those contexts belong and which widgets |
| 19 | reference them; |
| 20 | 3. the generator proves the references are valid; |
| 21 | 4. the generated JSON carries the normalized section tree, placements, |
| 22 | recursive items, and context snapshots. |
| 23 | |
| 24 | ## Inputs |
| 25 | |
| 26 | The taxonomy pipeline reads four source classes: |
| 27 | |
| 28 | - Collector `metadata.yaml` files. The generator loads collector |
| 29 | modules through the shared integrations loader and extracts metric |
| 30 | contexts from `metrics.scopes[].metrics[].name`; see |
| 31 | `integrations/gen_taxonomy.py:269-276`. |
| 32 | - Collector `taxonomy.yaml` files. These live next to collector |
| 33 | metadata and use the closed v1 authoring schema |
| 34 | `integrations/schemas/taxonomy_collector.json`. |
| 35 | - `integrations/taxonomy/sections.yaml`. This registry owns stable |
| 36 | `section_id` targets and parentage for the generated TOC section |
| 37 | tree; schema: `integrations/schemas/taxonomy_sections.json`. |
| 38 | - `integrations/taxonomy/icons.yaml`. This registry limits the icon IDs |
| 39 | sections and placements may reference. |
| 40 | |
| 41 | The field-level contract is documented in |
| 42 | `../schema-reference.md`. The contributor workflow is documented in |
| 43 | `../recipes/add-go-collector.md` and |
| 44 | `../recipes/update-collector.md`. |
| 45 | |
| 46 | ## Metadata indexing |
| 47 | |
| 48 | The generator first builds metadata indexes from all known collector |
| 49 | metadata: |
| 50 | |
| 51 | - `by_path_module`: matches a `taxonomy.yaml` file to its sibling |
| 52 | `metadata.yaml` module by path, `plugin_name`, and `module_name`. |
| 53 | - `all_contexts`: sorted global list of known metric contexts, used for |
| 54 | prefix resolution. |
| 55 | - `contexts_by_plugin`: contexts grouped by plugin name, used for |
| 56 | `collect_plugin` selectors. |
| 57 | - dynamic selector guardrails from |
| 58 | `metrics.dynamic_context_prefixes` and |
| 59 | `metrics.dynamic_collect_plugins`. |
| 60 | |
| 61 | The relevant implementation is `integrations/gen_taxonomy.py:286-315`. |
| 62 | |
| 63 | This is why `metadata.yaml` is the metric source of truth: a literal |
| 64 | context in taxonomy authoring is valid only if the sibling metadata |
| 65 | module declares it. A taxonomy file can organize and reference metric |
| 66 | contexts; it cannot invent static metric contexts. |
| 67 | |
| 68 | ## Taxonomy authoring validation |
| 69 | |
| 70 | Each collector `taxonomy.yaml` is loaded and validated against the |
| 71 | closed authoring schema before semantic validation. The schema rejects |
| 72 | old or ambiguous shapes such as placement-level `contexts:`, |
| 73 | `section_path:`, and string shorthand in display-only positions. |
| 74 | |
| 75 | After schema validation, the generator checks: |
| 76 | |
| 77 | - `section_id` exists in `sections.yaml`; |
| 78 | - icon IDs exist in `icons.yaml`; |
| 79 | - literal owned contexts exist in the sibling metadata; |
| 80 | - literal widget references exist in metadata unless they carry the |
| 81 | explicit `unresolved` escape hatch; |
| 82 | - dynamic selectors are declared by metadata guardrails; |
| 83 | - display widgets reference contexts but do not own them; |
| 84 | - every literal widget reference is owned somewhere else unless it is |
| 85 | deliberately unresolved. |
| 86 | |
| 87 | The matching and semantic validation start in |
| 88 | `integrations/gen_taxonomy.py:745-790`. Selector and literal-reference |
| 89 | validation live around `integrations/gen_taxonomy.py:438-526`. |
| 90 | |
| 91 | ## Ownership model |
| 92 | |
| 93 | The generated artifact separates ownership from display references: |
| 94 | |
| 95 | - Structural strings and `type: owned_context` own literal contexts. |
| 96 | - Structural `type: selector` owns the contexts matched by |
| 97 | `context_prefix` or `collect_plugin`. |
| 98 | - Containers such as `group`, `flatten`, `grid`, `first_available`, and |
| 99 | `view_switch` aggregate context snapshots from their children. |
| 100 | - `type: context` display widgets reference contexts through |
| 101 | `contexts:` but do not own them. |
| 102 | |
| 103 | Generated items and placements therefore carry: |
| 104 | |
| 105 | - `resolved_contexts`: contexts owned by that node after child and |
| 106 | selector aggregation. |
| 107 | - `referenced_contexts`: contexts referenced by display widgets. |
| 108 | - `unresolved_references`: staged widget references that intentionally |
| 109 | do not resolve yet, with `reason`, `owner`, `expires`, and |
| 110 | `item_path`. |
| 111 | |
| 112 | The recursive emission logic is in `integrations/gen_taxonomy.py:551-719`. |
| 113 | The FE-facing meaning of the generated fields is documented in |
| 114 | `../in-app-contract.md`. |
| 115 | |
| 116 | ## Output artifact |
| 117 | |
| 118 | The generated artifact is `integrations/taxonomy.json`. It is validated |
| 119 | against `integrations/schemas/taxonomy_output.json` and is intentionally |
| 120 | gitignored. |
| 121 | |
| 122 | Top-level shape: |
| 123 | |
| 124 | ```json |
| 125 | { |
| 126 | "taxonomy_schema_version": 1, |
| 127 | "source": {}, |
| 128 | "sections": [], |
| 129 | "placements": [], |
| 130 | "opted_out_collectors": [] |
| 131 | } |
| 132 | ``` |
| 133 | |
| 134 | Important output concepts: |
| 135 | |
| 136 | - `sections[]` is the resolved global section registry. |
| 137 | - `placements[]` is the ordered list of collector-owned TOC placements. |
| 138 | - `placements[].items[]` is the normalized recursive item tree. |
| 139 | - `collector_ids` links a placement back to the integration IDs produced |
| 140 | from metadata. |
| 141 | - `section_id` is the stable registry handle; `section_path` is the |
| 142 | resolved path for consumers. |
| 143 | |
| 144 | Assembly, deterministic placement sorting, and output schema validation |
| 145 | are handled in `integrations/gen_taxonomy.py:847-883`. Writing is handled |
| 146 | by the generator CLI in `integrations/gen_taxonomy.py:890-920`. |
| 147 | |
| 148 | ## CI flow |
| 149 | |
| 150 | Pull requests run the taxonomy checker from |
| 151 | `.github/workflows/check-markdown.yml`. The checker: |
| 152 | |
| 153 | - validates all committed taxonomy sources by building the artifact; |
| 154 | - enforces taxonomy coverage when a PR changes a collector |
| 155 | `taxonomy.yaml`, adds/removes it, or edits metric-bearing parts of |
| 156 | `metadata.yaml`; |
| 157 | - runs the taxonomy unit tests. |
| 158 | |
| 159 | See `.github/workflows/check-markdown.yml:45-58` and |
| 160 | `integrations/check_collector_taxonomy.py`. |
| 161 | |
| 162 | The master regeneration workflow runs `integrations/gen_taxonomy.py` as |
| 163 | part of the integrations regeneration job; see |
| 164 | `.github/workflows/generate-integrations.yml:59-68`. The generated |
| 165 | `taxonomy.json` is still a runtime/downstream contract artifact, not a |
| 166 | committed source file. |
| 167 | |
| 168 | ## Worked mental model |
| 169 | |
| 170 | For a static collector such as MySQL: |
| 171 | |
| 172 | 1. `metadata.yaml` declares `mysql.queries`. |
| 173 | 2. `mysql/taxonomy.yaml` owns `mysql.queries` in a structural item. |
| 174 | 3. A summary grid widget may also reference `mysql.queries`. |
| 175 | 4. The generated placement includes `mysql.queries` in |
| 176 | `resolved_contexts` because it is owned, and in |
| 177 | `referenced_contexts` where the widget uses it. |
| 178 | |
| 179 | For a dynamic collector such as SNMP: |
| 180 | |
| 181 | 1. `metadata.yaml` declares a dynamic namespace such as `snmp.`. |
| 182 | 2. `snmp/taxonomy.yaml` may use a narrower selector like |
| 183 | `snmp.device_prof_` under that declared namespace. |
| 184 | 3. Selector items own the matched context snapshot; selector references |
| 185 | inside widgets reference dynamic contexts without claiming ownership. |
| 186 | 4. The generated JSON preserves selector objects so downstream frontend |
| 187 | code can resolve runtime dynamic contexts cleanly. |
| 188 | |
| 189 | ## How I figured this out |
| 190 | |
| 191 | Files read: |
| 192 | |
| 193 | - `integrations/gen_taxonomy.py` |
| 194 | - `integrations/check_collector_taxonomy.py` |
| 195 | - `integrations/schemas/taxonomy_collector.json` |
| 196 | - `integrations/schemas/taxonomy_output.json` |
| 197 | - `.github/workflows/check-markdown.yml` |
| 198 | - `.github/workflows/generate-integrations.yml` |
| 199 | - `../schema-reference.md` |
| 200 | - `../in-app-contract.md` |
| 201 | |
| 202 | Commands used during the original analysis: |
| 203 | |
| 204 | ```bash |
| 205 | rg -n "def module_contexts|def build_metadata_indexes|def process_taxonomy_file|def emit_item|def build_taxonomy" integrations/gen_taxonomy.py |
| 206 | rg -n "gen_taxonomy|check_collector_taxonomy|taxonomy.json|taxonomy.yaml" .github/workflows integrations/README.md .agents/sow/specs/taxonomy.md |
| 207 | ``` |