| 1 | # ibm.d generation chain |
| 2 | |
| 3 | ibm.d is a Go collector framework whose modules generate their |
| 4 | own `metadata.yaml` (and `README.md`, `config_schema.json`, |
| 5 | `zz_generated_contexts.go`) from a small set of authoritative |
| 6 | inputs. This is fundamentally different from go.d / python.d / |
| 7 | charts.d collectors where `metadata.yaml` is hand-edited. |
| 8 | |
| 9 | **Maintainer rule**: for any ibm.d module, NEVER edit |
| 10 | `metadata.yaml`, `README.md`, or `config_schema.json` directly. |
| 11 | Edit `contexts.yaml`, `config.go`, or `module.yaml`, then run |
| 12 | `go generate ./...`. |
| 13 | |
| 14 | ## Layout per module |
| 15 | |
| 16 | ``` |
| 17 | src/go/plugin/ibm.d/modules/<m>/ |
| 18 | ├── module.yaml # display name, description, icon, categories, link, keywords |
| 19 | ├── config.go # Config struct -- parsed via Go AST |
| 20 | ├── contexts/ |
| 21 | │ ├── contexts.yaml # metric definitions: classes -> contexts -> dimensions |
| 22 | │ ├── doc.go # //go:generate go run ../../../metricgen/main.go ... |
| 23 | │ └── zz_generated_contexts.go # GENERATED -- DO NOT EDIT |
| 24 | ├── generate.go # //go:generate go run ../../docgen ... |
| 25 | ├── metadata.yaml # GENERATED -- DO NOT EDIT (consumed by gen_integrations.py) |
| 26 | ├── README.md # GENERATED -- DO NOT EDIT |
| 27 | ├── config_schema.json # GENERATED -- DO NOT EDIT |
| 28 | └── <module-source>.go ... # the collector implementation (hand-written) |
| 29 | ``` |
| 30 | |
| 31 | `websphere/` is a special parent: its sub-modules |
| 32 | `websphere/{jmx,mp,pmi}/` each have their own |
| 33 | `metadata.yaml`, `module.yaml`, etc. `gen_integrations.py:35` |
| 34 | adds `src/go/plugin/ibm.d/modules/websphere` separately to |
| 35 | `COLLECTOR_SOURCES` so these one-level-deeper paths get picked |
| 36 | up. |
| 37 | |
| 38 | ## The two generators |
| 39 | |
| 40 | ### `metricgen` -- contexts.yaml -> zz_generated_contexts.go |
| 41 | |
| 42 | Repo path: `src/go/plugin/ibm.d/metricgen/main.go`. |
| 43 | |
| 44 | Reads a module's `contexts/contexts.yaml`. The file declares |
| 45 | classes -> contexts -> dimensions in a structured form: |
| 46 | |
| 47 | ```yaml |
| 48 | classes: |
| 49 | - name: connection |
| 50 | contexts: |
| 51 | - name: connection_count |
| 52 | title: Connection count |
| 53 | units: connections |
| 54 | family: connections |
| 55 | type: line |
| 56 | dimensions: |
| 57 | - name: total |
| 58 | - name: active |
| 59 | ``` |
| 60 | |
| 61 | Writes `contexts/zz_generated_contexts.go` -- a Go source file |
| 62 | that registers these contexts with the ibm.d framework so the |
| 63 | collector can emit metrics by name. The generated file is |
| 64 | committed. |
| 65 | |
| 66 | Triggered by: |
| 67 | |
| 68 | ```go |
| 69 | //go:generate go run ../../../metricgen/main.go ... |
| 70 | ``` |
| 71 | |
| 72 | at `src/go/plugin/ibm.d/modules/<m>/contexts/doc.go:5`. |
| 73 | |
| 74 | ### `docgen` -- contexts.yaml + config.go + module.yaml -> metadata.yaml + README.md + config_schema.json |
| 75 | |
| 76 | Repo path: `src/go/plugin/ibm.d/docgen/main.go`. |
| 77 | |
| 78 | Inputs (per module): |
| 79 | |
| 80 | - `contexts/contexts.yaml` -- the same metric structure |
| 81 | metricgen reads. Parsed as `Config` with `Class` entries |
| 82 | (`docgen/main.go:28-55`). |
| 83 | - `config.go` -- the Go `Config` struct. Parsed via Go AST |
| 84 | (`docgen/config_parser.go`) to extract `ConfigField` records |
| 85 | (`docgen/main.go:57-78`). |
| 86 | - `module.yaml` -- module-level metadata: name, display name, |
| 87 | description, icon, categories, link, keywords. |
| 88 | |
| 89 | Outputs (per module): |
| 90 | |
| 91 | - `metadata.yaml` -- written from `metadataTemplate` |
| 92 | (`docgen/main.go:562`). The generated file opens with the |
| 93 | banner: `# Generated metadata.yaml for <module> module`. It |
| 94 | carries hardcoded scaffolding (`most_popular: false`, |
| 95 | default `update_every: 1` option, `endpoint: dummy://localhost`, |
| 96 | and a fixed prerequisite "Enable monitoring interface") |
| 97 | PLUS the dynamic content extracted from `contexts.yaml` and |
| 98 | `config.go`. Authors who want richer metadata.yaml content |
| 99 | must extend the template or `module.yaml`, NOT edit the |
| 100 | generated file. |
| 101 | - `config_schema.json` -- written from a separate template |
| 102 | (`docgen/main.go:528`). Used by the dashboard's DYNCFG |
| 103 | editor. |
| 104 | - `README.md` -- written from a readme template |
| 105 | (`docgen/main.go:552`). Includes module info, metric tables, |
| 106 | config tables. Banner depends on the template. |
| 107 | |
| 108 | Triggered by: |
| 109 | |
| 110 | ```go |
| 111 | //go:generate go run ../../docgen -module=<m> -contexts=contexts/contexts.yaml -config=config.go -module-info=module.yaml |
| 112 | ``` |
| 113 | |
| 114 | at `src/go/plugin/ibm.d/modules/<m>/generate.go:3`. |
| 115 | |
| 116 | ## End-to-end edit recipe (ibm.d module) |
| 117 | |
| 118 | 1. Edit one of: |
| 119 | - `contexts/contexts.yaml` to add/change/remove a metric |
| 120 | class, context, or dimension; |
| 121 | - `config.go` to add/change/remove a config field; |
| 122 | - `module.yaml` to change the display name, description, |
| 123 | categories, icon, etc. |
| 124 | 2. Run from the repo root: |
| 125 | ```bash |
| 126 | go generate ./src/go/plugin/ibm.d/modules/<m>/... |
| 127 | ``` |
| 128 | This invokes BOTH `metricgen` (on `contexts.yaml`) and |
| 129 | `docgen` (on the module). |
| 130 | 3. Commit ALL generated files together with the source change: |
| 131 | - `metadata.yaml` |
| 132 | - `README.md` |
| 133 | - `config_schema.json` |
| 134 | - `contexts/zz_generated_contexts.go` |
| 135 | 4. Run the integrations regen locally to update the |
| 136 | per-integration `.md` and the umbrella pages: |
| 137 | ```bash |
| 138 | ./integrations/pip.sh |
| 139 | python3 integrations/gen_integrations.py |
| 140 | python3 integrations/gen_docs_integrations.py -c ibm.d/<m> |
| 141 | python3 integrations/gen_doc_collector_page.py |
| 142 | python3 integrations/gen_doc_secrets_page.py |
| 143 | ``` |
| 144 | 5. Commit the regenerated `<plugin-dir>/integrations/<slug>.md` |
| 145 | and umbrella pages too, in the same PR. |
| 146 | |
| 147 | ## Why ibm.d is generated this way |
| 148 | |
| 149 | ibm.d collectors are typically heavy: many metrics, many |
| 150 | config fields, dense documentation. Generating ensures |
| 151 | consistency between: |
| 152 | - the runtime metric registration |
| 153 | (`zz_generated_contexts.go`), |
| 154 | - the integration metadata (`metadata.yaml`), |
| 155 | - the dashboard schema (`config_schema.json`), |
| 156 | - the user-facing documentation (`README.md`). |
| 157 | |
| 158 | It is the closest thing this repo has to enforcement of the |
| 159 | collector consistency rule for the integration-page side |
| 160 | (metadata + README + config_schema), but it does NOT cover |
| 161 | taxonomy.yaml, the stock `.conf`, or `health.d/<...>.conf` -- |
| 162 | those still need manual sync unless a module-specific generator |
| 163 | adds coverage. |
| 164 | |
| 165 | ## Risks and gotchas |
| 166 | |
| 167 | - **Hand edits to generated files are silently overwritten on |
| 168 | next `go generate`.** No warning. The DO-NOT-EDIT banner is |
| 169 | the only signal. |
| 170 | - **`module.yaml` is the right place for static prose** (e.g. |
| 171 | description text) that the metadata template inlines. Edits |
| 172 | to that file survive regeneration; edits to the generated |
| 173 | `metadata.yaml` do not. |
| 174 | - **The metadata template hardcodes some scaffolding** (e.g. |
| 175 | `endpoint: dummy://localhost`). Modules that need different |
| 176 | scaffolding must extend the template at |
| 177 | `docgen/main.go:562+` -- editing the generated `metadata.yaml` |
| 178 | is not a fix. |
| 179 | - **`go generate` does not auto-run `gen_integrations.py`**. |
| 180 | After regenerating ibm.d files, you still need to run the |
| 181 | integrations pipeline to refresh the per-integration `.md` |
| 182 | and umbrella pages. |
| 183 | - **`websphere/` sub-modules each have their own generation |
| 184 | cycle**. Running `go generate ./src/go/plugin/ibm.d/modules/websphere/...` |
| 185 | hits all three (`jmx`, `mp`, `pmi`). |