| 1 | # IBM.D Collector Framework |
| 2 | |
| 3 | The IBM.D framework is a thin layer on top of Netdata’s go.d engine that makes it easy – and predictable – to implement complex collectors. It focuses on: |
| 4 | |
| 5 | - **Type-safe metric exports** generated from declarative context definitions |
| 6 | - **Consistent configuration** driven by Go structs and JSON Schema |
| 7 | - **Automation** so documentation, metadata, and stock alerts stay in sync |
| 8 | |
| 9 | The framework is used by every collector inside `modules/` and is designed so AI assistants can safely extend it. |
| 10 | |
| 11 | ## Core Concepts |
| 12 | |
| 13 | ### 1. Contexts and Families |
| 14 | |
| 15 | Module metrics are described in `contexts/contexts.yaml`. Each entry declares the Netdata context name, family hierarchy, units, chart priority, and dimensions. Example: |
| 16 | |
| 17 | ```yaml |
| 18 | System: |
| 19 | labels: [] |
| 20 | contexts: |
| 21 | - name: CPUUtilization |
| 22 | context: as400.cpu_utilization |
| 23 | family: compute/cpu |
| 24 | units: percentage |
| 25 | type: line |
| 26 | priority: 101 |
| 27 | dimensions: |
| 28 | - { name: utilization, algo: absolute, div: 1000 } |
| 29 | ``` |
| 30 | |
| 31 | The generator produces strongly typed Go setters at `contexts/zz_generated_contexts.go`, so collectors can export metrics without stringly-typed code. |
| 32 | |
| 33 | ### 2. Collector Skeleton |
| 34 | |
| 35 | Each module embeds `framework.Collector` for lifecycle management: |
| 36 | |
| 37 | ```go |
| 38 | type Collector struct { |
| 39 | framework.Collector |
| 40 | |
| 41 | Config `yaml:",inline" json:",inline"` |
| 42 | client *protocol.Client |
| 43 | mx *metricsData |
| 44 | } |
| 45 | ``` |
| 46 | |
| 47 | Implement these hooks: |
| 48 | |
| 49 | - `InitOnce()` to allocate caches and parse configuration defaults |
| 50 | - `Collect(ctx)` to call protocol clients and populate the typed context setters |
| 51 | - Optional `Cleanup()` for protocol tear-down |
| 52 | |
| 53 | The base struct provides logging, state, and convenience helpers for instance tracking. |
| 54 | |
| 55 | ### 3. Configuration |
| 56 | |
| 57 | Configuration structs embed `framework.Config` and define module-specific options: |
| 58 | |
| 59 | ```go |
| 60 | type Config struct { |
| 61 | framework.Config `yaml:",inline" json:",inline"` |
| 62 | |
| 63 | Endpoint string `yaml:"endpoint" json:"endpoint"` |
| 64 | Timeout confopt.Duration `yaml:"timeout" json:"timeout"` |
| 65 | |
| 66 | MaxEntities int `yaml:"max_entities" json:"max_entities"` |
| 67 | MatchEntities matcher.Simple `yaml:"match_entities" json:"match_entities"` |
| 68 | } |
| 69 | ``` |
| 70 | |
| 71 | Run `go generate` (see below) and docgen will emit `config_schema.json` and README tables automatically. |
| 72 | |
| 73 | ### 4. Protocols and Shared Packages |
| 74 | |
| 75 | Framework collectors focus on orchestration. Low-level APIs live under: |
| 76 | |
| 77 | - `protocols/` – HTTP/OpenMetrics, PMI XML, JMX helper, MQ PCF, etc. |
| 78 | - `pkg/` – CGO shims (ODBC bridge, DB2 helper libraries). |
| 79 | |
| 80 | Protocols return typed data structures so collectors can be implemented with straightforward loops. |
| 81 | |
| 82 | ## Tooling |
| 83 | |
| 84 | The following generators keep modules aligned: |
| 85 | |
| 86 | | Tool | Location | Purpose | |
| 87 | |------|----------|---------| |
| 88 | | `docgen` | `../docgen` | Generates README, metadata.yaml, and config_schema.json | |
| 89 | | `metricgen` | `../metricgen` | Optional helper to scaffold metric exports | |
| 90 | | `go generate` directives | module directories | Invoke docgen + context generation | |
| 91 | |
| 92 | ### Source of Truth vs Generated Artifacts |
| 93 | |
| 94 | Author **only** the following files manually: |
| 95 | |
| 96 | - `contexts/contexts.yaml` |
| 97 | - Go sources (`config.go`, `collector.go`, `collect_*.go`, `module.go`, `init.go`) |
| 98 | - `module.yaml` and module-specific health alerts/extra docs |
| 99 | |
| 100 | Everything else is generated: |
| 101 | |
| 102 | - `contexts/zz_generated_contexts.go` (and other `contexts/zz_*` helpers) |
| 103 | - `metadata.yaml`, `config_schema.json`, and `README.md` |
| 104 | |
| 105 | To update generated files, run `go generate` in the module directory. Any manual edits to the generated outputs will be overwritten the next time docgen runs, so keep tweaks in the source files above. If additional prose is needed, place it in `module.yaml` (description, prerequisites, troubleshooting) or create a separate developer-facing document; the generated README is intended for users and should remain fully automated. |
| 106 | |
| 107 | Typical `go:generate` directives for a module: |
| 108 | |
| 109 | ```go |
| 110 | //go:generate go run ../../docgen -module=as400 -contexts=contexts/contexts.yaml -config=config.go -module-info=module.yaml |
| 111 | //go:generate go run ../../metricgen -module=as400 -contexts=contexts/contexts.yaml -out=contexts/zz_generated_contexts.go |
| 112 | ``` |
| 113 | |
| 114 | After editing `contexts.yaml`, `config.go`, or `module.yaml` run: |
| 115 | |
| 116 | ```bash |
| 117 | cd src/go/plugin/ibm.d/modules/<module> |
| 118 | go generate ./... |
| 119 | ``` |
| 120 | |
| 121 | ## Writing a New Collector |
| 122 | |
| 123 | 1. **Create module scaffold** under `modules/<name>/` using an existing module as a template. |
| 124 | 2. **Define contexts** in `contexts/contexts.yaml` and labels that describe your metrics. |
| 125 | 3. **Model configuration** in `config.go` – include sensible defaults and cardinality controls (`MaxX`, `MatchX`). |
| 126 | 4. **Implement protocol client(s)** if one doesn’t exist yet (place them in `protocols/<domain>/`). |
| 127 | 5. **Implement collector.go**: |
| 128 | - Parse config in `InitOnce` |
| 129 | - Call protocols in `Collect` |
| 130 | - Export metrics via the generated context setters |
| 131 | 6. **Run generators** (`go generate`) and review README, metadata, schema output. |
| 132 | 7. **Add stock alerts** under `src/health/health.d/` targeting contexts with safe thresholds. |
| 133 | 8. **Document** any runtime prerequisites (CGO libraries, environment variables) in the module README and metadata. |
| 134 | |
| 135 | ## Runtime Integration |
| 136 | |
| 137 | - Modules register themselves in `init()` using `framework.RegisterModule` (see existing modules for examples). |
| 138 | - The IBM plugin loads `/etc/netdata/ibm.d/<module>.conf` and constructs jobs according to the schema. |
| 139 | - Health alerts and dashboards automatically pick up the generated contexts; keep names stable. |
| 140 | |
| 141 | ## Debugging Tips |
| 142 | |
| 143 | - Run the plugin in dump mode: `script -c 'sudo /usr/libexec/netdata/plugins.d/ibm.d.plugin -d -m MODULE --dump=3s --dump-summary 2>&1' /dev/null` |
| 144 | - The summary tree should match what is declared in `contexts.yaml`. |
| 145 | - Use the framework logger (`c.Infof`, `c.Warningf`, etc.) for human-friendly messages when a feature is unavailable. |
| 146 | |
| 147 | ## Additional Resources |
| 148 | |
| 149 | - [`../README.md`](../README.md) – project overview, build instructions, and directory map |
| 150 | - [`../AGENTS.md`](../AGENTS.md) – authoring checklist and best practices for AI assistants |
| 151 | - [`../../../AGENTS.md`](../../../AGENTS.md) – Go-area rules and routing |
| 152 | |
| 153 | Contributions are welcome! Keep documentation, schemas, metadata, and health alerts synchronized to guarantee a smooth user experience. |