| 1 | # Automated Documentation Generator (docgen) |
| 2 | |
| 3 | The docgen tool automatically generates metadata.yaml, config_schema.json, and README.md files from the sources of truth in your IBM.D framework module. |
| 4 | |
| 5 | ## How It Works |
| 6 | |
| 7 | The tool extracts information from: |
| 8 | 1. **contexts.yaml** - Metric definitions, contexts, dimensions, families |
| 9 | 2. **config.go** - Configuration structure with field types and tags |
| 10 | 3. **module.yaml** - Module-specific metadata (name, description, categories, etc.) |
| 11 | |
| 12 | And generates: |
| 13 | 1. **metadata.yaml** - Complete Netdata marketplace metadata |
| 14 | 2. **config_schema.json** - JSON schema for web UI configuration |
| 15 | 3. **README.md** - Comprehensive documentation with metric tables and config options |
| 16 | |
| 17 | ## Usage |
| 18 | |
| 19 | ### Basic Usage |
| 20 | |
| 21 | ```bash |
| 22 | go run github.com/netdata/netdata/go/plugins/plugin/ibm.d/docgen \ |
| 23 | -module=mymodule \ |
| 24 | -contexts=contexts/contexts.yaml \ |
| 25 | -config=config.go \ |
| 26 | -module-info=module.yaml |
| 27 | ``` |
| 28 | |
| 29 | ### Using go:generate |
| 30 | |
| 31 | Add this to any Go file in your module: |
| 32 | |
| 33 | ```go |
| 34 | //go:generate go run ../../docgen -module=mymodule -contexts=contexts/contexts.yaml -config=config.go -module-info=module.yaml |
| 35 | ``` |
| 36 | |
| 37 | Then run: |
| 38 | |
| 39 | ```bash |
| 40 | go generate |
| 41 | ``` |
| 42 | |
| 43 | ## Required Files |
| 44 | |
| 45 | ### 1. contexts.yaml (Required) |
| 46 | |
| 47 | Your standard framework contexts file: |
| 48 | |
| 49 | ```yaml |
| 50 | System: |
| 51 | labels: [] |
| 52 | contexts: |
| 53 | - name: CPU |
| 54 | context: mymodule.cpu_usage |
| 55 | family: cpu |
| 56 | title: CPU Usage |
| 57 | units: percentage |
| 58 | type: line |
| 59 | priority: 1000 |
| 60 | dimensions: |
| 61 | - { name: usage, algo: absolute } |
| 62 | |
| 63 | Database: |
| 64 | labels: [name, type] |
| 65 | contexts: |
| 66 | - name: Connections |
| 67 | context: mymodule.db_connections |
| 68 | family: databases |
| 69 | title: Database Connections |
| 70 | units: connections |
| 71 | type: line |
| 72 | priority: 2000 |
| 73 | dimensions: |
| 74 | - { name: active, algo: absolute } |
| 75 | - { name: idle, algo: absolute } |
| 76 | ``` |
| 77 | |
| 78 | ### 2. config.go (Required) |
| 79 | |
| 80 | Your module configuration struct: |
| 81 | |
| 82 | ```go |
| 83 | type Config struct { |
| 84 | framework.Config `yaml:",inline" json:",inline"` |
| 85 | |
| 86 | Endpoint string `yaml:"endpoint,omitempty" json:"endpoint"` |
| 87 | Username string `yaml:"username,omitempty" json:"username"` |
| 88 | Password string `yaml:"password,omitempty" json:"password"` |
| 89 | |
| 90 | CollectDatabases bool `yaml:"collect_databases" json:"collect_databases"` |
| 91 | MaxDatabases int `yaml:"max_databases,omitempty" json:"max_databases"` |
| 92 | } |
| 93 | ``` |
| 94 | |
| 95 | The tool will automatically: |
| 96 | - Parse field types and convert to JSON Schema types |
| 97 | - Extract YAML/JSON tag names |
| 98 | - Determine required vs optional fields based on `omitempty` |
| 99 | - Generate appropriate descriptions and constraints |
| 100 | |
| 101 | ### 3. module.yaml (Optional) |
| 102 | |
| 103 | Module-specific metadata: |
| 104 | |
| 105 | ```yaml |
| 106 | name: mymodule |
| 107 | display_name: My Application Monitor |
| 108 | description: | |
| 109 | Monitors My Application metrics including performance, |
| 110 | connections, and resource utilization. |
| 111 | icon: myapp.svg |
| 112 | categories: |
| 113 | - data-collection.databases |
| 114 | - data-collection.apm |
| 115 | link: https://myapp.com/monitoring |
| 116 | keywords: |
| 117 | - database |
| 118 | - performance |
| 119 | - monitoring |
| 120 | ``` |
| 121 | |
| 122 | If this file doesn't exist, the tool creates reasonable defaults. |
| 123 | |
| 124 | ## Generated Files |
| 125 | |
| 126 | ### metadata.yaml |
| 127 | |
| 128 | Complete Netdata marketplace metadata including: |
| 129 | - Module information and categorization |
| 130 | - Metric scopes with proper labels and dimensions |
| 131 | - Configuration options with descriptions |
| 132 | - Setup and troubleshooting sections |
| 133 | |
| 134 | ### config_schema.json |
| 135 | |
| 136 | Valid JSON Schema for the Netdata web UI: |
| 137 | - Field types, defaults, and constraints |
| 138 | - Validation rules (min/max values) |
| 139 | - Example values |
| 140 | - Required vs optional fields |
| 141 | |
| 142 | ### README.md |
| 143 | |
| 144 | Comprehensive documentation with: |
| 145 | - Overview and description |
| 146 | - Metric tables organized by scope |
| 147 | - Configuration options with examples |
| 148 | - Troubleshooting and debug instructions |
| 149 | |
| 150 | ## Benefits |
| 151 | |
| 152 | ### 1. Perfect Synchronization |
| 153 | All generated files are 100% consistent with your code. No more: |
| 154 | - Mismatched context names between code and documentation |
| 155 | - Outdated configuration options in schemas |
| 156 | - Missing metrics in metadata.yaml |
| 157 | |
| 158 | ### 2. Automatic Updates |
| 159 | When you change contexts.yaml or config.go: |
| 160 | 1. Run `go generate` |
| 161 | 2. All documentation automatically updates |
| 162 | 3. Zero manual synchronization needed |
| 163 | |
| 164 | ### 3. Reduced Development Time |
| 165 | - 80%+ reduction in documentation maintenance |
| 166 | - No more manually writing 4-5 different files |
| 167 | - Focus on code, not documentation |
| 168 | |
| 169 | ### 4. Error Prevention |
| 170 | - AST parsing ensures accuracy |
| 171 | - Type-safe configuration schema generation |
| 172 | - Impossible to have documentation mismatches |
| 173 | |
| 174 | ## Integration with Framework |
| 175 | |
| 176 | The docgen tool is designed specifically for the IBM.D framework: |
| 177 | |
| 178 | - **Understands framework.Config** - Automatically excludes embedded framework fields |
| 179 | - **Supports precision/mul/div** - Correctly documents unit conversions |
| 180 | - **Handles dynamic instances** - Generates proper scope documentation for labeled metrics |
| 181 | - **Framework-aware defaults** - Knows about standard framework patterns |
| 182 | |
| 183 | ## Advanced Usage |
| 184 | |
| 185 | ### Custom Field Descriptions |
| 186 | |
| 187 | The tool includes smart defaults for common field names, but you can extend it: |
| 188 | |
| 189 | ```go |
| 190 | // Add field descriptions based on patterns |
| 191 | descriptions := map[string]string{ |
| 192 | "ConnectionString": "Database connection string in format server:port/database", |
| 193 | "QueryTimeout": "Maximum time to wait for query response in seconds", |
| 194 | "EnableSSL": "Use SSL/TLS encryption for connections", |
| 195 | } |
| 196 | ``` |
| 197 | |
| 198 | ### Module Categories |
| 199 | |
| 200 | Use standard Netdata categories in module.yaml: |
| 201 | |
| 202 | ```yaml |
| 203 | categories: |
| 204 | - data-collection.databases # Database systems |
| 205 | - data-collection.apm # Application Performance Monitoring |
| 206 | - data-collection.web-servers # Web servers |
| 207 | - data-collection.message-brokers # Message queues |
| 208 | - data-collection.generic # Generic/other |
| 209 | ``` |
| 210 | |
| 211 | ### Complex Field Types |
| 212 | |
| 213 | The tool handles various Go types: |
| 214 | |
| 215 | - `string` → `"string"` |
| 216 | - `int`, `int64` → `"integer"` |
| 217 | - `bool` → `"boolean"` |
| 218 | - `time.Duration` → `"integer"` (with seconds constraint) |
| 219 | - Custom types → `"string"` (fallback) |
| 220 | |
| 221 | ## Best Practices |
| 222 | |
| 223 | ### 1. Keep module.yaml Updated |
| 224 | |
| 225 | Always maintain module.yaml with: |
| 226 | - Accurate description |
| 227 | - Proper categorization |
| 228 | - Useful keywords for search |
| 229 | |
| 230 | ### 2. Use Descriptive Field Names |
| 231 | |
| 232 | Field names like `QueryTimeout` generate better documentation than `QTO`. |
| 233 | |
| 234 | ### 3. Add Examples to Configs |
| 235 | |
| 236 | The tool can extract examples from field comments or provide defaults. |
| 237 | |
| 238 | ### 4. Run on Every Change |
| 239 | |
| 240 | Add `go generate` to your development workflow: |
| 241 | |
| 242 | ```bash |
| 243 | # After modifying contexts.yaml or config.go |
| 244 | go generate |
| 245 | git add metadata.yaml config_schema.json README.md |
| 246 | ``` |
| 247 | |
| 248 | ### 5. Review Generated Docs |
| 249 | |
| 250 | The tool generates good defaults, but always review: |
| 251 | - Metric descriptions make sense |
| 252 | - Configuration examples are realistic |
| 253 | - Troubleshooting steps are appropriate |
| 254 | |
| 255 | ## Limitations |
| 256 | |
| 257 | ### Current Limitations |
| 258 | |
| 259 | 1. **Comment Parsing**: Doesn't extract field descriptions from Go comments yet |
| 260 | 2. **Complex Validations**: Basic min/max only, no regex patterns |
| 261 | 3. **Custom Examples**: Limited to hardcoded examples per field |
| 262 | |
| 263 | ### Future Enhancements |
| 264 | |
| 265 | 1. **AST Comment Parsing** - Extract documentation from Go comments |
| 266 | 2. **Validation Rules** - Support regex, enum values, conditional requirements |
| 267 | 3. **Custom Templates** - Allow module-specific documentation templates |
| 268 | 4. **Multi-Language** - Generate documentation in multiple languages |
| 269 | |
| 270 | ## Troubleshooting |
| 271 | |
| 272 | ### "Failed to parse Go file" |
| 273 | |
| 274 | - Ensure config.go has valid Go syntax |
| 275 | - Check that the Config struct is exported |
| 276 | - Verify YAML/JSON struct tags are properly formatted |
| 277 | |
| 278 | ### "No fields extracted" |
| 279 | |
| 280 | - Config struct must be named exactly "Config" |
| 281 | - Fields must be exported (start with capital letter) |
| 282 | - Framework embedded fields are automatically excluded |
| 283 | |
| 284 | ### "Template execution failed" |
| 285 | |
| 286 | - Check that contexts.yaml is valid YAML |
| 287 | - Ensure all required context fields are present |
| 288 | - Verify module.yaml follows the expected structure |
| 289 | |
| 290 | ### Generated Schema Invalid |
| 291 | |
| 292 | - Check JSON syntax with `jq . config_schema.json` |
| 293 | - Verify all field types are supported |
| 294 | - Ensure required field logic is correct |
| 295 | |
| 296 | ## Example Integration |
| 297 | |
| 298 | Review any existing production module (for example `modules/mq/`) for a complete integration: |
| 299 | |
| 300 | ``` |
| 301 | modules/mq/ |
| 302 | ├── contexts/contexts.yaml # Metric definitions |
| 303 | ├── config.go # Configuration struct |
| 304 | ├── module.yaml # Module metadata |
| 305 | ├── generate.go # go:generate directive |
| 306 | ├── metadata.yaml # Generated ✓ |
| 307 | ├── config_schema.json # Generated ✓ |
| 308 | └── README.md # Generated ✓ |
| 309 | ``` |
| 310 | |
| 311 | Run `go generate` in the module directory to see it in action. |