#Automated Documentation Generator (docgen)

The docgen tool automatically generates metadata.yaml, config_schema.json, and README.md files from the sources of truth in your IBM.D framework module.

#How It Works

The tool extracts information from: 1. contexts.yaml - Metric definitions, contexts, dimensions, families 2. config.go - Configuration structure with field types and tags 3. module.yaml - Module-specific metadata (name, description, categories, etc.)

And generates: 1. metadata.yaml - Complete Netdata marketplace metadata 2. config_schema.json - JSON schema for web UI configuration 3. README.md - Comprehensive documentation with metric tables and config options

#Usage

#Basic Usage

go run github.com/netdata/netdata/go/plugins/plugin/ibm.d/docgen \
  -module=mymodule \
  -contexts=contexts/contexts.yaml \
  -config=config.go \
  -module-info=module.yaml

#Using go:generate

Add this to any Go file in your module:

//go:generate go run ../../docgen -module=mymodule -contexts=contexts/contexts.yaml -config=config.go -module-info=module.yaml

Then run:

go generate

#Required Files

#1. contexts.yaml (Required)

Your standard framework contexts file:

System:
  labels: []
  contexts:
    - name: CPU
      context: mymodule.cpu_usage
      family: cpu
      title: CPU Usage
      units: percentage
      type: line
      priority: 1000
      dimensions:
        - { name: usage, algo: absolute }

Database:
  labels: [name, type]
  contexts:
    - name: Connections
      context: mymodule.db_connections
      family: databases
      title: Database Connections
      units: connections
      type: line
      priority: 2000
      dimensions:
        - { name: active, algo: absolute }
        - { name: idle, algo: absolute }

#2. config.go (Required)

Your module configuration struct:

type Config struct {
    framework.Config `yaml:",inline" json:",inline"`

    Endpoint string `yaml:"endpoint,omitempty" json:"endpoint"`
    Username string `yaml:"username,omitempty" json:"username"`
    Password string `yaml:"password,omitempty" json:"password"`

    CollectDatabases bool `yaml:"collect_databases" json:"collect_databases"`
    MaxDatabases     int  `yaml:"max_databases,omitempty" json:"max_databases"`
}

The tool will automatically: - Parse field types and convert to JSON Schema types - Extract YAML/JSON tag names - Determine required vs optional fields based on omitempty - Generate appropriate descriptions and constraints

#3. module.yaml (Optional)

Module-specific metadata:

name: mymodule
display_name: My Application Monitor
description: |
  Monitors My Application metrics including performance,
  connections, and resource utilization.
icon: myapp.svg
categories:
  - data-collection.databases
  - data-collection.apm
link: https://myapp.com/monitoring
keywords:
  - database
  - performance
  - monitoring

If this file doesn't exist, the tool creates reasonable defaults.

#Generated Files

#metadata.yaml

Complete Netdata marketplace metadata including: - Module information and categorization - Metric scopes with proper labels and dimensions - Configuration options with descriptions - Setup and troubleshooting sections

#config_schema.json

Valid JSON Schema for the Netdata web UI: - Field types, defaults, and constraints - Validation rules (min/max values) - Example values - Required vs optional fields

#README.md

Comprehensive documentation with: - Overview and description - Metric tables organized by scope - Configuration options with examples - Troubleshooting and debug instructions

#Benefits

#1. Perfect Synchronization

All generated files are 100% consistent with your code. No more: - Mismatched context names between code and documentation - Outdated configuration options in schemas - Missing metrics in metadata.yaml

#2. Automatic Updates

When you change contexts.yaml or config.go: 1. Run go generate 2. All documentation automatically updates 3. Zero manual synchronization needed

#3. Reduced Development Time

• 80%+ reduction in documentation maintenance
• No more manually writing 4-5 different files
• Focus on code, not documentation

#4. Error Prevention

• AST parsing ensures accuracy
• Type-safe configuration schema generation
• Impossible to have documentation mismatches

#Integration with Framework

The docgen tool is designed specifically for the IBM.D framework:

• Understands framework.Config - Automatically excludes embedded framework fields
• Supports precision/mul/div - Correctly documents unit conversions
• Handles dynamic instances - Generates proper scope documentation for labeled metrics
• Framework-aware defaults - Knows about standard framework patterns

#Advanced Usage

#Custom Field Descriptions

The tool includes smart defaults for common field names, but you can extend it:

// Add field descriptions based on patterns
descriptions := map[string]string{
    "ConnectionString": "Database connection string in format server:port/database",
    "QueryTimeout":     "Maximum time to wait for query response in seconds",
    "EnableSSL":        "Use SSL/TLS encryption for connections",
}

#Module Categories

Use standard Netdata categories in module.yaml:

categories:
  - data-collection.databases      # Database systems
  - data-collection.apm           # Application Performance Monitoring  
  - data-collection.web-servers   # Web servers
  - data-collection.message-brokers # Message queues
  - data-collection.generic       # Generic/other

#Complex Field Types

The tool handles various Go types:

• string → "string"
• int, int64 → "integer"
• bool → "boolean"
• time.Duration → "integer" (with seconds constraint)
• Custom types → "string" (fallback)

#Best Practices

#1. Keep module.yaml Updated

Always maintain module.yaml with: - Accurate description - Proper categorization - Useful keywords for search

#2. Use Descriptive Field Names

Field names like QueryTimeout generate better documentation than QTO.

#3. Add Examples to Configs

The tool can extract examples from field comments or provide defaults.

#4. Run on Every Change

Add go generate to your development workflow:

# After modifying contexts.yaml or config.go
go generate
git add metadata.yaml config_schema.json README.md

#5. Review Generated Docs

The tool generates good defaults, but always review: - Metric descriptions make sense - Configuration examples are realistic - Troubleshooting steps are appropriate

#Limitations

#Current Limitations

1. Comment Parsing: Doesn't extract field descriptions from Go comments yet
2. Complex Validations: Basic min/max only, no regex patterns
3. Custom Examples: Limited to hardcoded examples per field

#Future Enhancements

1. AST Comment Parsing - Extract documentation from Go comments
2. Validation Rules - Support regex, enum values, conditional requirements
3. Custom Templates - Allow module-specific documentation templates
4. Multi-Language - Generate documentation in multiple languages

#Troubleshooting

#"Failed to parse Go file"

• Ensure config.go has valid Go syntax
• Check that the Config struct is exported
• Verify YAML/JSON struct tags are properly formatted

#"No fields extracted"

• Config struct must be named exactly "Config"
• Fields must be exported (start with capital letter)
• Framework embedded fields are automatically excluded

#"Template execution failed"

• Check that contexts.yaml is valid YAML
• Ensure all required context fields are present
• Verify module.yaml follows the expected structure

#Generated Schema Invalid

• Check JSON syntax with jq . config_schema.json
• Verify all field types are supported
• Ensure required field logic is correct

#Example Integration

Review any existing production module (for example modules/mq/) for a complete integration:

modules/mq/
├── contexts/contexts.yaml    # Metric definitions
├── config.go                 # Configuration struct
├── module.yaml               # Module metadata
├── generate.go               # go:generate directive
├── metadata.yaml             # Generated ✓
├── config_schema.json        # Generated ✓
└── README.md                 # Generated ✓

Run go generate in the module directory to see it in action.