feat(go.d/sd): add dyncfg support for service discovery (#21680)

* refactor(go.d/sd): extract PipelineManager and move FunctionRegistry Refactoring in preparation for adding dyncfg support to service discovery: - Move FunctionRegistry interface from jobmgr/di.go to functions/registry.go so both jobmgr and SD can use it without duplication. - Extract PipelineManager from ServiceDiscovery to handle: - Pipeline lifecycle (start, stop, restart) - Source tracking for cleanup when pipelines are disabled - Grace period mechanism (30s) for restarts to avoid gaps in metrics - Group forwarding with source interception * feat(go.d/sd): wire FunctionRegistry to service discovery Add FnReg to SD config and basic dyncfg infrastructure wiring: - Add functions.Registry parameter to buildDiscoveryConf - Add FnReg and ConfigDefaults to sd.Config - Initialize dyncfg.Responder in ServiceDiscovery - Move fnMgr creation before buildDiscoveryConf to pass it through * feat(go.d/sd): add dyncfg template registration for discoverers Add dyncfg infrastructure for service discovery: - Create dyncfg.go with template registration for discoverer types (net_listeners, docker, k8s, snmp) - Define ID format: {executable}:sd:{discovererType}:{name} - Define supported commands for templates and jobs - Register templates on SD startup, unregister on shutdown - Add placeholder config handler (commands will be implemented next) * feat(go.d/sd): add state cache for SD dyncfg Add exposedSDConfigs cache to track SD pipeline configs exposed via dyncfg: - sdConfig struct to represent pipeline config with discoverer type, name, source info, status, and raw content - Thread-safe exposedSDConfigs cache with lookup by type:name key - Helper methods for iteration and counting by discoverer type * feat(go.d/sd): implement dyncfg command handlers Implement dyncfg command handlers for service discovery: - schema: return JSON schema for discoverer type (placeholder for now) - get: return current config for a job - add: create a new dyncfg job - update: update an existing job config - enable: enable a job (change status to running) - disable: disable a job (change status to disabled) - remove: remove a dyncfg job Add helper functions: - extractDiscovererAndName: parse dyncfg ID into type and name - isValidDiscovererType: validate discoverer type - getDyncfgCommand: extract command from function args - getFnSourceValue: extract value from function source Pipeline start/stop is stubbed with TODOs for follow-up work. * feat(go.d/sd): add JSON schemas for discoverer types Add JSON schemas for dyncfg UI configuration forms: - net_listeners: interval, timeout, tags, services - docker: address, timeout, tags, services - k8s: role, namespaces, selector, pod options, tags, services - snmp: credentials, networks, rescan_interval, timeout, services Each schema includes: - Field descriptions and validation - Service rule definitions with match expressions and config templates - Appropriate defaults and examples * feat(go.d/sd): expose file configs through dyncfg Add functionality to expose file-based pipeline configs via dyncfg: - exposeFileConfig: creates dyncfg job when file config is loaded - removeExposedFileConfig: removes dyncfg job when file is removed - Extracts discoverer type from config to create correct dyncfg job ID - File configs are exposed with sourceType="file" and status=running - Fix test to initialize exposedConfigs cache * refactor(go.d/sd): extract config schemas to JSON files Move JSON schemas from Go string constants to separate files: - config_schema_net_listeners.json - config_schema_docker.json - config_schema_k8s.json - config_schema_snmp.json Use go:embed to load schemas at compile time, matching the pattern used by collectors for their config_schema.json files. * feat(go.d/sd): improve dyncfg config handling and add userconfig - Fix JSON schemas to use proper jsonSchema + uiSchema format for react-jsonschema-form compatibility - Add typed dyncfg config structs (DyncfgNetListenersConfig, DyncfgDockerConfig, DyncfgK8sConfig, DyncfgSNMPConfig) - Transform file configs to dyncfg format on load for consistent UI presentation - Add userconfig command for templates and jobs to return YAML representation - Remove convertToJSONCompatible hack in favor of proper structs * fix(go.d/sd): address review comments for dyncfg and pipeline manager - Fix potential deadlock in cleanup: use timeout context instead of context.Background() in cleanupSourcesLocked to prevent blocking forever if consumer stopped reading - Return 501 Not Implemented for unimplemented dyncfg commands: update, enable, disable, remove now return 501 instead of misleading 200 OK since pipeline control is not yet wired up - Fix lock contention in grace period cleanup: collect removals while holding lock, then send outside lock to avoid blocking other operations - Fix lock contention during pipeline shutdown: refactor to wait for pipeline stop outside the lock, preventing freeze of entire PipelineManager during shutdown (up to 10 seconds) * fix(go.d/sd): fix pipeline manager types and schema issues - Add missing waitForPipeline method for restart grace period - Fix Stop/StopAll to use waitAndCleanup with correct types - Merge pending removals on consecutive restarts to avoid leaks - Fix SNMP schema: aes192C/aes256C -> aes192c/aes256c - Fix generic schema to use jsonSchema/uiSchema wrapper format - Remove stale TODO comment from dyncfg.go * fix(go.d/sd): fix data races and implement dyncfg command serialization - Fix data races in dyncfg command handlers by using thread-safe getter methods instead of lookup() followed by field access - Add thread-safe getter methods to exposedSDConfigs: getContent(), getPipelineKey(), getSource(), getSourceType(), getStatus() - Fix Source field for file-based configs on enable (use "file=" prefix) - Implement dyncfg command serialization following jobmgr pattern: - Read-only commands (schema, get, userconfig) execute directly - State-changing commands queue via dyncfgCh channel - Commands processed serially in run() goroutine - Add pipelineKey field to sdConfig for correct pipeline identification - Implement enable, disable, update, remove commands - Add duplicate check in add command - Remove enable/disable from template commands (not needed) - Add dyncfg_parse.go for parsing dyncfg JSON payloads * refactor(go.d/dyncfg): extract shared utility functions Move GetCommand() and GetSourceValue() to the dyncfg package to eliminate code duplication between sd and jobmgr dyncfg handlers. - Add dyncfg.GetCommand() to extract command from function args - Add dyncfg.GetSourceValue() to extract values from function source - Update sd/dyncfg.go to use shared functions - Update jobmgr/dyncfg_collector.go to use shared functions - Remove duplicate local function definitions * refactor(go.d/dyncfg): add Function wrapper type with accessor methods Add dyncfg.Function wrapper that encapsulates functions.Function and provides dyncfg-specific accessor methods for cleaner API: - Fn() returns underlying functions.Function for responder calls - UID(), Source(), Payload(), ContentType() for direct field access - ID(), Command(), JobName(), User() for dyncfg-specific arg extraction - ValidateArgs(), ValidateHasPayload() for validation - UnmarshalPayload(), UnmarshalJSON(), UnmarshalYAML() for payload handling Update sd/dyncfg.go to use the new wrapper type throughout all command handlers, replacing direct field access with method calls. * fix(go.d/sd): address review feedback for dyncfg handlers Fix issues identified in code review: 1. Update command now preserves original source type (file vs dyncfg) instead of always overwriting to dyncfg source 2. Transform failure no longer caches raw YAML content that would be incorrectly served as JSON - config is not exposed if transform fails 3. Enable command now refreshes status to 'running' even when pipeline is already running, fixing stale status display in UI 4. Duration parsing now returns 400 errors with details on invalid values instead of silently falling back to defaults * refactor(go.d/jobmgr): use dyncfg.Function wrapper type consistently Update jobmgr package to use the dyncfg.Function wrapper type instead of directly accessing functions.Function fields. Changes: - Update dyncfg_vnode.go to use accessor methods (ID(), JobName(), Source(), Command(), HasPayload(), UnmarshalPayload(), ValidateArgs()) - Update dyncfg_collector.go to use fn.UnmarshalPayload() - Add handler wrapper in dyncfg.go to convert at entry point - Update manager.go to wrap functions.Function with dyncfg.NewFunction() - Update tests to use dyncfg.NewFunction() wrapper This aligns the jobmgr package with the dyncfg.Function pattern established in the sd package refactoring. * refactor(go.d/dyncfg): update Responder to accept Function wrapper Update dyncfg.Responder methods to accept dyncfg.Function instead of functions.Function for consistency within the package. Changes: - Update Responder methods (SendCodef, SendJSON, SendJSONWithCode, SendYAML) to accept Function wrapper type - Remove redundant GetCommand() and GetSourceValue() helper functions (now available as methods on Function) - Simplify callers by passing fn directly instead of fn.Fn() - Add dyncfg.NewFunction() wrapper at funcshandler.go boundary This makes the dyncfg package internally consistent - both the Function wrapper and Responder now work with the same type. * test(go.d/sd): add PipelineManager and dyncfg integration tests Add comprehensive tests for ServiceDiscovery: PipelineManager tests (pipeline_manager_test.go): - Start: new pipeline, replace existing, invalid config - Stop: removal groups for tracked sources, non-existent no-op - Restart: grace period for overlapping sources, invalid config - StopAll: stops all pipelines, sends removal groups - RunGracePeriodCleanup: expired/non-expired pending removals - IsRunning/Keys: state verification - ConcurrentOperations: thread safety Dyncfg integration tests (dyncfg_test.go): - Schema: template and unknown discoverer type - Add: new job, missing payload, duplicate detection - Get: existing and non-existent jobs - Enable/Disable: pipeline lifecycle management - Update: existing and non-existent jobs - Remove: dyncfg jobs, running jobs, non-existent jobs - Userconfig: template and job configurations - FileConfig: prevent removal of file-based configs - MultipleJobs: complete lifecycle with multiple jobs * feat(go.d/sd): improve dyncfg schema UI for service discovery - Add tabs layout (Base + Services) for all discoverer types - Add textarea widget for match field in service rules - Add list flavour for services, credentials, and networks arrays - Remove unused 'tags' config option (legacy from old classify/compose format) - Add markdown documentation for match field with available target fields - SNMP schema has 4 tabs: Base, Credentials, Networks, Services * test(go.d/sd): add tests for docker/k8s/snmp configs and update-while-running Add comprehensive dyncfg tests for: - Docker config parsing (add, enable, get) - Kubernetes config parsing (pod role, service role, selectors) - SNMP config parsing (v2c, v3 with auth/priv) - Update command while pipeline is running (verifies restart) Also implement the "test" dyncfg command that validates config payloads without creating jobs or starting pipelines. * add snmp version 2c * style(go.d/sd): fix gofmt alignment in pipeline_manager.go * refactor(go.d/sd): use non-pointer Duration fields and fix JSON unmarshal - Fix confopt.Duration.UnmarshalJSON to handle both JSON numbers (30) and quoted strings ("30s", "5m") - Change all Duration fields from pointer to non-pointer throughout: - Dyncfg configs (Interval, Timeout, RescanInterval, DeviceCacheTTL) - Discoverer configs (netlistensd, snmpsd) - Update discoverer code to use zero-means-default pattern: - Zero = use default value - Positive = use specified value - Negative (snmpsd only) = special meaning (disable rescan/cache never expires) - Increase grace period from 30 seconds to 1 minute - Update tests for new Duration semantics * refactor(go.d/sd): remove legacy Tags field from discoverer configs The Tags field was only used by the legacy Classify/Compose system. The new Services format uses match expressions directly against target fields, making discoverer-level tags unnecessary. - Remove Tags field from netlistensd, dockersd, k8ssd configs - Remove tags parsing and propagation code from discoverers - Remove tags: "unknown" from file-based SD configs - Update tests to not expect tags on discovered targets * fix(go.d/sd): fix race conditions in dyncfg update and pipeline shutdown - Fix update storing config before validating restart success - Fix unbuffered dyncfgCh causing potential shutdown hang - Fix race where removed pipeline could orphan source tracking entries * fix(go.d/dyncfg): rename UnmarshalJSON to avoid interface conflict go vet reports error when method named UnmarshalJSON doesn't match the json.Unmarshaler interface signature. Renamed to private methods since they're only used internally. * fix(go.d/sd): validate config on add and prevent file config name collisions - Validate dyncfg payload in add command before storing - Prevent name collisions for file configs by using source path when another config with same discovererType:name already exists * fix(go.d/sd): address review findings for race conditions and validation - Fix race in Restart: cancel existing pipeline before starting new one in startPipelineWithInstanceLocked to prevent orphaned goroutines - Fix panic risk: use > 0 instead of != 0 for duration checks in netlistensd and dockersd to reject negative values - Fix stale dyncfg jobs: remove existing jobs before exposing updated file config to handle name/type changes - Improve concurrent test: track created vs stopped pipelines to detect goroutine leaks * refactor(go.d/sd): extract shared functions and fix naming consistency - Extract duplicated utilities to model/funcs.go: - CalcHash: centralized hash calculation using hashstructure - MapAny: convert string maps to any maps - SendTargetGroup: context-aware target group sending - Update all discoverers to use shared functions: - netlistensd: use model.CalcHash - dockersd: use model.CalcHash and model.MapAny - k8ssd: use model.CalcHash, model.MapAny, model.SendTargetGroup - snmpsd: use model.CalcHash - Fix naming inconsistency: - Rename k8ssd.NewKubeDiscoverer to k8ssd.NewDiscoverer - All discoverers now consistently use NewDiscoverer - Fix pipeline manager race: - Preserve interloper pipeline sources in pendingRemovals - Prevents orphaned jobs when race occurs during Restart * refactor(go.d/sd): remove unnecessary race check in pipeline manager Remove defensive check for concurrent Start/Restart on same pipeline key. This race cannot occur because ServiceDiscovery.run() processes all events (file config changes and dyncfg commands) sequentially in a single select loop. Update test to verify valid concurrent operations (different keys) instead of unsupported same-key concurrency. * style(go.d/sd): go fmt discoverers * feat(go.d/sd): implement wait-for-enable pattern for file configs - Add wait-for-enable flow matching jobmgr behavior: expose file configs with Accepted status and wait for netdata's enable/disable command - Add terminal detection using isatty to auto-enable in terminal mode - Add name sanitization (spaces/colons to underscores) via: - pipeline.Config.CleanName() for SD configs - dyncfg.Function.JobName() for dyncfg add commands - Fix dyncfg ID parsing issues caused by spaces/colons in names * refactor(go.d/sd): unify dyncfg and file config formats Restructure dyncfg config types to include discoverer wrapper, matching JSON schema and file-based config formats. This allows users to copy YAML from dyncfg userconfig action and paste it directly into configuration files. Changes: - Add discoverer wrapper to dyncfg config types - Update transform functions to produce new format - Update parse functions to read new format - Add automatic legacy format conversion during unmarshal - Update JSON schemas to match unified format - Update tests with helper functions for new format * refactor(go.d/sd): remove dyncfg config duplication Use pipeline.Config directly for dyncfg serialization instead of maintaining separate duplicate types. Changes: - Add JSON tags to discoverer configs (netlistensd, dockersd, k8ssd, snmpsd) - Add JSON tags to pipeline.Config (exclude internal fields) - Remove dyncfg_config.go (duplicate types no longer needed) - Remove dyncfg_transform.go (use json.Marshal directly) - Simplify dyncfg_parse.go (unmarshal into pipeline.Config directly) - Update tests to use actual types from their packages * fix(go.d/sd): add JSON tags to ServiceRuleConfig * refactor(go.d/sd): implement dual cache and priority handling - Add dual cache system (seenConfigs + exposedConfigs) matching jobmgr pattern - Implement config priority: dyncfg (16) > user file (8) > stock file (2) - Add name sanitization for dyncfg IDs (replace spaces/colons with underscores) - Use Restart when pipeline already running (validates first, preserves old if new fails) - Change sdConfig to map[string]any with __ metadata fields - Add newSDConfigFromYAML and newSDConfigFromJSON with name cleaning - Update tests to use proper discoverer configs instead of special handling * fix(go.d/sd): fix data race and add priority tests - Fix data race: lookup methods now return clones instead of references - Fix Clone() to use JSON marshal/unmarshal for proper deep copy - Fix Status() to handle both dyncfg.Status and string after JSON clone - Fix sourceTypeFromPath to correctly identify stock vs user paths - Add priority tests for stock/user config combinations - Add dyncfg priority tests for file vs dyncfg scenarios * refactor(go.d/sd): restore all metadata in Clone() Move type restoration from Status() getter to Clone() method. Restore all metadata keys after JSON unmarshal for consistency. * fix(go.d/sd): add full validation to dyncfg add/test commands - Export ValidateConfig in pipeline package for dyncfg validation - Add full semantic validation in parseDyncfgPayload (name, discoverer, services rules) - Reject empty discoverer upfront instead of at enable time - Add minItems:1 to credentials/networks in SNMP schema - Add services to required fields in all SD schemas - Update tests to use valid configs with service rules * fix(go.d/sd): review fixes for dyncfg and pipeline - dyncfg update: convert file config to dyncfg (creates persistent override) - dyncfg update: clean old dyncfg from seenConfigs, keep file for re-exposure - dyncfg_cache: add nil map check after JSON unmarshal - pipeline: remove dead legacy code in New() (services always required) * remove legacy code from sd pipeline * fix(go.d/sd): fix name mismatch and error handling in dyncfg update - Force name from dyncfg job ID onto sdConfig (matching jobmgr pattern) This ensures sdConfig.Key() matches the dyncfg job ID regardless of payload content, fixing lookups for enable/update/remove commands. - Fix dyncfgCmdUpdate error handling: - On conversion start failure: set status to Failed, update dyncfg UI, return 200 (user can retry via enable) - On restart failure: set status to Failed, update dyncfg UI, return 200 (was returning 422 without status update) - Remove defer for dyncfgSDJobCreate (captured status at wrong time) - Add TODO for promotion of lower-priority config after removal * some fixes * refactor(go.d/sd): align userconfig command with jobmgr pattern - Always require payload for userconfig (both templates and jobs) - Unmarshal JSON into pipeline.Config, then marshal to YAML - This ensures consistent field ordering and validates structure - Add userConfigFromPayload helper in dyncfg_parse.go - Update test to send payload for jobs * refactor(sd/dyncfg): align get command with jobmgr pattern - Add configToJSON helper that unmarshals JSON into pipeline.Config then marshals back to JSON for consistent field ordering - Update dyncfgCmdGet to use configToJSON instead of raw DataJSON() - This matches jobmgr's pattern of using typed structs for marshaling * refactor(sd/dyncfg): align add command with jobmgr pattern Changes to dyncfgCmdAdd: - Add job name validation (reject spaces, '.', ':' characters) - Move Info log after all validation, before cache operations - Replace duplicate handling with jobmgr pattern: - Always allow replacing existing config - Only remove from seenConfigs if existing is dyncfg - File configs stay in seenConfigs for potential re-exposure - Send 202 response before creating dyncfg job (matching jobmgr order) Add validateJobName helper to dyncfg_parse.go. Update tests to reflect new behavior: - Duplicate add now replaces instead of failing - Response sent before job create - No CONFIG delete when replacing (CONFIG create updates existing) * refactor(sd/dyncfg): use sdConfig for cache lookups instead of strings - Change cache lookup/remove methods to accept sdConfig, derive key internally - Add newLookupConfig helper for creating minimal lookup configs - Remove lookupByName method in favor of consistent lookup(cfg) pattern - Simplify add command: remove redundant IsRunning check before Stop * refactor(sd/dyncfg): remove redundant IsRunning check in remove command * refactor(sd/dyncfg): align enable command with jobmgr pattern - Add status check: Running returns early (idempotent), Accepted/Disabled/Failed proceed, others reject with 405 - Only log "enable by user" when status was Disabled - Use simple Start() without graceful degradation - config should reflect current file state - Response before status update (matches jobmgr order) * refactor(sd/dyncfg): align disable command with jobmgr pattern - Use status check instead of IsRunning: Disabled returns early (idempotent), Running stops pipeline, others proceed - Log "disable by user" once at the end - Response before status update (matches jobmgr order) * refactor(sd/dyncfg): align update command with jobmgr pattern - Use hash-based comparison instead of raw JSON string comparison for detecting unchanged configs (JSON field order is not guaranteed) - Add Hash() and HashIncludeMap() methods to sdConfig using hashstructure - Fix response order: send response before status update - Skip unchanged config optimization only for dyncfg-to-dyncfg updates (file-to-dyncfg conversions must always proceed) * refactor(sd/dyncfg): align update/remove with jobmgr patterns - Block update in Accepted state (return 403), matching jobmgr behavior - Fix remove response order: response before delete (matching jobmgr) - Add test for update in accepted state fails scenario - Update test expectations for new response order * refactor(sd/dyncfg): use Restart for non-conversion updates Use mgr.Restart() instead of Stop()+Start() for non-conversion dyncfg updates. Restart validates new config before stopping old pipeline and uses grace period for seamless transition. * test(go.d/sd): add missing dyncfg test cases and remove unused methods - Remove unused forEach/forEachBySource methods from exposedSDConfigs - Add test: update running pipeline with same config (hash optimization) - Add test: update config in Failed state - Add test: enable config from Failed state - Add test: file->dyncfg conversion via update command - Add test: restart error handling (old pipeline keeps running) - Add test: file removal with dyncfg override * refactor(go.d/sd): fix schema durations and add test command for jobs - Fix JSON schema files to use numeric duration values (seconds) instead of string patterns, matching confopt.Duration's JSON marshaling - Add test command to job dyncfg commands for config validation - Update dyncfgCmdTest to handle both templates and jobs with appropriate logging * feat(confopt): add LongDuration type for human-friendly duration strings Add LongDuration type that marshals to human-friendly strings ("12h", "1d", "30m") instead of raw seconds. Supports all duration units: y, mo, w, d, h, m, s, ms. Update SD discoverer configs to use LongDuration for interval/TTL fields: - netlistensd.Config.Interval - snmpsd.Config.RescanInterval - snmpsd.Config.DeviceCacheTTL Keep Duration (seconds) for timeout fields where numeric values are more appropriate. Update JSON schemas to use string type with pattern for LongDuration fields. * fix(go.d): address review comments for dyncfg - Validate that only one discoverer type is configured, reject configs with multiple discoverers instead of silently ignoring some - Use fn.JobName() in vnodeUserconfigFromPayload for consistent name sanitization (replaces spaces/colons with underscores) - Use json.Unmarshal instead of yaml.Unmarshal in Config.UnmarshalJSON to properly implement json.Unmarshaler contract * remove legacy classificator and composer * refactor(go.d/sd): remove unnecessary UnmarshalJSON wrapper * feat(go.d/sd): temporarily disable dyncfg integration for release Add disableDyncfg flag (default true) to bypass SD dyncfg integration: - Skip template registration when flag is true - Start pipelines directly without dyncfg job creation - Tests set flag to false to preserve existing behavior This allows the release to proceed while SD dyncfg feature is further validated. To re-enable: set disableDyncfg = false.