| 1 | # Dynamic Configuration (DynCfg) |
| 2 | |
| 3 | Dynamic Configuration (DynCfg) is a system in Netdata that enables both internal and external plugins/modules to expose their configurations dynamically to users through a unified interface. This document explains how DynCfg works and how to integrate your module with it. |
| 4 | |
| 5 | ## Overview |
| 6 | |
| 7 | DynCfg provides a centralized mechanism for: |
| 8 | |
| 9 | 1. Registering configuration objects from any plugin or module |
| 10 | 2. Providing a unified interface for users to view and modify these configurations |
| 11 | 3. Persisting configurations between Netdata agent restarts |
| 12 | 4. Validating configuration changes through the originating plugin/module |
| 13 | 5. Standardizing configuration UI using JSON Schema |
| 14 | |
| 15 | Key features: |
| 16 | |
| 17 | - Plugins can expose multiple configuration objects |
| 18 | - Each configuration object has a unique ID |
| 19 | - The owning plugin validates configuration changes before being committed |
| 20 | - The DynCfg manager maintains the state of all dynamic configurations |
| 21 | - JSON Schema is used to define the structure of configuration objects |
| 22 | - The UI is based on adaptations of the react-jsonschema-form project |
| 23 | |
| 24 | ## Architecture |
| 25 | |
| 26 | DynCfg consists of several API layers: |
| 27 | |
| 28 | 1. **Low-level API**: Core functionality used by both internal and external plugins |
| 29 | 2. **High-level API**: Simplified API for internal plugins and modules |
| 30 | 3. **External Plugin API**: Used by external plugins like go.d.plugin |
| 31 | |
| 32 | ## Integration Approaches |
| 33 | |
| 34 | There are two main ways to integrate with DynCfg: |
| 35 | |
| 36 | 1. **High-level API for Internal Plugins**: Used by the health alerts system (documented here) |
| 37 | 2. **External Plugin API**: Used by go.d.plugin (see [src/plugins.d/DYNCFG.md](/src/plugins.d/DYNCFG.md) for detailed documentation) |
| 38 | |
| 39 | ### Core Concepts |
| 40 | |
| 41 | ### Configuration ID Structure |
| 42 | |
| 43 | The configuration ID is a crucial part of the DynCfg system. It serves as a unique identifier and determines how the configuration appears in the UI. |
| 44 | |
| 45 | #### ID Format and Hierarchy |
| 46 | |
| 47 | Configuration IDs typically follow a colon-separated hierarchical structure: |
| 48 | |
| 49 | ``` |
| 50 | component:category:name |
| 51 | ``` |
| 52 | |
| 53 | Where: |
| 54 | |
| 55 | - **component**: The main module or plugin (e.g., "health", "go.d", "systemd-journal") |
| 56 | - **category**: Optional subcategory (e.g., "alert", "job") |
| 57 | - **name**: Specific configuration name |
| 58 | |
| 59 | For example: |
| 60 | |
| 61 | - `health:alert:prototype`: Health alert prototype template |
| 62 | - `health:alert:prototype:ram_usage`: Specific health alert prototype |
| 63 | - `go.d:nginx`: Nginx collector template |
| 64 | - `go.d:nginx:local_server`: Specific Nginx collector job |
| 65 | - `systemd-journal:monitored-directories`: Systemd journal configuration |
| 66 | |
| 67 | #### Templates and Jobs |
| 68 | |
| 69 | For templates and jobs, the ID follows a specific pattern: |
| 70 | |
| 71 | 1. **Template ID**: `component:template_name` |
| 72 | 2. **Job ID**: `component:template_name:job_name` |
| 73 | |
| 74 | The part before the last colon in a job ID must match an existing template ID. |
| 75 | |
| 76 | #### UI Organization |
| 77 | |
| 78 | The first component of the ID is used to organize configurations in the UI, creating separate tabs or sections. This allows for logical grouping of related configurations. |
| 79 | |
| 80 | When choosing an ID, consider: |
| 81 | |
| 82 | - UI organization (first component) |
| 83 | - Logical grouping (middle components) |
| 84 | - Uniqueness (complete ID) |
| 85 | - Readability for users |
| 86 | |
| 87 | ### Configuration Types |
| 88 | |
| 89 | - **DYNCFG_TYPE_SINGLE**: A single configuration object |
| 90 | - **DYNCFG_TYPE_TEMPLATE**: A template for creating multiple job configurations |
| 91 | - **DYNCFG_TYPE_JOB**: A specific job configuration (derived from a template) |
| 92 | |
| 93 | ### Response Codes |
| 94 | |
| 95 | DynCfg uses HTTP-like response codes to indicate the status of operations. These are crucial for both internal and external plugins to properly communicate with the DynCfg system: |
| 96 | |
| 97 | #### Success Codes (2xx) |
| 98 | |
| 99 | - **DYNCFG_RESP_RUNNING (200)**: Configuration was accepted and is currently running |
| 100 | - **DYNCFG_RESP_ACCEPTED (202)**: Configuration was accepted but not yet running |
| 101 | - **DYNCFG_RESP_ACCEPTED_DISABLED (298)**: Configuration was accepted but is currently disabled |
| 102 | - **DYNCFG_RESP_ACCEPTED_RESTART_REQUIRED (299)**: Configuration was accepted but requires a restart to apply |
| 103 | |
| 104 | #### Error Codes (4xx, 5xx) |
| 105 | |
| 106 | Standard HTTP error codes are used, including: |
| 107 | |
| 108 | - **HTTP_RESP_BAD_REQUEST (400)**: Invalid request or configuration |
| 109 | - **HTTP_RESP_NOT_FOUND (404)**: Requested configuration was not found |
| 110 | - **HTTP_RESP_INTERNAL_SERVER_ERROR (500)**: An internal error occurred |
| 111 | - **HTTP_RESP_NOT_IMPLEMENTED (501)**: The requested operation is not implemented |
| 112 | |
| 113 | When implementing a callback function, always return the appropriate response code to indicate the status of the operation. The DynCfg system uses these codes to determine how to handle the configuration and what to display to the user. |
| 114 | |
| 115 | ### Source Types |
| 116 | |
| 117 | - **DYNCFG_SOURCE_TYPE_INTERNAL**: Configuration defined internally within Netdata |
| 118 | - **DYNCFG_SOURCE_TYPE_DYNCFG**: Configuration created/modified through DynCfg |
| 119 | - **DYNCFG_SOURCE_TYPE_USER**: Configuration from user-provided files |
| 120 | |
| 121 | #### Supported Commands |
| 122 | |
| 123 | - **DYNCFG_CMD_SCHEMA**: Get JSON schema for the configuration |
| 124 | - **DYNCFG_CMD_GET**: Get the current configuration |
| 125 | - **DYNCFG_CMD_UPDATE**: Update the configuration |
| 126 | - **DYNCFG_CMD_DISABLE**: Disable the configuration |
| 127 | - **DYNCFG_CMD_ENABLE**: Enable the configuration |
| 128 | - **DYNCFG_CMD_ADD**: Add a new job (for templates only) |
| 129 | - **DYNCFG_CMD_REMOVE**: Remove a job (for DYNCFG_SOURCE_TYPE_DYNCFG jobs only) |
| 130 | - **DYNCFG_CMD_TEST**: Test a configuration without applying it |
| 131 | - **DYNCFG_CMD_RESTART**: Restart the configuration |
| 132 | - **DYNCFG_CMD_USERCONFIG**: Get the configuration in a user-friendly format (used for conf files) |
| 133 | |
| 134 | ## Implementing DynCfg for Internal Plugins |
| 135 | |
| 136 | Here's how to implement DynCfg for an internal plugin/module: |
| 137 | |
| 138 | ### 1. Register Your Configuration |
| 139 | |
| 140 | For internal plugins, the Netdata daemon already handles initialization and shutdown of the DynCfg system. You don't need to call `dyncfg_init()` or `dyncfg_shutdown()` in your plugin code. |
| 141 | |
| 142 | Register your configurations when your plugin initializes: |
| 143 | |
| 144 | ```c |
| 145 | bool dyncfg_add( |
| 146 | RRDHOST *host, // The host this configuration belongs to (localhost for global configs) |
| 147 | const char *id, // Unique ID for this configuration |
| 148 | const char *path, // Path for UI organization |
| 149 | DYNCFG_STATUS status, // Initial status (ACCEPTED, DISABLED, etc.) |
| 150 | DYNCFG_TYPE type, // SINGLE, TEMPLATE, or JOB |
| 151 | DYNCFG_SOURCE_TYPE source_type, // INTERNAL, DYNCFG, USER |
| 152 | const char *source, // Source identifier (e.g., "internal") |
| 153 | DYNCFG_CMDS cmds, // Supported commands (bitwise OR of DYNCFG_CMD_* values) |
| 154 | HTTP_ACCESS view_access, // Access permissions for viewing |
| 155 | HTTP_ACCESS edit_access, // Access permissions for editing |
| 156 | dyncfg_cb_t cb, // Callback function for handling commands |
| 157 | void *data // User data passed to the callback |
| 158 | ); |
| 159 | ``` |
| 160 | |
| 161 | Example (from health_dyncfg.c): |
| 162 | |
| 163 | ```c |
| 164 | dyncfg_add( |
| 165 | localhost, |
| 166 | DYNCFG_HEALTH_ALERT_PROTOTYPE_PREFIX, |
| 167 | "/health/alerts/prototypes", |
| 168 | DYNCFG_STATUS_ACCEPTED, |
| 169 | DYNCFG_TYPE_TEMPLATE, |
| 170 | DYNCFG_SOURCE_TYPE_INTERNAL, |
| 171 | "internal", |
| 172 | DYNCFG_CMD_SCHEMA | DYNCFG_CMD_ADD | DYNCFG_CMD_ENABLE | DYNCFG_CMD_DISABLE | DYNCFG_CMD_USERCONFIG, |
| 173 | HTTP_ACCESS_NONE, |
| 174 | HTTP_ACCESS_NONE, |
| 175 | dyncfg_health_cb, |
| 176 | NULL |
| 177 | ); |
| 178 | ``` |
| 179 | |
| 180 | ### 3. Implement a Callback Function |
| 181 | |
| 182 | ```c |
| 183 | int your_dyncfg_callback( |
| 184 | const char *transaction, // Transaction ID |
| 185 | const char *id, // Configuration ID |
| 186 | DYNCFG_CMDS cmd, // Command being executed |
| 187 | const char *add_name, // For ADD: the name to add |
| 188 | BUFFER *payload, // Command payload |
| 189 | usec_t *stop_monotonic_ut, // Timeout info |
| 190 | bool *cancelled, // Whether the operation was cancelled |
| 191 | BUFFER *result, // Buffer to write results to |
| 192 | HTTP_ACCESS access, // Access level of the user |
| 193 | const char *source, // Configuration source |
| 194 | void *data // User data passed during registration |
| 195 | ) { |
| 196 | // Handle the command |
| 197 | switch(cmd) { |
| 198 | case DYNCFG_CMD_SCHEMA: |
| 199 | // Return JSON schema |
| 200 | buffer_json_initialize(result, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY); |
| 201 | // Add your schema here |
| 202 | buffer_json_finalize(result); |
| 203 | return HTTP_RESP_OK; |
| 204 | |
| 205 | case DYNCFG_CMD_GET: |
| 206 | // Return current configuration |
| 207 | buffer_json_initialize(result, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY); |
| 208 | // Add your configuration here |
| 209 | buffer_json_finalize(result); |
| 210 | return HTTP_RESP_OK; |
| 211 | |
| 212 | case DYNCFG_CMD_UPDATE: |
| 213 | // Process configuration update |
| 214 | // If update successful |
| 215 | return DYNCFG_RESP_RUNNING; // or DYNCFG_RESP_ACCEPTED if not running yet |
| 216 | |
| 217 | // If restart is required |
| 218 | // return DYNCFG_RESP_ACCEPTED_RESTART_REQUIRED; |
| 219 | |
| 220 | // If validation fails |
| 221 | // return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, "your error message"); |
| 222 | |
| 223 | case DYNCFG_CMD_DISABLE: |
| 224 | // Handle disabling the configuration |
| 225 | return DYNCFG_RESP_ACCEPTED_DISABLED; |
| 226 | |
| 227 | case DYNCFG_CMD_ENABLE: |
| 228 | // Handle enabling the configuration |
| 229 | return DYNCFG_RESP_RUNNING; // or DYNCFG_RESP_ACCEPTED if not running yet |
| 230 | |
| 231 | // Handle other commands |
| 232 | |
| 233 | default: |
| 234 | // Unsupported command |
| 235 | return dyncfg_default_response(result, HTTP_RESP_BAD_REQUEST, "unsupported command"); |
| 236 | } |
| 237 | } |
| 238 | ``` |
| 239 | |
| 240 | ### 4. Configuration Lifecycle Management |
| 241 | |
| 242 | Unregister configurations only when they’re no longer conceptually relevant (such as when a feature becomes unavailable), not during plugin or module shutdown: |
| 243 | |
| 244 | ```c |
| 245 | dyncfg_del(host, id); // Remove a specific configuration that no longer applies |
| 246 | ``` |
| 247 | |
| 248 | When a plugin or module exits, the Netdata Functions manager automatically stops accepting requests for its functions. The configurations will remain in the DynCfg system and will become active again when the plugin restarts. |
| 249 | |
| 250 | The Netdata daemon also handles shutting down the entire DynCfg system, so you don't need to call `dyncfg_shutdown()` in your plugin code. |
| 251 | |
| 252 | ## Implementing DynCfg for External Plugins |
| 253 | |
| 254 | External plugins like go.d.plugin use a different approach based on the plugins.d protocol. For detailed information on implementing DynCfg for external plugins, please refer to the [External Plugins DynCfg documentation](/src/plugins.d/DYNCFG.md). |
| 255 | |
| 256 | This documentation covers: |
| 257 | |
| 258 | - How to register configurations using the CONFIG command |
| 259 | - How to respond to configuration commands |
| 260 | - How to update configuration status |
| 261 | - How to delete configurations |
| 262 | - Detailed examples and best practices |
| 263 | |
| 264 | ## JSON Schema for Configuration UI |
| 265 | |
| 266 | The UI for modifying configurations is generated from JSON Schema. Netdata supports two ways to provide schema definitions: |
| 267 | |
| 268 | ### 1. Static Schema Files |
| 269 | |
| 270 | Netdata first attempts to load schema files from the disk before calling the plugin or module. Schema files should be placed in: |
| 271 | |
| 272 | - `CONFIG_DIR/schema.d/` (user-provided schemas, typically `/etc/netdata/schema.d/`) |
| 273 | - `LIBCONFIG_DIR/schema.d/` (stock schemas, typically `/usr/lib/netdata/conf.d/schema.d/`) |
| 274 | |
| 275 | Schema files should be named after the configuration ID with `.json` extension, for example: |
| 276 | |
| 277 | - `health:alert:prototype.json` |
| 278 | - `go.d:nginx.json` |
| 279 | |
| 280 | ### 2. Dynamic Schema Generation |
| 281 | |
| 282 | If no static schema file is found, Netdata will call the plugin or module with the `DYNCFG_CMD_SCHEMA` command: |
| 283 | |
| 284 | 1. For internal plugins, the callback function should return a JSON Schema document |
| 285 | 2. For external plugins, the plugin should respond to the schema command with a JSON Schema document |
| 286 | |
| 287 | This gives you flexibility to either: |
| 288 | |
| 289 | - Use static schema files for simple, fixed configurations |
| 290 | - Generate schemas dynamically for more complex configurations that may change based on runtime conditions |
| 291 | |
| 292 | Example JSON Schema: |
| 293 | |
| 294 | ```json |
| 295 | { |
| 296 | "type": "object", |
| 297 | "properties": { |
| 298 | "url": { |
| 299 | "type": "string", |
| 300 | "format": "uri", |
| 301 | "title": "Server URL", |
| 302 | "description": "The URL of the server to connect to" |
| 303 | }, |
| 304 | "timeout": { |
| 305 | "type": "integer", |
| 306 | "minimum": 1, |
| 307 | "maximum": 60, |
| 308 | "title": "Timeout", |
| 309 | "description": "Connection timeout in seconds" |
| 310 | }, |
| 311 | "auth": { |
| 312 | "type": "object", |
| 313 | "title": "Authentication", |
| 314 | "properties": { |
| 315 | "username": { |
| 316 | "type": "string", |
| 317 | "title": "Username" |
| 318 | }, |
| 319 | "password": { |
| 320 | "type": "string", |
| 321 | "format": "password", |
| 322 | "title": "Password" |
| 323 | } |
| 324 | } |
| 325 | } |
| 326 | }, |
| 327 | "required": [ |
| 328 | "url" |
| 329 | ] |
| 330 | } |
| 331 | ``` |
| 332 | |
| 333 | ## Action Behavior by Configuration Type |
| 334 | |
| 335 | Different actions behave differently depending on the configuration type. The following table explains what happens when each action is applied to different configuration types: |
| 336 | |
| 337 | | Action | SINGLE | TEMPLATE | JOB | |
| 338 | |----------------|--------------------------------------|--------------------------------------------------|-------------------------------------------------------| |
| 339 | | **SCHEMA** | Returns schema from file or plugin | Returns schema from file or plugin | Uses template's schema | |
| 340 | | **GET** | Returns current configuration | Not supported (templates don't maintain data) | Returns current configuration | |
| 341 | | **UPDATE** | Updates single configuration | Not supported (templates don't maintain data) | Updates job configuration | |
| 342 | | **ADD** | Not supported | Creates a new job based on template | Not supported | |
| 343 | | **REMOVE** | Not supported | Not supported | Removes job (only for DYNCFG_SOURCE_TYPE_DYNCFG jobs) | |
| 344 | | **ENABLE** | Enables single configuration | Enables template AND all its jobs | Enables job (fails if its template is disabled) | |
| 345 | | **DISABLE** | Disables single configuration | Disables template AND all its jobs | Disables job | |
| 346 | | **RESTART** | Sends restart command to plugin | Restarts all template's jobs | Restarts specific job | |
| 347 | | **TEST** | Tests configuration without applying | Tests a new job configuration | Tests updated job configuration | |
| 348 | | **USERCONFIG** | Returns user-friendly configuration | Returns template for user-friendly configuration | Returns user-friendly configuration | |
| 349 | |
| 350 | **Important Notes:** |
| 351 | |
| 352 | - When a template is disabled, all its jobs are automatically disabled regardless of their individual settings |
| 353 | - Jobs can’t be enabled if their parent template is disabled |
| 354 | - Template schemas are used for all jobs created from that template |
| 355 | - The REMOVE action is only available for jobs created via DynCfg (with source type DYNCFG_SOURCE_TYPE_DYNCFG) |
| 356 | - RESTART on a template recursively restarts all jobs associated with that template |
| 357 | |
| 358 | ## API Access |
| 359 | |
| 360 | The DynCfg system exposes configurations via the Netdata API for management: |
| 361 | |
| 362 | ### Tree API |
| 363 | |
| 364 | Netdata provides a tree API that returns the entire DynCfg tree or a specific subpath: |
| 365 | |
| 366 | ``` |
| 367 | /api/v3/config?action=tree&path=/collectors |
| 368 | ``` |
| 369 | |
| 370 | Parameters: |
| 371 | |
| 372 | - `action`: Set to "tree" to get the configuration tree |
| 373 | - `path`: The configuration path to start from (e.g., "/collectors", "/health/alerts") |
| 374 | - `id` (optional): Return only a specific configuration ID |
| 375 | |
| 376 | This returns a JSON structure of all configurations, organized by path, with their status, commands, access controls, and other metadata. |
| 377 | |
| 378 | ### Configuration Actions API |
| 379 | |
| 380 | Individual configurations can be managed via: |
| 381 | |
| 382 | ``` |
| 383 | /api/v3/config?action=<command>&id=<configuration_id>&name=<name> |
| 384 | ``` |
| 385 | |
| 386 | Where: |
| 387 | |
| 388 | - `action`: One of the supported commands (get, update, add, remove, etc.) |
| 389 | - `id`: The configuration ID to act upon |
| 390 | - `name`: Required for add/test actions (new job name) |
| 391 | |
| 392 | The API automatically routes commands to the appropriate plugin or module that registered the configuration. |
| 393 | |
| 394 | ## Best Practices |
| 395 | |
| 396 | 1. **Use Clear IDs**: Configuration IDs should be clear and hierarchical (e.g., "mymodule:config1") |
| 397 | 2. **Choose Appropriate Paths**: The path parameter controls UI organization - select logical paths for easy navigation |
| 398 | 3. **Validate Thoroughly**: Always validate configuration changes before accepting them |
| 399 | 4. **Provide Helpful Errors**: Return detailed error messages when rejections occur |
| 400 | 5. **Document Your Schema**: Include descriptions for all properties in your JSON Schema |
| 401 | 6. **Clean Up**: Remove configurations when modules are unloaded or no longer needed |
| 402 | 7. **Security**: Set appropriate view/edit access controls for sensitive configurations |
| 403 | |
| 404 | ## Example Implementations |
| 405 | |
| 406 | ### Health Alerts System (Internal Plugin) |
| 407 | |
| 408 | Health alerts use DynCfg to manage alert definitions. Key files: |
| 409 | |
| 410 | - `src/health/health_dyncfg.c`: Implements DynCfg integration for health alerts |
| 411 | - Handles alert prototype templates and individual alert configurations |
| 412 | |
| 413 | The health module uses the high-level API with IDs like: |
| 414 | |
| 415 | - `health:alert:prototype` for the alert template |
| 416 | - `health:alert:prototype:ram_usage` for specific alert prototypes |
| 417 | |
| 418 | It supports multiple configuration objects, validation of alert definitions, and conversion between different configuration formats (JSON and traditional Netdata health configuration syntax). |
| 419 | |
| 420 | ### systemd-journal.plugin (External Plugin) |
| 421 | |
| 422 | The systemd-journal.plugin is an external plugin written in C that uses DynCfg to manage journal directory configurations: |
| 423 | |
| 424 | - `src/collectors/systemd-journal.plugin/systemd-journal-dyncfg.c`: Implements DynCfg integration |
| 425 | - Registers as `systemd-journal:monitored-directories` ID |
| 426 | - Uses a SINGLE configuration type for its directory list |
| 427 | - Provides validation of directory paths for security |
| 428 | - Implements GET and UPDATE commands |
| 429 | |
| 430 | This is a good example of an external plugin using the DynCfg system for a single configuration object. |
| 431 | |
| 432 | ### go.d.plugin (External Plugin) |
| 433 | |
| 434 | go.d.plugin uses DynCfg to manage job configurations. It: |
| 435 | |
| 436 | 1. Registers configurations through the plugins.d protocol |
| 437 | 2. Generates dynamic JSON Schema based on Go struct tags |
| 438 | 3. Handles configuration updates for collecting jobs |
| 439 | |
| 440 | It uses IDs like: |
| 441 | |
| 442 | - `go.d:nginx` for the Nginx collector template |
| 443 | - `go.d:nginx:local_server` for a specific Nginx collector job |
| 444 | |
| 445 | go.d.plugin demonstrates how external plugins can leverage DynCfg to provide dynamic configuration capabilities through the plugins.d protocol. |
| 446 | |
| 447 | ## Debugging Tips |
| 448 | |
| 449 | 1. Set the environment variable `NETDATA_DEBUG_DYNCFG=1` to enable debug logging for DynCfg |
| 450 | 2. Check `/var/lib/netdata/config/` for persisted configuration files |
| 451 | 3. Inspect configurations via the API: `/api/v3/config?id=<your-config-id>` |