| 1 | # Dynamic Configuration for External Plugins |
| 2 | |
| 3 | External plugins in Netdata can expose dynamic configuration capabilities through the DynCfg system. This document explains how to implement DynCfg in external plugins using the plugins.d protocol. |
| 4 | |
| 5 | ## Overview |
| 6 | |
| 7 | The DynCfg system allows external plugins to: |
| 8 | |
| 9 | 1. Register configurable entities (both single configurations and templates for creating jobs) |
| 10 | 2. Receive configuration commands from users |
| 11 | 3. Validate and apply configurations |
| 12 | 4. Persist configurations between Netdata agent restarts |
| 13 | |
| 14 | ## Protocol Commands |
| 15 | |
| 16 | DynCfg for external plugins uses the following plugins.d protocol commands: |
| 17 | |
| 18 | 1. `CONFIG`: Sent from the plugin to Netdata to register, update status, or delete configurations |
| 19 | 2. `FUNCTION`/`FUNCTION_PAYLOAD_BEGIN`: Received by the plugin to handle configuration commands |
| 20 | 3. `FUNCTION_RESULT_BEGIN`: Sent from the plugin to respond to commands |
| 21 | |
| 22 | ## Implementing DynCfg in External Plugins |
| 23 | |
| 24 | ### 1. Register a Configuration |
| 25 | |
| 26 | To register a configuration, the plugin sends the CONFIG command: |
| 27 | |
| 28 | ``` |
| 29 | CONFIG <id> CREATE <status> <type> <path> <source_type> <source> <cmds> <view_access> <edit_access> |
| 30 | ``` |
| 31 | |
| 32 | Where: |
| 33 | |
| 34 | - `id` is a unique identifier for the configurable entity (e.g., "go.d:nginx") |
| 35 | - `status` can be: |
| 36 | - `accepted`: Configuration is accepted but not running |
| 37 | - `running`: Configuration is accepted and running |
| 38 | - `failed`: Plugin fails to run the configuration |
| 39 | - `incomplete`: Plugin needs additional settings |
| 40 | - `disabled`: Configuration is disabled by a user |
| 41 | - `type` can be: |
| 42 | - `single`: A single configuration object (not addable or removable by users) |
| 43 | - `template`: A template for creating multiple job configurations |
| 44 | - `job`: A specific job configuration (derived from a template) |
| 45 | - `path` is the UI organization path (usually "/collectors") that determines where in the configuration tree the item will appear in the UI. This is separate from the ID and controls the hierarchical navigation structure. |
| 46 | - `source_type` can be: |
| 47 | - `internal`: Based on internal code settings |
| 48 | - `stock`: Default configurations |
| 49 | - `user`: User configurations via a file |
| 50 | - `dyncfg`: Configuration received via this mechanism |
| 51 | - `discovered`: Dynamically discovered by the plugin |
| 52 | - `source` provides more details about the exact source |
| 53 | - `cmds` is a space or pipe (|) separated list of supported commands: |
| 54 | - `schema`: Get JSON schema for the configuration |
| 55 | - `get`: Get current configuration values |
| 56 | - `update`: Receive configuration updates |
| 57 | - `add`: Receive job creation commands (templates only) |
| 58 | - `remove`: Remove a configuration (jobs only) |
| 59 | - `enable`/`disable`: Enable or disable the configuration |
| 60 | - `test`: Test a configuration without applying it |
| 61 | - `restart`: Restart the configuration |
| 62 | - `userconfig`: Get user-friendly configuration format |
| 63 | - `view_access` and `edit_access` are permission bitmaps (use 0 for default permissions) |
| 64 | |
| 65 | Example: |
| 66 | |
| 67 | ``` |
| 68 | CONFIG go.d:nginx CREATE accepted template /collectors internal internal schema|add|enable|disable 0 0 |
| 69 | CONFIG go.d:nginx:local_server CREATE running job /collectors dyncfg user schema|get|update|remove|enable|disable|restart 0 0 |
| 70 | ``` |
| 71 | |
| 72 | ### 2. Respond to Configuration Commands |
| 73 | |
| 74 | The plugin receives configuration commands from Netdata as plugin functions. These come in two forms: |
| 75 | |
| 76 | #### Without Payload: |
| 77 | |
| 78 | ``` |
| 79 | FUNCTION <transaction_id> <timeout_ms> "config <id> <command>" "<http_access>" "<source>" |
| 80 | ``` |
| 81 | |
| 82 | Used for commands like: `schema`, `get`, `remove`, `enable`, `disable`, `restart` |
| 83 | |
| 84 | Example: |
| 85 | |
| 86 | ``` |
| 87 | FUNCTION abcd1234 60 "config go.d:nginx:local_server get" "member" "netdata-cli" |
| 88 | ``` |
| 89 | |
| 90 | #### With Payload: |
| 91 | |
| 92 | ``` |
| 93 | FUNCTION_PAYLOAD_BEGIN <transaction_id> <timeout_ms> "config <id> <command>" "<http_access>" "<source>" "<content_type>" |
| 94 | <payload_data> |
| 95 | FUNCTION_PAYLOAD_END |
| 96 | ``` |
| 97 | |
| 98 | Used for commands like: `update`, `add`, `test` that require additional data. |
| 99 | |
| 100 | Example: |
| 101 | |
| 102 | ``` |
| 103 | FUNCTION_PAYLOAD_BEGIN abcd1234 60 "config go.d:nginx:local_server update" "member" "netdata-cli" "application/json" |
| 104 | { |
| 105 | "url": "http://localhost:80/stub_status", |
| 106 | "timeout": 5, |
| 107 | "update_every": 10 |
| 108 | } |
| 109 | FUNCTION_PAYLOAD_END |
| 110 | ``` |
| 111 | |
| 112 | ### 3. Process Commands and Respond |
| 113 | |
| 114 | After receiving a command, the plugin should process it and respond with a function result: |
| 115 | |
| 116 | ``` |
| 117 | FUNCTION_RESULT_BEGIN <transaction_id> <http_status_code> <content_type> <expiration> |
| 118 | <result_data> |
| 119 | FUNCTION_RESULT_END |
| 120 | ``` |
| 121 | |
| 122 | Where: |
| 123 | |
| 124 | - `transaction_id` is the same ID received in the original command |
| 125 | - `http_status_code` is the standard HTTP response code: |
| 126 | - `200`: Success (DYNCFG_RESP_RUNNING) - Configuration accepted and running |
| 127 | - `202`: Accepted (DYNCFG_RESP_ACCEPTED) - Configuration accepted but not running yet |
| 128 | - `298`: Accepted but disabled (DYNCFG_RESP_ACCEPTED_DISABLED) |
| 129 | - `299`: Accepted but restart required (DYNCFG_RESP_ACCEPTED_RESTART_REQUIRED) |
| 130 | - `400`: Bad request - Invalid configuration |
| 131 | - `404`: Not found - Configuration not found |
| 132 | - `500`: Internal server error |
| 133 | - `content_type` is typically "application/json" |
| 134 | - `expiration` is the absolute timestamp (unix epoch) for result expiration |
| 135 | |
| 136 | The result data depends on the command: |
| 137 | |
| 138 | - `schema`: Return JSON Schema document |
| 139 | - `get`: Return current configuration values |
| 140 | - Other commands: Return a success or error message |
| 141 | |
| 142 | Success response example: |
| 143 | |
| 144 | ``` |
| 145 | FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0 |
| 146 | { |
| 147 | "status": 200, |
| 148 | "message": "Configuration updated successfully" |
| 149 | } |
| 150 | FUNCTION_RESULT_END |
| 151 | ``` |
| 152 | |
| 153 | Error response example: |
| 154 | |
| 155 | ``` |
| 156 | FUNCTION_RESULT_BEGIN abcd1234 400 application/json 0 |
| 157 | { |
| 158 | "status": 400, |
| 159 | "error_message": "Invalid URL format" |
| 160 | } |
| 161 | FUNCTION_RESULT_END |
| 162 | ``` |
| 163 | |
| 164 | ### 4. Update Configuration Status |
| 165 | |
| 166 | To update the status of a configuration after it's been created: |
| 167 | |
| 168 | ``` |
| 169 | CONFIG <id> STATUS <new_status> |
| 170 | ``` |
| 171 | |
| 172 | Example: |
| 173 | |
| 174 | ``` |
| 175 | CONFIG go.d:nginx:local_server STATUS running |
| 176 | ``` |
| 177 | |
| 178 | This is useful when a configuration transitions from "accepted" to "running" or "failed" after being tested. |
| 179 | |
| 180 | ### 5. Delete a Configuration |
| 181 | |
| 182 | When a configuration is no longer available (e.g., the monitored service is removed): |
| 183 | |
| 184 | ``` |
| 185 | CONFIG <id> DELETE |
| 186 | ``` |
| 187 | |
| 188 | Example: |
| 189 | |
| 190 | ``` |
| 191 | CONFIG go.d:nginx:local_server DELETE |
| 192 | ``` |
| 193 | |
| 194 | ## JSON Schema for Configuration UI |
| 195 | |
| 196 | DynCfg uses JSON Schema to define the structure of configuration objects, which is used to generate the UI. |
| 197 | |
| 198 | ### Static Schema Files (Optional) |
| 199 | |
| 200 | Before calling the plugin, Netdata will first attempt to find a static schema file. You can provide static schema files in: |
| 201 | |
| 202 | - `CONFIG_DIR/schema.d/` (user-provided schemas, typically `/etc/netdata/schema.d/`) |
| 203 | - `LIBCONFIG_DIR/schema.d/` (stock schemas, typically `/usr/lib/netdata/conf.d/schema.d/`) |
| 204 | |
| 205 | Schema files should be named after the configuration ID with `.json` extension: |
| 206 | |
| 207 | ``` |
| 208 | /etc/netdata/schema.d/go.d:nginx.json |
| 209 | ``` |
| 210 | |
| 211 | This approach is useful for stable schemas that don't change frequently. |
| 212 | |
| 213 | ### Dynamic Schema Generation |
| 214 | |
| 215 | If no static schema file is found, Netdata will send a `schema` command to the plugin. When handling a `schema` request, the plugin should return a JSON Schema document: |
| 216 | |
| 217 | ```json |
| 218 | { |
| 219 | "type": "object", |
| 220 | "properties": { |
| 221 | "url": { |
| 222 | "type": "string", |
| 223 | "format": "uri", |
| 224 | "title": "Server URL", |
| 225 | "description": "The URL of the Nginx stub_status endpoint" |
| 226 | }, |
| 227 | "timeout": { |
| 228 | "type": "integer", |
| 229 | "minimum": 1, |
| 230 | "maximum": 60, |
| 231 | "title": "Timeout", |
| 232 | "description": "Connection timeout in seconds" |
| 233 | }, |
| 234 | "update_every": { |
| 235 | "type": "integer", |
| 236 | "minimum": 1, |
| 237 | "title": "Update Every", |
| 238 | "description": "Data collection frequency in seconds" |
| 239 | } |
| 240 | }, |
| 241 | "required": [ |
| 242 | "url" |
| 243 | ] |
| 244 | } |
| 245 | ``` |
| 246 | |
| 247 | For templates, the schema will be used when users add new jobs based on the template. |
| 248 | |
| 249 | ## Action Behavior Reference |
| 250 | |
| 251 | When implementing DynCfg in your external plugin, be aware of how actions should behave based on the configuration type: |
| 252 | |
| 253 | | Action | TEMPLATE | JOB | |
| 254 | |----------------|-----------------------------------------|-----------------------------------------| |
| 255 | | **SCHEMA** | Return schema for creating new jobs | Use template's schema | |
| 256 | | **GET** | Not applicable | Return current configuration | |
| 257 | | **UPDATE** | Not applicable | Update configuration and apply if valid | |
| 258 | | **ADD** | Create new job from template | Not applicable | |
| 259 | | **REMOVE** | Not supported | Remove job (only for user-created jobs) | |
| 260 | | **ENABLE** | Enable template and all its jobs | Enable specific job | |
| 261 | | **DISABLE** | Disable template and all its jobs | Disable specific job | |
| 262 | | **RESTART** | Restart all jobs based on template | Restart specific job | |
| 263 | | **TEST** | Test a potential job configuration | Test configuration changes | |
| 264 | | **USERCONFIG** | Return template in user-friendly format | Return job in user-friendly format | |
| 265 | |
| 266 | **Important Implementation Notes:** |
| 267 | |
| 268 | - When a template is disabled, send DISABLE commands to all jobs of that template |
| 269 | - Reject ENABLE commands for jobs if their template is disabled |
| 270 | - For job SCHEMA requests, return the same schema as the template |
| 271 | - REMOVE should only work on dynamically added jobs, not ones from static configurations |
| 272 | - Return appropriate response codes to indicate the status (running, accepted, disabled) |
| 273 | |
| 274 | ## External Plugin Examples |
| 275 | |
| 276 | ### C-based External Plugin (systemd-journal.plugin) |
| 277 | |
| 278 | The systemd-journal.plugin is a C-based external plugin that uses DynCfg to manage journal directory configurations. It implements a SINGLE configuration type to manage the list of journald directories to monitor: |
| 279 | |
| 280 | ```c |
| 281 | // Register the configuration |
| 282 | functions_evloop_dyncfg_add( |
| 283 | wg, |
| 284 | "systemd-journal:monitored-directories", // ID |
| 285 | "/logs/systemd-journal", // UI Path |
| 286 | DYNCFG_STATUS_RUNNING, // Status |
| 287 | DYNCFG_TYPE_SINGLE, // Type - single configuration |
| 288 | DYNCFG_SOURCE_TYPE_INTERNAL, // Source type |
| 289 | "internal", // Source |
| 290 | DYNCFG_CMD_SCHEMA | DYNCFG_CMD_GET | DYNCFG_CMD_UPDATE, // Supported commands |
| 291 | HTTP_ACCESS_NONE, // View permissions |
| 292 | HTTP_ACCESS_NONE, // Edit permissions |
| 293 | systemd_journal_directories_dyncfg_cb, // Callback function |
| 294 | NULL // User data |
| 295 | ); |
| 296 | ``` |
| 297 | |
| 298 | Key points about its implementation: |
| 299 | |
| 300 | - Uses a single, non-removable configuration object |
| 301 | - Supports schema, get, and update commands |
| 302 | - Validates directory paths for security |
| 303 | - Updates the systemd-journal watcher when configuration changes |
| 304 | |
| 305 | ### Go-based External Plugin (go.d.plugin) |
| 306 | |
| 307 | Here's a complete example showing how a Go-based external plugin might implement DynCfg for an Nginx module: |
| 308 | |
| 309 | ### 1. Register the Template and Jobs on Startup |
| 310 | |
| 311 | ``` |
| 312 | # Register the template for Nginx configurations |
| 313 | CONFIG go.d:nginx CREATE accepted template /collectors internal internal schema|add|enable|disable 0 0 |
| 314 | |
| 315 | # Register existing jobs |
| 316 | CONFIG go.d:nginx:local_server CREATE running job /collectors user /etc/netdata/go.d/nginx.conf schema|get|update|remove|enable|disable|restart 0 0 |
| 317 | CONFIG go.d:nginx:production CREATE running job /collectors user /etc/netdata/go.d/nginx.conf schema|get|update|remove|enable|disable|restart 0 0 |
| 318 | ``` |
| 319 | |
| 320 | ### 2. Handle Schema Command |
| 321 | |
| 322 | When receiving: |
| 323 | |
| 324 | ``` |
| 325 | FUNCTION abcd1234 60 "config go.d:nginx schema" "member" "netdata-cli" |
| 326 | ``` |
| 327 | |
| 328 | Respond with: |
| 329 | |
| 330 | ``` |
| 331 | FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0 |
| 332 | { |
| 333 | "type": "object", |
| 334 | "properties": { |
| 335 | "url": { |
| 336 | "type": "string", |
| 337 | "format": "uri", |
| 338 | "title": "Server URL", |
| 339 | "description": "The URL of the Nginx stub_status endpoint" |
| 340 | }, |
| 341 | "timeout": { |
| 342 | "type": "integer", |
| 343 | "minimum": 1, |
| 344 | "maximum": 60, |
| 345 | "title": "Timeout", |
| 346 | "description": "Connection timeout in seconds" |
| 347 | }, |
| 348 | "update_every": { |
| 349 | "type": "integer", |
| 350 | "minimum": 1, |
| 351 | "title": "Update Every", |
| 352 | "description": "Data collection frequency in seconds" |
| 353 | } |
| 354 | }, |
| 355 | "required": ["url"] |
| 356 | } |
| 357 | FUNCTION_RESULT_END |
| 358 | ``` |
| 359 | |
| 360 | ### 3. Handle Get Command |
| 361 | |
| 362 | When receiving: |
| 363 | |
| 364 | ``` |
| 365 | FUNCTION abcd1234 60 "config go.d:nginx:local_server get" "member" "netdata-cli" |
| 366 | ``` |
| 367 | |
| 368 | Respond with: |
| 369 | |
| 370 | ``` |
| 371 | FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0 |
| 372 | { |
| 373 | "url": "http://localhost:80/stub_status", |
| 374 | "timeout": 5, |
| 375 | "update_every": 10 |
| 376 | } |
| 377 | FUNCTION_RESULT_END |
| 378 | ``` |
| 379 | |
| 380 | ### 4. Handle Update Command |
| 381 | |
| 382 | When receiving: |
| 383 | |
| 384 | ``` |
| 385 | FUNCTION_PAYLOAD_BEGIN abcd1234 60 "config go.d:nginx:local_server update" "member" "netdata-cli" "application/json" |
| 386 | { |
| 387 | "url": "http://localhost:8080/stub_status", |
| 388 | "timeout": 3, |
| 389 | "update_every": 5 |
| 390 | } |
| 391 | FUNCTION_PAYLOAD_END |
| 392 | ``` |
| 393 | |
| 394 | Process the update and respond: |
| 395 | |
| 396 | ``` |
| 397 | FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0 |
| 398 | { |
| 399 | "status": 200, |
| 400 | "message": "Configuration updated successfully" |
| 401 | } |
| 402 | FUNCTION_RESULT_END |
| 403 | ``` |
| 404 | |
| 405 | If a restart is required: |
| 406 | |
| 407 | ``` |
| 408 | FUNCTION_RESULT_BEGIN abcd1234 299 application/json 0 |
| 409 | { |
| 410 | "status": 299, |
| 411 | "message": "Configuration updated, restart required to apply changes" |
| 412 | } |
| 413 | FUNCTION_RESULT_END |
| 414 | ``` |
| 415 | |
| 416 | ### 5. Handle Add Command (for templates) |
| 417 | |
| 418 | When receiving: |
| 419 | |
| 420 | ``` |
| 421 | FUNCTION_PAYLOAD_BEGIN abcd1234 60 "config go.d:nginx add" "member" "netdata-cli" "application/json" |
| 422 | { |
| 423 | "name": "staging", |
| 424 | "url": "http://staging:80/stub_status", |
| 425 | "timeout": 5, |
| 426 | "update_every": 10 |
| 427 | } |
| 428 | FUNCTION_PAYLOAD_END |
| 429 | ``` |
| 430 | |
| 431 | Process the new job and respond: |
| 432 | |
| 433 | ``` |
| 434 | FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0 |
| 435 | { |
| 436 | "status": 200, |
| 437 | "message": "Job 'staging' created successfully" |
| 438 | } |
| 439 | FUNCTION_RESULT_END |
| 440 | ``` |
| 441 | |
| 442 | Then register the new job: |
| 443 | |
| 444 | ``` |
| 445 | CONFIG go.d:nginx:staging CREATE running job /collectors dyncfg netdata-cli schema|get|update|remove|enable|disable|restart 0 0 |
| 446 | ``` |
| 447 | |
| 448 | ## Best Practices |
| 449 | |
| 450 | 1. **Use Consistent IDs**: Follow the pattern `component:template_name` for templates and `component:template_name:job_name` for jobs |
| 451 | 2. **Validate Thoroughly**: Always validate configuration changes before accepting them |
| 452 | 3. **Include Descriptive Messages**: Provide helpful error messages when rejections occur |
| 453 | 4. **Document Your Schema**: Include clear titles and descriptions for all properties in your JSON Schema |
| 454 | 5. **Handle Errors Gracefully**: Return appropriate HTTP status codes and error messages |
| 455 | 6. **Update Status Promptly**: When a configuration changes state (e.g., from "accepted" to "running"), update its status |
| 456 | 7. **Clean Up Configurations**: When a monitored resource is gone, delete its configuration with `CONFIG id DELETE` |
| 457 | |
| 458 | ## Debugging Tips |
| 459 | |
| 460 | 1. Set `NETDATA_DEBUG_DYNCFG=1` environment variable when running Netdata to see detailed logs |
| 461 | 2. If configurations aren't being registered, check for errors in the plugin output |
| 462 | 3. Verify configuration files are saved in `/var/lib/netdata/config/` |
| 463 | 4. Test configurations via the API: `/api/v3/config?id=<your-config-id>` |
| 464 | |
| 465 | ## Related Documentation |
| 466 | |
| 467 | - [Main DynCfg Documentation](/src/daemon/dyncfg/README.md) - Core DynCfg system concepts and APIs |
| 468 | - [Plugins.d Protocol](/src/plugins.d/README.md) - Complete documentation of the plugins.d protocol |