| 1 | # Fleet Deployment and Configuration Management |
| 2 | |
| 3 | As infrastructures grow from a handful of servers to thousands of nodes across mixed environments (Linux, Kubernetes, Windows, macOS, FreeBSD), managing observability agents becomes one of the most painful operational tasks. |
| 4 | |
| 5 | **Without a coherent strategy, teams face:** |
| 6 | - Dozens of exporters or collectors, each with its own configs and update cycles |
| 7 | - Manual configuration files scattered across nodes |
| 8 | - Service discovery gaps that lead to blind spots |
| 9 | - Downtime during upgrades and redeployments |
| 10 | - Compliance challenges when configuration states drift |
| 11 | |
| 12 | ## How Netdata Solves It |
| 13 | |
| 14 | **Netdata's Fleet Management Philosophy:** |
| 15 | - **Zero-Configuration**: We eliminate 99% of manual configuration through comprehensive auto-discovery on supported platforms |
| 16 | - **Single Agent**: The Netdata Agent replaces dozens of exporters, dramatically reducing configuration complexity |
| 17 | - **Automatic Everything**: Installation and updates are fully automated when possible, with zero-downtime updates and automatic rollback |
| 18 | - **Flexible Management**: Choose Infrastructure as Code (IaC) for compliance, Dynamic Configuration for agility, or combine both |
| 19 | - **Strong Backwards Compatibility**: Ensures upgrades don't break existing configurations and data - Netdata maintains compatibility across versions allowing seamless updates |
| 20 | |
| 21 | ## Platform Capabilities |
| 22 | |
| 23 | | Capability | Linux | Kubernetes | FreeBSD | macOS | Windows | |
| 24 | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------- | -------------- | -------------------------------------------------------------------------------------- | |
| 25 | | **Auto-deploy** | ✅ [kickstart.sh](https://learn.netdata.cloud/docs/netdata-agent/installation/one-line-installer-for-all-linux-systems)¹ | ✅ [Helm](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes) | ✅ kickstart.sh | ✅ kickstart.sh | ✅ [MSI](https://learn.netdata.cloud/docs/netdata-agent/installation/windows)² (silent) | |
| 26 | | **Auto-update** | ✅ Built-in | ✅ Built-in | ✅ Built-in | ✅ Built-in | ⚠️ Manual³ | |
| 27 | | **Auto-discover system metrics** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | |
| 28 | | **Auto-discover all processes** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | |
| 29 | | **Auto-discover containers & VMs** | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ✅ Hyper-V | |
| 30 | | **Auto-discover Docker apps** | ✅ Yes | ✅ Via k8s⁴ | ✅ If Docker | ✅ If Docker | ✅ If Docker | |
| 31 | | **Auto-discover system services** | ✅ systemd | ✅ Yes | ⚠️ Limited | ⚠️ launchd | ✅ Windows Services | |
| 32 | | **Auto-discover enterprise apps** | ✅ netlistensd⁵ | ✅ Via k8s | ❌ Manual | ❌ Manual | ✅ perflib⁶ | |
| 33 | | **Infrastructure as Code** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | |
| 34 | | **Dynamic Configuration** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | |
| 35 | |
| 36 | **Legend**: ✅ Full support | ⚠️ Partial support | ❌ Not available |
| 37 | |
| 38 | **Footnotes**: |
| 39 | 1. **kickstart.sh**: [Universal installer script](https://learn.netdata.cloud/docs/netdata-agent/installation/one-line-installer-for-all-linux-systems) that auto-detects the best installation method |
| 40 | 2. **MSI**: Microsoft Software Installer package for Windows deployment |
| 41 | 3. **Manual**: Auto-updates coming Q3 2025; currently requires PowerShell/SCCM/GPO automation |
| 42 | 4. **Via k8s**: Uses Kubernetes API for service discovery instead of Docker API |
| 43 | 5. **netlistensd**: Local network service discovery - scans for listening services on Linux |
| 44 | 6. **perflib**: Windows Performance Library - provides metrics for Windows applications |
| 45 | |
| 46 | ## Configuration Management Paradigms |
| 47 | |
| 48 | The observability industry uses two primary approaches for configuration management: |
| 49 | |
| 50 | ### Infrastructure as Code (IaC) |
| 51 | |
| 52 | IaC treats configurations as code artifacts that can be versioned, reviewed, and deployed through automated processes. |
| 53 | |
| 54 | **Common tools**: |
| 55 | - **Ansible**: Agentless automation using YAML playbooks |
| 56 | - **Terraform**: Declarative infrastructure provisioning |
| 57 | - **Puppet/Chef**: Agent-based configuration management |
| 58 | - **Salt**: Event-driven automation platform |
| 59 | |
| 60 | **Typical workflow**: |
| 61 | 1. Define configuration in code |
| 62 | 2. Store in version control |
| 63 | 3. Review through pull requests |
| 64 | 4. Deploy via CI/CD pipeline |
| 65 | 5. Validate deployment state |
| 66 | |
| 67 | ### Dynamic Configuration Management |
| 68 | |
| 69 | Dynamic configuration uses a central control plane to manage configurations without requiring code deployments. |
| 70 | |
| 71 | **Common implementations**: |
| 72 | - Web-based configuration interfaces |
| 73 | - API-driven configuration updates |
| 74 | - Real-time configuration synchronization |
| 75 | - Central configuration databases |
| 76 | |
| 77 | **Typical workflow**: |
| 78 | 1. Access central management interface |
| 79 | 2. Modify configuration through UI/API |
| 80 | 3. Changes propagate to agents |
| 81 | 4. Validation occurs at edge nodes |
| 82 | 5. Status reported back to central system |
| 83 | |
| 84 | ### What to Use |
| 85 | |
| 86 | | Approach | When to Use | How | |
| 87 | | ----------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | |
| 88 | | **IaC Only** | Compliance requirements, audit trails | Ansible, Terraform, Puppet, Chef | |
| 89 | | **Dynamic Only** | Small teams, rapid iteration | Netdata Cloud UI or REST API | |
| 90 | | **Hybrid (Best)** | Most organizations | Base config in Git, credentials/thresholds via [UI](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager) | |
| 91 | |
| 92 | ## How Netdata Solves Deployment Challenges at Scale |
| 93 | |
| 94 | Netdata addresses fleet deployment through a comprehensive strategy that minimizes operational overhead while maximizing flexibility: |
| 95 | |
| 96 | ### Universal Single-Script Installation |
| 97 | |
| 98 | Netdata provides a unified installation experience across all platforms (except Windows) through `kickstart.sh`, which implements an intelligent cascade: |
| 99 | |
| 100 | **Installation priority order on Linux**: |
| 101 | 1. **Native binary packages** - For supported distributions (RPM, DEB, SUSE packages) |
| 102 | 2. **Static builds** - Pre-compiled binaries for x86_64, armv7l, armv6l, aarch64 architectures |
| 103 | 3. **Source compilation** - Automatic build from source as final fallback |
| 104 | |
| 105 | **Platform-specific behavior**: |
| 106 | - **Linux/FreeBSD/macOS**: Single command installation via `kickstart.sh` |
| 107 | - **FreeBSD/macOS**: Direct compilation from source (no native packages or static builds) |
| 108 | - **Windows**: Separate MSI installer (auto-updates not yet available) |
| 109 | |
| 110 | ### Automatic Updates |
| 111 | |
| 112 | All Netdata installations (except Windows) auto-update to the latest version: |
| 113 | - Updates are scheduled via systemd timers, cron, or interval scripts |
| 114 | - Maintains the same release channel (stable/nightly) as initially installed |
| 115 | - Zero-downtime updates with automatic rollback on failure |
| 116 | - No manual intervention required for security patches and new features |
| 117 | |
| 118 | **Strong backwards compatibility**: |
| 119 | Netdata ensures upgrades don't break existing configurations and data - Netdata maintains compatibility across versions allowing seamless updates (see [Netdata Infrastructure](https://learn.netdata.cloud/docs/welcome-to-netdata) for architectural details) |
| 120 | |
| 121 | ## How Netdata Solves Fleet Configuration Management at Scale |
| 122 | |
| 123 | Netdata supports both IaC and Dynamic Configuration while minimizing the need for manual configuration through extensive auto-discovery capabilities. |
| 124 | |
| 125 | ### Auto-Discovery and Zero Configuration |
| 126 | |
| 127 | Netdata's primary operational approach is to eliminate manual configuration through comprehensive auto-discovery: |
| 128 | - **On Linux**: Achieves true zero-configuration - 99% of services are automatically detected and monitored |
| 129 | - **On Kubernetes**: Uses Kubernetes API for comprehensive service discovery |
| 130 | - **On Windows**: Auto-discovers enterprise applications via perflib |
| 131 | - **On FreeBSD/macOS**: System metrics are automatic, but application monitoring requires manual configuration |
| 132 | - Discovers new services at runtime through periodic scanning (where supported) |
| 133 | |
| 134 | ### Infrastructure as Code (IaC) Support |
| 135 | |
| 136 | For organizations with established DevOps practices, Netdata fully supports configuration management through traditional IaC tools like [Ansible](https://learn.netdata.cloud/docs/netdata-agent/installation/ansible), allowing version-controlled, auditable deployments. |
| 137 | |
| 138 | ### Dynamic Configuration via the dashboard |
| 139 | |
| 140 | Through [Netdata Cloud and the Dynamic Configuration Manager](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager) and authenticated Netdata Agent and Parent dashboards (the user must sign-in to the dashboard), users can manage collector configurations and alert rules across their entire fleet without touching configuration files, restarting or redeploying agents. |
| 141 | |
| 142 | ### Configuration Priority Order |
| 143 | |
| 144 | When multiple configuration sources exist for the same component, Netdata applies them in the following priority order (highest to lowest): |
| 145 | |
| 146 | 1. **[Dynamic Configuration (DynCfg)](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager)** - Runtime configurations via Netdata Cloud UI or API |
| 147 | 2. **[User Configuration](https://learn.netdata.cloud/docs/netdata-agent/configuration/configuration)** - Files in `/etc/netdata` or `/opt/netdata/etc/netdata` |
| 148 | 3. **Auto-discovered Configuration** - Settings from service discovery mechanisms |
| 149 | 4. **Stock Configuration** - Default configuration files shipped with Netdata |
| 150 | 5. **Internal Defaults** - Built-in defaults in the code |
| 151 | |
| 152 | ## Automatic Discovery |
| 153 | |
| 154 | ### Operating System Metrics |
| 155 | |
| 156 | Netdata auto-detects operating system metrics (compute, memory, networking stack, storage, etc) on all platforms. |
| 157 | |
| 158 | On Linux, Netdata autodetect all kernel modules and technologies which have been instrumented, including firewalls, DDoS protections systems, storage technologies and filesystems, etc. Usually all these technologies require zero configuration. |
| 159 | |
| 160 | Similarly for Windows, Netdata will autodetect everything exposed via Perflib. |
| 161 | |
| 162 | ### [Process and Application Monitoring](https://learn.netdata.cloud/docs/collecting-metrics/processes-and-system-services/applications) (apps.plugin) |
| 163 | The apps.plugin provides intelligent process tree aggregation and monitoring on all platforms (Linux, FreeBSD, macOS, Windows): |
| 164 | |
| 165 | **Intelligent Process Tree Aggregation**: |
| 166 | - Automatically traverses the entire process tree to understand process relationships |
| 167 | - Identifies process managers (spawn servers/orchestrators like systemd, docker, containerd, etc.) |
| 168 | - Creates a finite, manageable set of metrics by intelligently grouping the entire process tree |
| 169 | |
| 170 | **Resource Monitoring** (per application group): |
| 171 | - CPU utilization (user/system, context switches) |
| 172 | - Memory usage (real/virtual, page faults) |
| 173 | - Disk I/O (physical/logical reads/writes) |
| 174 | - Network traffic (if eBPF is enabled on Linux) |
| 175 | - File descriptors and handles |
| 176 | - Process/thread counts |
| 177 | - Accumulated uptime |
| 178 | |
| 179 | **Key Benefits**: |
| 180 | - Zero configuration required - starts monitoring immediately with intelligent defaults |
| 181 | - Captures both running and exited processes, ensuring short-lived processes are accounted for |
| 182 | - Provides instant visibility into resource usage for any application |
| 183 | - Particularly valuable for shell scripts that spawn numerous short-lived subprocesses |
| 184 | - On Windows, automatically monitors all processes and Windows services |
| 185 | |
| 186 | This provides comprehensive application monitoring even for software without specific collectors, making it an essential first line of observability. |
| 187 | |
| 188 | ### [Applications](https://learn.netdata.cloud/docs/collecting-metrics/collectors-configuration) (go.d.plugin) |
| 189 | The go.d.plugin provides auto-discovery for 150+ applications through multiple mechanisms: |
| 190 | |
| 191 | **Service discovery mechanisms**: |
| 192 | - **Local network service discovery (netlistensd)** - Linux-only, uses `local-listeners` binary to detect listening services |
| 193 | - **[Docker container discovery](https://learn.netdata.cloud/docs/collecting-metrics/container-services/docker) (dockersd)** - Discovers applications running in Docker containers |
| 194 | - **[Kubernetes service discovery](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes) (k8ssd)** - Discovers services running in Kubernetes pods |
| 195 | - **[SNMP device discovery](https://learn.netdata.cloud/docs/collecting-metrics/network-devices/snmp) (snmpsd)** - Discovers and profiles SNMP-enabled network devices |
| 196 | - **HTTP service discovery (http)** - Fetches JSON or YAML discovery items from an HTTP endpoint and creates collector jobs through service templates |
| 197 | - **Configuration file scanning** - Detects applications based on their configuration files |
| 198 | |
| 199 | HTTP service discovery templates must generate job configs with `name` and `module`. The `module` is the collector name, such as `httpcheck` or `ping`; it can be omitted only in curated rules where the service rule ID is the intended collector module. |
| 200 | |
| 201 | **Platform-specific behavior**: |
| 202 | |
| 203 | **Linux systems (non-Kubernetes)**: |
| 204 | Full auto-discovery is available through the `local-listeners` utility which: |
| 205 | - Scans for TCP/UDP sockets in LISTEN state every 2 minutes |
| 206 | - Identifies services by their listening ports and process information |
| 207 | - Maintains a cache with 10-minute expiry for discovered services |
| 208 | - Automatically creates collector jobs for recognized services |
| 209 | - Docker container discovery (dockersd) is enabled |
| 210 | |
| 211 | **Non-Linux platforms (FreeBSD, macOS, Windows)**: |
| 212 | - Auto-discovery via netlistensd is **NOT available** |
| 213 | - Docker container discovery (dockersd) works if Docker is available |
| 214 | |
| 215 | **Discovery status management**: |
| 216 | When services are discovered but cannot be monitored, the Dynamic Configuration (DynCfg) system tracks their status: |
| 217 | - **failed** - Collector failed to connect or collect data |
| 218 | - **incomplete** - Configuration requires additional parameters (e.g., credentials) |
| 219 | |
| 220 | Users can view discovered services in the Netdata Cloud UI and supply missing credentials or configuration parameters through the interface, allowing the collectors to retry connection without manual file editing and without restarting Netdata. |
| 221 | |
| 222 | Supported applications include databases (MySQL, PostgreSQL, Redis, MongoDB), web servers (NGINX, Apache, HAProxy), message queues (RabbitMQ, Kafka), SNMP, and [many more](https://learn.netdata.cloud/docs/collecting-metrics/collectors-configuration). |
| 223 | |
| 224 | ### Microsoft Windows |
| 225 | |
| 226 | **Installation**: Windows requires the MSI installer instead of kickstart.sh: |
| 227 | ```powershell |
| 228 | # Silent installation for fleet deployment |
| 229 | msiexec /i netdata-installer.msi /qn /norestart ` |
| 230 | CLAIMING_TOKEN="YOUR_TOKEN" ` |
| 231 | CLAIMING_ROOMS="YOUR_ROOM_ID" ` |
| 232 | CLAIMING_URL="https://app.netdata.cloud" |
| 233 | |
| 234 | # Via Group Policy or SCCM |
| 235 | # Deploy MSI with TRANSFORMS for site-specific configuration |
| 236 | ``` |
| 237 | |
| 238 | **Update Management** (Manual - auto-updates coming): |
| 239 | - Use Windows Update Services (WSUS) or System Center Configuration Manager (SCCM) |
| 240 | - PowerShell DSC (Desired State Configuration) for version enforcement |
| 241 | - Scheduled task to check and download new versions |
| 242 | |
| 243 | **[Windows-Specific Auto-Discovery](https://learn.netdata.cloud/docs/netdata-agent/installation/windows)**: |
| 244 | Windows monitoring is handled by the native **windows.plugin** which uses Windows Performance Counters (perflib) to automatically discover and monitor: |
| 245 | |
| 246 | **Enterprise Applications** (auto-discovered via perflib): |
| 247 | - **IIS** (Internet Information Services) - Web server metrics, site traffic, requests, connections |
| 248 | - **MS SQL Server** - Database performance, transactions, locks, buffer cache |
| 249 | - **MS Exchange** - Mail server metrics, mailbox statistics, transport queues |
| 250 | - **Active Directory** - Domain controller metrics, LDAP operations, replication |
| 251 | - **Active Directory Certificate Services** - Certificate enrollment, requests, revocations |
| 252 | - **Active Directory Federation Services** - Authentication metrics, token issuance |
| 253 | - **Hyper-V** - Virtual machine metrics, CPU/memory allocation, network statistics |
| 254 | - **ASP.NET** - Application performance, request execution, session state |
| 255 | - **.NET Framework** - CLR performance, garbage collection, exceptions |
| 256 | |
| 257 | **System Monitoring** (automatically enabled): |
| 258 | - Windows Services status and health |
| 259 | - Process and thread statistics |
| 260 | - Memory management and paging |
| 261 | - Network interfaces and protocols |
| 262 | - Physical and logical disk performance |
| 263 | - NUMA architecture metrics |
| 264 | - Power supply and battery status |
| 265 | - Thermal zones and sensors |
| 266 | - Semaphore and synchronization objects |
| 267 | |
| 268 | **Process Monitoring**: |
| 269 | - apps.plugin provides detailed per-process and per-application monitoring on Windows |
| 270 | - Automatically groups Windows services and applications |
| 271 | - Monitors all processes for CPU, memory, handles, and I/O usage |
| 272 | |
| 273 | ### Kubernetes Environments |
| 274 | When Netdata runs inside a Kubernetes cluster, it provides comprehensive multi-level discovery: |
| 275 | |
| 276 | **Cluster Monitoring**: |
| 277 | - **[Cluster state](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes)** (k8s_state): Monitors nodes, pods, containers, deployments, services |
| 278 | - **[Kubelet metrics](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes) (k8s_kubelet)**: Container and pod resource usage, volume statistics |
| 279 | - **[Control plane](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes)** (kube-proxy, kube-scheduler, kube-controller-manager): When accessible |
| 280 | |
| 281 | **[Service Discovery](https://learn.netdata.cloud/docs/collecting-metrics/metrics-centralization-points/clustering-and-high-availability-of-netdata-parents)** (k8ssd): |
| 282 | - Automatically enabled, replacing traditional host-based discovery |
| 283 | - Uses Kubernetes API to monitor all pods and services |
| 284 | - Discovers applications by inspecting pod containers and their exposed ports |
| 285 | - Extracts environment variables, ConfigMaps, and Secrets for configuration |
| 286 | - Groups targets by pod, service, and namespace metadata |
| 287 | - Creates monitoring targets for each container: |
| 288 | - One target per exposed port if ports are defined |
| 289 | - One target using pod IP if no ports are exposed |
| 290 | - Tags all metrics with Kubernetes metadata (namespace, pod name, labels, annotations) |
| 291 | |
| 292 | **Discovery Behavior in Kubernetes**: |
| 293 | - **k8ssd** becomes the primary discovery mechanism |
| 294 | - **dockersd** is automatically disabled to avoid conflicts |
| 295 | - **netlistensd** (local-listeners) still works if pod has host network access |
| 296 | - Stock configuration files are ignored (only user-provided configs are loaded) |
| 297 | - Application metrics from Prometheus endpoints are automatically collected |
| 298 | |
| 299 | Most Kubernetes deployments require no configuration beyond the initial [Helm chart installation](https://learn.netdata.cloud/docs/netdata-agent/installation/kubernetes-helm-chart-reference). |
| 300 | |
| 301 | ## Comparison with Other Platforms |
| 302 | |
| 303 | Understanding how different platforms handle configuration helps in planning migrations or hybrid deployments: |
| 304 | |
| 305 | ### Configuration Requirements by Platform |
| 306 | |
| 307 | | Platform | Components per Node | Configuration Method | Auto-Discovery | Updates & Restarts | |
| 308 | | ------------- | ---------------------------- | ----------------------------- | -------------- | ------------------------------------ | |
| 309 | | Netdata | **Single agent** | Files, Ansible, or Dynamic UI | **Extensive** | **Zero-downtime for Dynamic Config** | |
| 310 | | Prometheus | 5-20 exporters | YAML files per exporter | Limited | Rolling restarts | |
| 311 | | Datadog | Agent + integrations | YAML files | Moderate | Agent restart | |
| 312 | | OpenTelemetry | Collectors + instrumentation | YAML files per collector | Limited | Full restart | |
| 313 | |
| 314 | ## Related Documentation |
| 315 | - [Parent-Child Configuration Reference](https://learn.netdata.cloud/docs/netdata-parents/parent-child-configuration-reference) |
| 316 | - [Parent Configuration Best Practices](https://learn.netdata.cloud/docs/netdata-parents/parent-configuration-best-practices) |
| 317 | - [Streaming and Routing Reference](https://learn.netdata.cloud/docs/netdata-parents/streaming-routing-reference) |
| 318 | - [Dynamic Configuration Manager](https://learn.netdata.cloud/docs/netdata-agent/configuration/dynamic-configuration-manager) |