0a8997669 · cryptotaxi247/netdata-1

@@ -1,29 +1,73 @@

1 +

2 # Netdata Container Security Considerations

3

3 -This document outlines the necessary permissions and security considerations when deploying Netdata within containerized environments, such as Kubernetes. Achieving comprehensive monitoring of the host system and other containers necessitates granting specific privileges to the Netdata agent container. Understanding the implications of these permissions is crucial for balancing observability requirements with security posture.

4 +Netdata provides powerful, real-time performance monitoring and health visibility, offering deep insights essential for managing modern infrastructure, particularly dynamic containerized environments like Kubernetes. Its high-granularity metrics and comprehensive system overview help teams troubleshoot issues quickly and optimize performance effectively.

5 +

6 +To deliver observability that extends beyond the agent's own container to encompass the entire host system and other running containers, the Netdata agent ('child' pods in a typical Kubernetes deployment) requires specific access to host resources. This access typically involves mounting certain host paths (usually read-only), utilizing host namespaces (like the PID and Network namespaces), and leveraging specific Linux capabilities (such as `SYS_PTRACE` or `SYS_ADMIN` for certain advanced features).

7 +

8 +Granting such permissions inherently involves security considerations. This document aims to provide transparency regarding the permissions Netdata might request for full functionality. We will detail these permissions, explain the rationale behind them, and discuss Netdata's security design features – including its privilege separation model and constrained network architecture – which are intended to mitigate potential risks.

9 +

10 +The goal is to enable organizations to tailor their Netdata deployment, striking an appropriate balance between achieving deep, actionable observability and adhering to their specific security policies and risk tolerance. Understanding these configurations and their trade-offs is key to deploying Netdata effectively and securely within your containerized environment.

11 +

12 +## Security Architecture: The Privilege Separation Model

13 +

14 +Netdata is architected with security as a core consideration, employing a robust privilege separation model designed to minimize attack surfaces and isolate operations that require elevated permissions. This model applies both to the overall deployment architecture and the internal workings of the Netdata agent itself.

15 +

16 +### Parent / Child Deployment Architecture

17 +In typical Kubernetes deployments, Netdata utilizes a hierarchical structure with two main roles:

18

5 -## Recommended Permissions for Comprehensive Host and Container Monitoring

19 +- **`Parent` Pods:** These act as central aggregation points. They receive data streams from multiple child pods. Crucially, `Parent` pods are designed to run as **completely unprivileged containers**. They do not require access to host paths, host namespaces, or any special Linux capabilities. Their primary role is data aggregation, storage, and exposing Netdata's API and dashboard or connecting to Netdata Cloud for servicing the dashboard.

20 +- **`Child` Pods (Agents):** A `child` pod runs as a DaemonSet on each monitored node. Its responsibility is to collect detailed metrics, query logs, and other information directly from the node it resides on. To achieve comprehensive host and container monitoring, the `child` pod is the component that may require specific host resource access (mounts, namespaces) and potentially elevated Linux capabilities.

21

7 -To enable full monitoring capabilities, including host-level metrics, process information, and container resource usage, the Netdata child pods (agents running on each node) require access to specific host resources and namespaces, as well as certain Linux capabilities.

22 +This architecture ensures that the components responsible for broader network exposure and data storage (Parents) run with minimal privileges, while the necessary host access is confined to the node-local `child` agents.

23

9 -### Privilege Separation Security Model

24 +### Internal Privilege Separation within `Child` Agents

25 +Even within the `child` agent container, which might be granted elevated permissions by Kubernetes (e.g., host mounts, capabilities), Netdata enforces strict internal privilege separation:

26

11 -Netdata uses a privilege separation model for its processes in order to isolate the risk associated with permissions and capabilities required to achieve comprehensive observability:

27 +- **Unprivileged Core Daemon:** The main `netdata` daemon process, which orchestrates data collection, communicates with the parent, and handles basic tasks, runs as a **non-root user** (typically the `netdata` user). Key internal plugins responsible for collecting core metrics, such as `proc.plugin` (for `/proc` based metrics) and `cgroups.plugin` (for container metrics via `/sys/fs/cgroup`), also run as threads within this unprivileged daemon process.

28 +- **Dedicated Privileged Helpers/Plugins:** Operations that require elevated privileges or specific capabilities to access protected host resources (e.g., detailed process information via `apps.plugin`, certain network operations via `network-viewer.plugin` or `cgroup-network`, logs querying via `systemd-journal.plugin`) are delegated to **dedicated external plugins or helper processes**.

29 +- **Isolation via `setuid`:** These external plugins/helpers run as separate, isolated processes spawned by the main daemon. Within the container, the specific helper executables requiring elevated rights are configured with the **`setuid` root** permission bit. This standard Linux mechanism allows only these designated programs to execute specific, pre-defined tasks with root privileges _within the container's environment_.

30 +- **Contained Capabilities:** Consequently, even if the _container_ is granted capabilities like `SYS_ADMIN` or `SYS_PTRACE`, the unprivileged main `netdata` daemon and its non-`setuid` plugins **cannot** directly utilize these capabilities. Only the specific `setuid` root helpers designed for tasks needing those privileges can effectively leverage them.

31 +- **Narrowly Scoped Tasks:** Each privileged helper is designed to perform a narrow, hard-coded task – typically reading specific system information and passing it back to the main daemon.

32 +- **Text-Based Communication:** Communication between the main daemon and its external plugins occurs over a simple, text-based protocol, further reducing the risk associated with potentially malicious binary data exchange.

33

13 -1. The Netdata daemon and it internal plugins run unprivileged as a normal user, without any access to sensitive system information or capabilities. This includes `proc.plugin`, `cgroups.plugin` and the core Netdata daemon.

14 -2. When elevated permissions are needed to access sensitive or protected information, either a dedicated/isolated external plugin is used, or an external helper process is utilized to collect the needed information. When Netdata runs in a container, only some of the Netdata external plugins and helpers have elevated privileges, while the core of Netdata with its internal plugins do not.

34 +### Security Benefit

35 +This multi-layered privilege separation model significantly minimizes the attack surface. The vast majority of the Netdata agent's code, including the core daemon and primary metric collectors, runs without root privileges. Elevated permissions are isolated to specific, purpose-built helper processes with limited scope. This design helps contain the risk even when the Netdata `child` container requires privileged access to the host system for comprehensive monitoring. Compromising the main unprivileged daemon does not automatically grant an attacker root access or the ability to use the container's granted capabilities.

36

16 -Netdata talks to its plugins using a text based protocol, which eliminates the possibility of any binary data being exchanged, especially from the Netdata daemon to its plugins.

37 +## Network Exposure and Access Control

38 +Beyond internal process separation within the agent, Netdata's deployment architecture incorporates several features designed to limit network exposure and manage data access securely, particularly in the recommended parent/child configuration.

39

18 -Using this design, Netdata minimizes the potential attack surface. The main daemon does not require anything special and external plugins perform a hard-coded task for reading some specific sensitive information and returning it back to the main daemon.

40 +### Child Agent Network Limitations

41 +Netdata `child` agents (running on each monitored node) are designed with minimal required network interaction to reduce their direct attack surface:

42

20 -### Netdata Network Exposure

43 +- **Outbound-Only to Parent:** Their primary network function is to stream collected data **outbound** exclusively to their designated `parent` pod(s) within the cluster. This connection can be further secured using transport layer security (TLS) and optional mutual TLS (mTLS) for certificate-based authentication between child and parent.

44 +- **Restricted Inbound Connections:** By default, standard configurations (like the official Helm chart) configure `child` agents to listen **only on the localhost interface (127.0.0.1)**. This prevents them from accepting incoming network connections from other pods, nodes, or external sources.

45 +- **No Direct Cloud Connectivity Needed:** `Child` agents **do not require** direct internet connectivity to the Netdata Cloud service. All communication is proxied through the parent(s).

46

22 -Netdata helm charts limit Netdata children connectivity to `localhost`. Netdata children are not allowed to accept any Network connections from outside the local node, and the only outbound network connection they need is towards the Netdata Parent to stream their data. The Netdata children also do not maintain a database on disk. All information they collect is streamed in real-time to their Netdata Parent.

47 +### Parent Agent Role and Centralized Access via Netdata Cloud

48 +`Parent` pods act as the aggregation points and data access gateways:

49

24 -Netdata Parents on the other hand, do not require any mounts, host namespaces, or capabilities. They run in an unprivileged container, ingesting data in real-time from Netdata children and exposing this information via their APIs.

50 +- **Central Aggregation:** They receive data streams from multiple child agents.

51 +- **Optional Direct Access:** Parents expose the Netdata API and dashboard. While direct access is possible, it requires careful network policy configuration within the cluster.

52 +- **Recommended Cloud Integration:** The typical and recommended approach, especially for managing multiple nodes or clusters, involves securely connecting the `parent` pods **outbound** to Netdata Cloud.

53 +- **Centralized RBAC via Cloud:** Netdata Cloud serves as the primary user interface and provides essential access control mechanisms. This includes **Role-Based Access Control (RBAC)**, allowing administrators to:

54 + - Manage users and team memberships.

55 + - Segment the infrastructure into "rooms" (by service, component type, location, etc)

56 + - Assign roles with differing permission levels.

57 + - Associate users with infrastructure, by granting users access to specific "rooms".

58

26 -Netdata Children do not need to connect to Netdata Cloud. Netdata Parents only need to connect to Netdata Cloud.

59 +### Security Benefits

60 +

61 +This layered network architecture significantly enhances security:

62 +

63 +- It drastically limits the direct network exposure of the potentially privileged `child` agents.

64 +- It centralizes data egress and user access points at the `parent` level.

65 +- It leverages Netdata Cloud for robust, centralized authentication and Role-Based Access Control, moving access management off the individual nodes.

66 +- It supports encrypted and mutually authenticated communication channels between agents and parents.

67 +

68 +## Permissions for Complete Host and Container Monitoring

69 +

70 +The following sections describe all the possible mounts, capabilities and host namespaces Netdata children may use:

71

72 ### Container Mounts

73

@@ -39,7 +83,7 @@ Mounting specific host directories into the Netdata container provides essential

@@ -57,55 +101,78 @@ Notes:

101

102 Utilizing host namespaces allows Netdata to observe network activity and processes as they appear on the host, rather than being confined to the container's isolated view.

103

61 -|:---:|:---:|:---:|:---|

105 +|:---:|:---:|:---:|:---:|:---|

106 +| Host Network Namespace | child | `proc.plugin` | Internal Plugin | Monitor host's networking stack (the IPv4 and IPv6 Networking Stack). `proc.plugin` runs unprivileged.   Without this, Netdata will monitor the networking stack of the container it runs, not the one of the host. |

107 +| Host Network Namespace | child | `cgroup-network` | External Helper | Used by `cgroups.plugin` to map virtual network interfaces to the containers they belong. The helper requires also special capabilities. `cgroups.plugin` runs unprivileged.   Without this, Netdata will monitor only the network interfaces of the container it runs, not the network interfaces of its sibling containers (or the host). |

108 +| Host Network Namespace | child | `local-listeners` | External Helper | Used by `go.d.plugin` to discover applications running at the host (including sibling containers). It requires also special capabilities. `go.d.plugin` runs unprivileged.   Without this, Netdata will not be able to auto-detect and monitor applications running at the host or its sibling containers. Kubernetes based service discovery is not affected. |

109 +| Host Network Namespace | child | `network-viewer.plugin` | External Plugin | Discover and continuously monitor the host's network connections (including the ones of sibling containers). It requires also special capabilities. `network-viewer.plugin` runs with escalated privileges.   Without this, Netdata will monitor only the connections of the container it runs, not the connections of the host or its sibling containers. |

110 +| Host PID Namespace | child | `cgroup-network` | External Helper | Used by `cgroups.plugin` to detect containers' network interfaces. `cgroups.plugin` runs unprivileged.   Without this, Netdata will not be able to associate virtual network interfaces to the containers they belong. |

111

68 -### Container Capabilities

112 +### Linux Capabilities

113

114 Specific Linux capabilities grant elevated privileges necessary for certain monitoring functions, particularly those involving process inspection and namespace manipulation.

115

73 -|:---:|:---:|:---:|:---|

74 -| SYS_ADMIN | child | `cgroup-network` | Associate containers' network interfaces with the containers (it does so by switching Network Namespaces). Without it, `veth` network interfaces will not be associated to their respective containers, so they will be monitored as host network interfaces. |

75 -| SYS_ADMIN | child | `network-viewer.plugin` | Discover containers' network connections (it does so by switching Network Namespaces). Without it, network connections of other containers will not be monitored, limiting the scope of network connections to the host system. |

77 -| SYS_PTRACE | child | `network-viewer.plugin` | Discover host's network connections per application. Without it all host network connections will still be monitored, but Netdata will not be able to associate them with processes. |

78 -| SYS_PTRACE | child | `go.d.plugin` | Discover listening applications running at the host. Without it, localhost service discovery (not the kubernetes one), `go.d.plugin` will judge about the applications running based only on port number, without the process name. |

117 +|:---:|:---:|:---:|:---:|:---|

118 +| SYS_ADMIN | child | `cgroup-network` | External Helper | Used by `cgroups.plugin` to associate virtual network interfaces to the containers the belong.   Without this, virtual network interfaces will not be associated to their respective containers, so they will be monitored as host network interfaces. |

122 +| SYS_PTRACE | child | `local-listeners` | External Helper | Helper used by `go.d.plugin` to discover listening applications running at the host.   Without this, localhost service discovery (not the kubernetes one) will be based only on port number, without the process name. |

124

125 IMPORTANT: All the plugins or helpers that utilize these capabilities are isolated from the rest of Netdata. This means that all the other plugins of Netdata and the main Netdata daemon cannot utilize these capabilities, even when the capabilities have been given to the whole container Netdata is installed.

126

82 -## Impact of Security Restrictions on Monitoring Scope

127 +## Configuration Strategies and Monitoring Scope

128 +

129 +Administrators can configure Netdata `child` agents with varying levels of host access, directly impacting both the depth of observability achieved and the security posture of the deployment. Understanding the trade-offs associated with different configurations is essential for aligning Netdata with organizational security policies and monitoring requirements. We outline three primary approaches below:

130

84 -### Operating with Minimal Privileges (Restricted Permissions)

131 +### Full Monitoring Configuration

132

86 -If Netdata child pods are deployed without the aforementioned host path mounts, host namespace access, and container capabilities, the monitoring scope will be significantly reduced.

133 +This configuration grants the Netdata `child` agent **all** the permissions detailed in the preceding section. This includes all listed host path mounts (read-only where specified), both Host PID and Host Network namespaces, and the `SYS_ADMIN` and `SYS_PTRACE` capabilities.

134

88 -- **Functionality:** Netdata will start and operate, but it will primarily monitor **only the resources consumed by the Netdata container itself**. This includes its own CPU/memory usage, internal processes, and network activity within its isolated namespace.

89 -- **Limitations:** Host-level metrics (overall CPU, memory, network stack, disk I/O), processes running outside the Netdata container, direct container resource usage monitoring (via cgroups), and host system logs will **not** be available.

90 -- **Kubernetes Integration:** If appropriate RBAC permissions are granted to query the Kubernetes API, Netdata can still provide Kubernetes state metrics (e.g., pod counts, node status). Kubernetes-based application discovery may also function, allowing Netdata to collect metrics from _other_ containers if they expose compatible endpoints accessible via the K8s API/network, but without the deep host-level process correlation.

91 -- **Persistence Issue:** Without the persistent `/var/lib/netdata` volume mount, the Netdata agent will lose its unique identity upon restart. Each time the pod is rescheduled or restarted, it will appear as a completely new node in the Netdata UI or Cloud dashboard.

135 +**Monitoring Scope (Benefit):** Provides the **maximum possible observability** offered by Netdata. This includes deep insights into the host system's performance, detailed metrics for all running containers, comprehensive process monitoring (including I/O), full network connection visibility and attribution, host system log integration, and accurate mapping of all resources.

136

93 -### Balanced approach

137 +**Security Risk Profile:** **Highest.** While Netdata's internal privilege separation and network exposure controls mitigate certain threats, this configuration grants the container extensive access to host resources and powerful capabilities.

138

95 -This approach aims to provide substantial host and container observability while significantly reducing the security risks compared to the "full monitoring" configuration. It operates on the principle of least privilege, excluding the most dangerous capabilities and mounts by default, while retaining access needed for core monitoring functions. This involves accepting certain trade-offs in the depth or context of collected data.

139 +**Use Case:** Environments where absolute maximum visibility is the top priority, and the organization understands, formally accepts, and has potentially implemented compensating controls (e.g., advanced runtime security, strict network policies) to manage the associated risks.

140

97 -#### Capabilities: Exclude `SYS_ADMIN` and `SYS_PTRACE`

141 +### Minimal Privileges / Isolated Configuration

142

99 -These capabilities grant excessive privileges with high potential for misuse or exploitation. `SYS_ADMIN` offers broad administrative control, while `SYS_PTRACE` allows invasive inspection of any process. Excluding them dramatically reduces the potential impact of a container compromise.

143 +This configuration runs the Netdata `child` agent with **minimal privileges**, mirroring a standard, non-privileged container workload. It **excludes all specific host path mounts** (like `/proc`, `/sys`, `/var/log`, `/`), **host namespaces** (PID and Network), and **elevated capabilities** (`SYS_ADMIN`, `SYS_PTRACE`). Persistence via `/var/lib/netdata` might be optionally retained but only benefits the isolated container's identity.

144

101 -**Impact:**

102 -* Container network interfaces (`veth`) may appear as host interfaces, lacking direct container attribution within Netdata's network interface metrics.

103 -* Direct discovery of _other_ containers' network connections via namespace switching will be disabled.

104 -* Per-process physical/logical disk I/O metrics via `apps.plugin` will be unavailable.

105 -* Network connections and listening ports will not be directly associated with specific process names by `network-viewer.plugin` or `go.d.plugin`.

145 +**Monitoring Scope (Limitation):** **Severely limited.** Netdata primarily monitors **only its own container's resource consumption** (CPU, memory, its own processes, network traffic within its isolated namespace). It gains **no visibility** into the host's overall performance, other containers, host processes, host network stack, or host system logs.

146 +

147 + _Note:_ If Kubernetes RBAC permissions are granted, Netdata can still collect **Kubernetes state metrics** (node/pod counts, status, etc.) via the K8s API. Application metrics might be collected if exposed by other pods and reachable via standard K8s networking, but without host-level correlation.

148 +

149 +**Security Risk Profile:** **Lowest.** This configuration offers the best security isolation for the Netdata agent, posing minimal direct risk to the host or other containers from the agent itself.

150 +

151 +**Use Case:** Netdata is used _only_ for Kubernetes API-level state metrics.

152 +

153 +### Balanced Approach Configuration

154 +

155 +This configuration aims for substantial observability while mitigating the highest risks, assuming Netdata is the primary observability platform and acknowledging its security practices.

156 +

157 + - **Permissions Included:** Host path mounts `/proc` (ro), `/sys` (ro), `/etc/os-release` (ro), `/etc/passwd` (ro), `/etc/group` (ro), `/var/log` (ro), `/var/lib/netdata` (rw); Host PID and Network Namespace.

158 +

159 + - **Permissions Excluded (by Default):** Linux capabilities `SYS_ADMIN` and `SYS_PTRACE`; Host path mount `/` (for K8s).

160 +

161 +**Monitoring Scope (Benefit):** Provides **comprehensive core monitoring** capabilities: host system performance (CPU, Mem, Disk, Basic Net I/O via `/proc`), container discovery and resource usage (via cgroups in `/sys`), host-wide process listing (via Host PID and `/proc`), and integrated host system log analysis (via `/var/log` mount). Agent persistence is maintained.

162 +

163 +**Security Risk Profile:** **Medium.** This configuration still requires significant host access via read-only mounts and the Host PID and Network namespaces, carrying inherent risks of information disclosure (sensitive host system logs, process lists, system configuration) and reduced process isolation. However, it avoids the most dangerous capabilities (`SYS_ADMIN`, `SYS_PTRACE`), significantly reducing the potential for direct system compromise or interference compared to the Full Monitoring setup. The risk associated with `/var/log` access is explicitly included but limited to read-only information disclosure.

164 +

165 +**Key Trade-offs (Limitations vs. Full Monitoring):** Excludes advanced features requiring the omitted capabilities, such as per-process disk I/O metrics, precise container `veth` interface attribution, direct monitoring of other containers' network connections, process name association for network activity, and certain host application discovery methods.

166 +

167 +**Use Case:** A **sensible default baseline** for organizations deploying Netdata as their primary observability platform in security-conscious environments. It offers a strong balance of visibility and risk reduction. Enabling the excluded higher-risk permissions should be a conscious decision, requiring justification and formal risk acceptance if specific advanced features are deemed essential.

168

107 -We suggest to keep the rest (mounts and host namespaces) enabled, so that Netdata can still provide its full monitoring features.

169

170 ## Conclusion

171

111 -Deploying Netdata for comprehensive system monitoring requires granting access to host resources. While these permissions enable deep visibility, they represent a security trade-off. Administrators must carefully evaluate their organization's security policies and monitoring requirements to determine the appropriate level of privilege for Netdata agents. Running with restricted permissions enhances security isolation but severely limits Netdata's ability to monitor the underlying host and other containerized workloads effectively.

172 +Netdata offers extensive real-time visibility into containerized environments like Kubernetes, providing invaluable data for performance optimization, troubleshooting, and understanding system health. Achieving this deep level of insight, however, inherently requires granting the Netdata agent specific permissions to access host resources, including file system mounts, host namespaces, and potentially powerful Linux capabilities.

173 +

174 +This document has detailed the permissions utilized by Netdata for comprehensive monitoring, explained the rationale behind them, and outlined Netdata's security architecture features – such as its privilege separation model and constrained network exposure – designed to mitigate the associated risks. We have also explored different configuration strategies, ranging from maximum observability with corresponding higher risk ("Full Monitoring") to maximum isolation with limited visibility ("Minimal Privileges").

175 +

176 +The optimal configuration depends heavily on each organization's specific security posture, risk tolerance, compliance requirements, and observability needs. Administrators are strongly encouraged to carefully review the trade-offs detailed for each approach and select the configuration that best aligns with their environment's specific context.

177 +

178 +Effective infrastructure management requires continuous attention. Regularly reviewing Netdata's configuration and permissions, alongside evolving security best practices and organizational requirements, is essential for maintaining a secure and effectively monitored system.

Update netdata-kubernetes.md