| 1 | # Applications monitoring (apps.plugin) |
| 2 | |
| 3 | `apps.plugin` monitors the resources utilization of all processes running. |
| 4 | |
| 5 | ## Process Aggregation and Grouping |
| 6 | |
| 7 | `apps.plugin` aggregates processes in three distinct ways to provide a more insightful breakdown of resource utilization: |
| 8 | |
| 9 | | Grouping | Description | |
| 10 | |------------|----------------------------------------------------------------------------------------------------------------------------------------------------| |
| 11 | | App | Grouped by the position in the process tree. This is customizable and allows aggregation by process managers and individual processes of interest. | |
| 12 | | User | Grouped by the effective user (UID) under which the processes run. | |
| 13 | | User Group | Grouped by the effective group (GID) under which the processes run. | |
| 14 | |
| 15 | ## Short-Lived Process Handling |
| 16 | |
| 17 | `apps.plugin` accurately captures resource utilization for both running and exited processes, ensuring that the impact of short-lived subprocesses is fully accounted for. |
| 18 | This is particularly valuable for scenarios where processes spawn numerous short-lived subprocesses, such as shell scripts that fork hundreds or thousands of times per second. |
| 19 | Even though these subprocesses may have a brief lifespan, `apps.plugin` effectively aggregates their resource utilization, providing a comprehensive overview of how resources are shared among all processes within the system. |
| 20 | |
| 21 | ## PSS Memory Estimation |
| 22 | |
| 23 | On Linux systems with kernel 4.14 or later, `apps.plugin` uses Proportional Set Size (PSS) data to provide more accurate memory usage estimates for processes that use shared memory. |
| 24 | |
| 25 | PSS is an expensive kernel operation that requires scanning all shared memory segments of a process to determine which memory pages are shared with other processes, and then proportionally dividing the shared memory among them to calculate each process's actual memory footprint. Since PSS for any process can change due to actions by other processes (such as mapping or unmapping the same files, or processes exiting), maintaining accurate real-time PSS data for all processes would be prohibitively expensive. |
| 26 | |
| 27 | To balance accuracy with performance, `apps.plugin` uses a **ratio-based estimation approach**: it periodically samples PSS values to calculate a PSS/RSS ratio for each process, then applies this cached ratio to the current RSS values **every second** to estimate memory usage. This means that estimated memory values are updated every second based on current RSS, while the ratio itself is recalibrated adaptively based on process priority. |
| 28 | |
| 29 | The plugin implements an **adaptive sampling strategy** designed to prioritize the largest memory consumers and processes with significant memory changes, refreshing them within seconds of detection, while guaranteeing that all processes are eventually sampled within **twice the configured interval** (10 minutes by default for a 5-minute interval). The plugin alternates between two complementary prioritization strategies each iteration: |
| 30 | |
| 31 | 1. **Delta-Based Strategy**: Prioritizes processes with the largest changes in shared memory, ensuring rapid detection and response to memory growth. Large memory consumers (databases, cache servers, etc.) are typically refreshed within seconds when their memory footprint changes significantly. |
| 32 | |
| 33 | 2. **Age-Based Strategy**: Prioritizes processes that haven't been sampled longest, ensuring eventual consistency for all processes. Even the smallest processes are guaranteed to be refreshed within twice the configured interval. |
| 34 | |
| 35 | Both strategies sort candidates by priority and refresh the top N processes within the configured budget each iteration. By alternating between these strategies, the plugin ensures responsive tracking of significant memory changes while maintaining bounded staleness for all processes. |
| 36 | |
| 37 | Additionally, on the first iteration after startup, the plugin samples all processes to establish accurate initial estimates before switching to the adaptive sampling strategy. |
| 38 | |
| 39 | ## Charts |
| 40 | |
| 41 | `apps.plugin` offers a set of charts for three groups within the **System->Processes** section of the Netdata dashboard: **Apps**, **Users**, and **Groups**. |
| 42 | |
| 43 | Each of these sections presents the same number of charts: |
| 44 | |
| 45 | - CPU utilization |
| 46 | - Total CPU usage |
| 47 | - User/system CPU usage |
| 48 | - Memory |
| 49 | - Estimated Memory Usage (RSS with PSS scaling, default on Linux 4.14+) |
| 50 | - Memory RSS Usage |
| 51 | - Real Memory Used (non-shared) |
| 52 | - Virtual Memory Allocated |
| 53 | - Minor page faults (i.e. memory activity) |
| 54 | - Swap memory |
| 55 | - Swap memory used |
| 56 | - Major page faults (i.e. swap activity) |
| 57 | - Disk |
| 58 | - Physical reads/writes |
| 59 | - Logical reads/writes |
| 60 | - Tasks |
| 61 | - Threads |
| 62 | - Processes |
| 63 | - FDs |
| 64 | - Open file descriptors limit % |
| 65 | - Open file descriptors |
| 66 | - Uptime |
| 67 | - Carried over uptime (since the last Netdata Agent restart) |
| 68 | |
| 69 | In addition, if the [eBPF collector](/src/collectors/ebpf.plugin/README.md) is running, your dashboard will also show an |
| 70 | additional [list of charts](/src/collectors/ebpf.plugin/README.md#integration-with-appsplugin) using low-level Linux |
| 71 | metrics. |
| 72 | |
| 73 | ## Performance |
| 74 | |
| 75 | `apps.plugin` is designed to be highly efficient, collecting significantly more process information than other similar tools while maintaining exceptional speed. |
| 76 | However, due to its comprehensive approach of traversing the entire process tree on each iteration, its resource usage may become noticeable, especially on systems with a large number of processes. |
| 77 | |
| 78 | Under Linux, `apps.plugin` reads multiple `/proc` files for each running process, performing this operation on a per-second basis. |
| 79 | This can lead to increased CPU consumption on hosts with several thousands of processes. |
| 80 | |
| 81 | In such cases, you may need to adjust the data collection frequency to reduce the plugin's resource usage. |
| 82 | |
| 83 | To do this, edit `/etc/netdata/netdata.conf` and find this section: |
| 84 | |
| 85 | ```text |
| 86 | [plugin:apps] |
| 87 | # update every = 1 |
| 88 | # command options = |
| 89 | ``` |
| 90 | |
| 91 | Uncomment the `update every` line and set it to a higher value. |
| 92 | For example, setting it to 2 will halve the plugin's CPU usage and collect data once every 2 seconds. |
| 93 | |
| 94 | ## Configuration |
| 95 | |
| 96 | The configuration file is `/etc/netdata/apps_groups.conf`. You can edit this |
| 97 | file using our [`edit-config`](/docs/netdata-agent/configuration/README.md#edit-configuration-files) script. |
| 98 | |
| 99 | ### Configuring process managers |
| 100 | |
| 101 | `apps.plugin` needs to know the common process managers, which are the processes that spawn other processes. |
| 102 | These process managers allow `apps.plugin` to automatically include their subprocesses in the monitoring process, ensuring that important processes are not overlooked. |
| 103 | |
| 104 | - Process managers are configured in the `apps_groups.conf` file using the `managers:` prefix, as follows: |
| 105 | |
| 106 | ```text |
| 107 | managers: process1 process2 process3 |
| 108 | ``` |
| 109 | |
| 110 | - Multiple lines can be used to define additional process managers, all starting with `managers:`. |
| 111 | |
| 112 | - If you want to clear all existing process managers, you can use the line `managers: clear`. This will remove all previously configured managers, allowing you to provide a new list. |
| 113 | |
| 114 | ### Configuring interpreters |
| 115 | |
| 116 | Interpreted languages like `python`, `bash`, `sh`, `node`, and others may obfuscate the actual name of a process. |
| 117 | |
| 118 | To address this, `apps.plugin` allows you to configure interpreters and specify that the actual process name can be found in one of the command-line parameters of the interpreter. |
| 119 | When a process matches a configured interpreter, `apps.plugin` will examine all the parameters of the interpreter and locate the first parameter that is an absolute filename existing on disk. If such a filename is found, `apps.plugin` will name the process using the name of that filename. |
| 120 | |
| 121 | - Interpreters are configured in the `apps_groups.conf` file using the `interpreters:` prefix, as follows: |
| 122 | |
| 123 | ```text |
| 124 | interpreters: process1 process2 process3 |
| 125 | ``` |
| 126 | |
| 127 | - Multiple lines can be used to define additional process managers, all starting with `interpreters:`. |
| 128 | |
| 129 | - If you want to clear all existing process interpreters, you can use the line `interpreters: clear`. This will remove all previously configured interpreters, allowing you to provide a new list. |
| 130 | |
| 131 | ### Configuring process groups and renaming processes |
| 132 | |
| 133 | - The configuration file supports multiple lines, each following this format: |
| 134 | |
| 135 | ```text |
| 136 | group: process1 process2 ... |
| 137 | ``` |
| 138 | |
| 139 | - You can define a group multiple times to include additional processes within it. |
| 140 | |
| 141 | - For each process specified, all of its subprocesses will be automatically grouped, not just the matched process itself. |
| 142 | |
| 143 | ### Matching processes |
| 144 | |
| 145 | `apps.plugin` uses different fields for process matching depending on the operating system: |
| 146 | |
| 147 | #### Unix-like systems (Linux, FreeBSD, macOS) |
| 148 | |
| 149 | | Field | Description | Example | |
| 150 | |---------|----------------------------------|-----------------------------------------| |
| 151 | | comm | Process name (command) | `chrome` | |
| 152 | | cmdline | Full command line with arguments | `/usr/bin/chrome --enable-features=...` | |
| 153 | |
| 154 | > **Note:** On Linux specifically, the **comm** field is limited to 15 characters from `/proc/{PID}/comm`. |
| 155 | > `apps.plugin` attempts to obtain the full process name by searching for it in the **cmdline**. |
| 156 | > If successful, the entire process name is used; otherwise, the shortened version is used. |
| 157 | |
| 158 | #### Windows process fields |
| 159 | |
| 160 | | Field | Description | Example | |
| 161 | |---------|------------------------------------------------------------------|---------------------------------------------------------| |
| 162 | | comm | Performance Monitor instance name (may include instance numbers) | `chrome#12` | |
| 163 | | cmdline | Full path to the executable (without command line arguments) | `C:\Program Files\Google\Chrome\Application\chrome.exe` | |
| 164 | | name | Friendly name from file description or service display name | `Google Chrome` | |
| 165 | |
| 166 | > On Windows: |
| 167 | > - All pattern types (exact, prefix, suffix, substring) also match against the **name** field |
| 168 | > - The **name** field is preferred for default grouping when no pattern matches |
| 169 | > - Instance numbers (e.g., `#1`, `#2`) are automatically stripped from the **comm** field |
| 170 | > - The `.exe` extension is automatically removed for cleaner display |
| 171 | > - For services (especially `svchost.exe`), the service display name is resolved |
| 172 | |
| 173 | #### Pattern matching |
| 174 | |
| 175 | You can use asterisks (`*`) to create patterns: |
| 176 | |
| 177 | > **Version differences:** |
| 178 | > - **Netdata v2.5.2 and earlier**: Pattern matching is case sensitive |
| 179 | > - **Netdata v2.5.3 and later**: Pattern matching is case insensitive |
| 180 | > - **Netdata v2.5.2 and earlier**: Windows patterns match against `comm` and `cmdline` fields |
| 181 | > - **Netdata v2.5.3 and later**: Windows patterns match against `comm`, `cmdline`, and `name` (friendly name) fields |
| 182 | |
| 183 | | Mode | Pattern | Description | Unix-like | Windows | |
| 184 | |-----------|-------------|----------------------------------------|---------------------------|-------------------| |
| 185 | | exact | `firefox` | Matches **comm** exactly | ✓ Yes | ✓ Yes | |
| 186 | | prefix | `firefox*` | Matches **comm** starting with firefox | ✓ Yes | ✓ Yes | |
| 187 | | suffix | `*fox` | Matches **comm** ending with fox | ✓ Yes | ✓ Yes | |
| 188 | | substring | `*firefox*` | Searches within **cmdline** | ✓ Yes (full command line) | ✓ Yes (full path) | |
| 189 | |
| 190 | **Note on substring matching (`*pattern*`):** |
| 191 | |
| 192 | - On Unix-like systems: Searches within the full command line including arguments |
| 193 | - On Windows: Searches within the full executable path (e.g., `C:\Program Files\Mozilla Firefox\firefox.exe`) |
| 194 | |
| 195 | - Asterisks can be placed anywhere within pattern (e.g., `fi*fox`) without affecting the matching criteria (**comm** or **cmdline**). |
| 196 | - To include process names with spaces, enclose them in quotes (single or double), like this: `'Plex Media Serv'` or `"my other process"`. |
| 197 | - To include processes with single quotes, enclose them in double quotes: `"process with this ' single quote"`. |
| 198 | - To include processes with double quotes, enclose them in single quotes: `'process with this " double quote'`. |
| 199 | - The order of the entries in the configuration list is crucial. The first matching entry will be used, so it's important to follow a top-down hierarchy. Processes that don't match any entry will inherit the group from their parent processes. |
| 200 | |
| 201 | #### Windows default grouping behavior |
| 202 | |
| 203 | On Windows, when a process doesn't match any pattern in `apps_groups.conf`: |
| 204 | |
| 205 | - The **name** field (friendly name from file description or service display name) is used as the default group/category if available |
| 206 | - If no **name** field exists, the **comm** field is used |
| 207 | - This provides better default grouping for Windows services and applications with descriptive names |
| 208 | |
| 209 | For example, a process might have: |
| 210 | |
| 211 | - **comm**: `svchost` |
| 212 | - **name**: `Windows Update` |
| 213 | - **Default category**: `Windows Update` (uses the friendly name) |
| 214 | |
| 215 | ### Windows path handling |
| 216 | |
| 217 | When configuring `apps_groups.conf` on Windows systems: |
| 218 | |
| 219 | 1. **No backslash escaping needed** - Windows paths with backslashes are handled as literal strings: |
| 220 | ```text |
| 221 | sqlserver: "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Binn\sqlservr.exe" |
| 222 | ``` |
| 223 | |
| 224 | 2. **Use quotes for paths with spaces**: |
| 225 | ```text |
| 226 | office: "Microsoft Word" "Microsoft Excel" |
| 227 | browsers: chrome firefox msedge |
| 228 | ``` |
| 229 | |
| 230 | 3. **Prefer process names over full paths** - This is more portable and easier to maintain: |
| 231 | ```text |
| 232 | # Recommended - matches all SQL Server processes regardless of version/instance |
| 233 | sqlserver: sqlservr |
| 234 | |
| 235 | # Also works but less flexible |
| 236 | sqlserver: "C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Binn\sqlservr.exe" |
| 237 | ``` |
| 238 | |
| 239 | 4. **Use wildcards for flexible path matching**: |
| 240 | ```text |
| 241 | # Match anything from Program Files |
| 242 | programfiles: "*Program Files*" |
| 243 | |
| 244 | # Match SQL Server components across versions |
| 245 | mssql: "*\Microsoft SQL Server\*" |
| 246 | |
| 247 | # Match enterprise backup solutions |
| 248 | backup: "*\Veeam\*" "*\Veritas\*" "*\CommVault\*" |
| 249 | ``` |
| 250 | |
| 251 | ### Verifying your configuration |
| 252 | |
| 253 | You can use the Netdata `processes` function to verify that your `apps_groups.conf` configuration is working correctly: |
| 254 | |
| 255 | 1. **Access the processes function** through Netdata Cloud (required for security reasons) |
| 256 | 2. **Review the output** to see: |
| 257 | - Current running processes with their `comm`, `cmdline`, and (on Windows) `name` fields |
| 258 | - The **Category** column shows which group from `apps_groups.conf` each process has been assigned to |
| 259 | - Resource utilization for each process |
| 260 | |
| 261 | 3. **Troubleshooting tips**: |
| 262 | - If a process shows the wrong Category, check the exact process name in the function output |
| 263 | - On Windows, remember that the `name` field is used for default categories but NOT for pattern matching |
| 264 | - Remember that the first matching pattern wins - check your pattern order |
| 265 | - For inherited groups, verify the parent process has the correct Category |
| 266 | |
| 267 | There are a few command line options you can pass to `apps.plugin`. The list of available options can be acquired with the `--help` flag. |
| 268 | The options can be set in the `netdata.conf` using the [`edit-config` script](/docs/netdata-agent/configuration/README.md). |
| 269 | |
| 270 | For example, to disable user and user group charts you would set: |
| 271 | |
| 272 | ```text |
| 273 | [plugin:apps] |
| 274 | command options = without-users without-groups |
| 275 | ``` |
| 276 | |
| 277 | ### Memory Estimation with PSS Sampling |
| 278 | |
| 279 | On Linux systems with kernel 4.14 or later, `apps.plugin` uses Proportional Set Size (PSS) data from `/proc/<pid>/smaps_rollup` to provide more accurate memory usage estimates for processes that heavily use shared memory (e.g., databases, shared memory applications). |
| 280 | |
| 281 | **By default, PSS sampling is disabled**. When disabled, memory charts show traditional RSS (Resident Set Size), which may overstate usage for processes sharing memory pages. Enabling PSS sampling allows the plugin to periodically sample PSS values and use them to scale the shared portion of RSS, providing a significantly more accurate estimate without the overhead of reading smaps on every iteration. |
| 282 | |
| 283 | #### Configuration |
| 284 | |
| 285 | The `--pss` option controls PSS sampling behavior: |
| 286 | |
| 287 | ```text |
| 288 | [plugin:apps] |
| 289 | command options = --pss 5m |
| 290 | ``` |
| 291 | |
| 292 | **Valid values:** |
| 293 | |
| 294 | - Duration (e.g., `5m`, `300s`, `10m`): Sets the refresh interval for PSS sampling. Lower values provide more accurate estimates but increase CPU overhead. |
| 295 | - `off` or `0`: Completely disables PSS sampling. Memory charts will show traditional RSS-based measurements. |
| 296 | |
| 297 | **Default:** `off` |
| 298 | |
| 299 | **How it works:** |
| 300 | |
| 301 | - `apps.plugin` uses adaptive sampling that alternates between two strategies each iteration: |
| 302 | - **Delta-based**: Prioritizes processes with largest shared memory changes (refreshes big memory consumers within seconds) |
| 303 | - **Age-based**: Prioritizes processes with oldest samples (ensures all processes refreshed within 2× the interval) |
| 304 | - The sampled PSS/RSS ratio is cached and applied to subsequent RSS readings to estimate current memory usage |
| 305 | - This approach ensures rapid response to significant memory changes while guaranteeing bounded staleness for all processes |
| 306 | - When disabled (`--pss 0` or `--pss off`), no PSS sampling occurs and estimated memory charts are not shown |
| 307 | |
| 308 | **Performance considerations:** |
| 309 | |
| 310 | - Reading `/proc/<pid>/smaps_rollup` is more expensive than reading `/proc/<pid>/status` |
| 311 | - Shorter refresh periods provide more accurate estimates but increase CPU usage |
| 312 | - On systems with thousands of processes, consider increasing the refresh period (e.g., `10m` or `15m`) |
| 313 | - For systems without significant shared memory usage, disabling PSS sampling (`--pss off`) reduces overhead |
| 314 | |
| 315 | **Chart behavior:** |
| 316 | |
| 317 | - **When PSS is enabled:** Shows both "Estimated memory usage (RSS with shared scaling)" and "Memory RSS usage" charts |
| 318 | - **Default (PSS disabled):** Shows only "Memory RSS usage" charts |
| 319 | - The `processes` function API exposes additional columns (PSS, PssAge, SharedRatio) when PSS is enabled |
| 320 | |
| 321 | ### Integration with eBPF |
| 322 | |
| 323 | If you don't see charts under the **eBPF syscall** or **eBPF net** sections, you should edit your |
| 324 | [`ebpf.d.conf`](/src/collectors/ebpf.plugin/README.md#configure-the-ebpf-collector) file to ensure the eBPF program is enabled. |
| 325 | |
| 326 | Also see our [guide on troubleshooting apps with eBPF metrics](/docs/developer-and-contributor-corner/monitor-debug-applications-ebpf.md) for ideas on how to interpret these charts in a few scenarios. |
| 327 | |
| 328 | ## Permissions |
| 329 | |
| 330 | `apps.plugin` requires additional privileges to collect all the necessary information. |
| 331 | |
| 332 | During Netdata installation, `apps.plugin` is granted the `cap_dac_read_search` and `cap_sys_ptrace+ep` capabilities. |
| 333 | If this fails (i.e., `setcap` fails), `apps.plugin` is setuid to `root`. |
| 334 | |
| 335 | ## Security |
| 336 | |
| 337 | `apps.plugin` operates on a one-way communication model, sending metrics to Netdata without receiving instructions. This design minimizes potential security risks. |
| 338 | |
| 339 | Although `apps.plugin` can function without escalated privileges, it may not be able to collect all the necessary information. To ensure comprehensive data collection, it's recommended to grant the required privileges. |
| 340 | |
| 341 | The increased privileges are primarily used for building the process tree in memory, iterating over running processes, collecting metrics, and sending them to Netdata. This process does not involve any external communication or user interaction, further reducing security concerns. |