| 1 | # Function: Top / Processes |
| 2 | |
| 3 | ## Quick Info |
| 4 | |
| 5 | - **Plugin**: `apps.plugin` |
| 6 | - **Type**: Simple Table (real-time snapshot) |
| 7 | - **Availability**: Linux, Windows, FreeBSD, macOS |
| 8 | - **Required Access**: View Agent Config for command line visibility |
| 9 | |
| 10 | ### System Equivalents |
| 11 | |
| 12 | | Operating System | Traditional Tools | |
| 13 | | ---------------- | ---------------------------------------------------- | |
| 14 | | **Linux** | `top`, `htop`, `ps aux`, `pidstat` | |
| 15 | | **Windows** | Task Manager, `tasklist`, `Get-Process` (PowerShell) | |
| 16 | | **FreeBSD** | `top`, `ps aux`, `procstat` | |
| 17 | | **macOS** | Activity Monitor, `top`, `ps aux` | |
| 18 | |
| 19 | The Netdata `processes` function provides several advantages over traditional tools: |
| 20 | - More accurate resource accounting through child process accumulation (e.g., it can accurately provide the CPU utilization of shell scripts) |
| 21 | - Unified cross-platform view with consistent metrics |
| 22 | - Comprehensive I/O and file descriptor metrics per PID |
| 23 | - Direct correlation with Netdata Apps dashboard section |
| 24 | |
| 25 | ## Purpose |
| 26 | |
| 27 | The `processes` function is the drill-down companion to Apps (`apps.plugin`) charts, providing complete visibility into how system resources are broken down by individual processes and how they are aggregated into the categories shown in Netdata dashboards. |
| 28 | |
| 29 | `apps.plugin` intelligently groups processes into categories to avoid extreme cardinality issues (millions of potential PIDs). It identifies spawn managers (systemd, containerd, init, etc.) and groups process trees by their top-most parent - the direct children of these spawn managers. This creates a manageable set of categories with accumulated metrics from entire process trees, including exited children. |
| 30 | |
| 31 | When users see that "Application X" consumes significant resources in the charts, they need to understand: |
| 32 | - Which specific processes are included in that category |
| 33 | - How resources are distributed among those processes |
| 34 | - Why certain processes are grouped together |
| 35 | |
| 36 | The `processes` function answers these questions by showing: |
| 37 | - Every running PID with its assigned `Category` (matching the chart instances) |
| 38 | - Complete resource breakdown per individual process |
| 39 | - Accumulated metrics from exited children (unique capability) |
| 40 | |
| 41 | ### Key Capabilities |
| 42 | - **Accurate Resource Attribution**: More accurate than top/htop because it includes exited children and normalizes usage to match total system resources |
| 43 | - **Process Tree Understanding**: Shows how processes are grouped into categories via the `Category` field |
| 44 | - **Comprehensive Metrics**: Breaks down CPU (user/system), memory, I/O, file descriptors, threads, and more |
| 45 | - **Leak Detection**: Identify memory leaks, file descriptor leaks, socket leaks, thread leaks |
| 46 | - **Uptime Tracking**: Shows per-process uptime to spot restarts and long-running processes |
| 47 | |
| 48 | ## Data Fields |
| 49 | |
| 50 | | Field | Type | Description | Filterable | Sortable | Groupable | OS Availability | |
| 51 | | ------------------- | ---------- | --------------------------------------------------------------------- | ---------- | -------- | --------- | ---------------------- | |
| 52 | | **PID** | Integer | Process ID | ✓ | ✓ | ✓ | All | |
| 53 | | **Cmd** | String | Process command name | ✓ | ✓ | - | All | |
| 54 | | **Name** | String | Process friendly name (if available) | ✓ | ✓ | - | Windows only | |
| 55 | | **CmdLine** | String | Full command line with arguments (requires elevated access) | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 56 | | **PPID** | Integer | Parent process ID | ✓ | ✓ | ✓ | All | |
| 57 | | **Category** | String | Process category from apps_groups.conf | ✓ | ✓ | ✓ | All | |
| 58 | | **User** | String | User owner of the process | ✓ | ✓ | ✓ | All | |
| 59 | | **Uid** | Integer | User ID | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 60 | | **Group** | String | Group owner | ✓ | ✓ | ✓ | Linux, FreeBSD, macOS | |
| 61 | | **Gid** | Integer | Group ID | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 62 | | **CPU** | Percentage | Total CPU usage (100% = 1 core) | ✓ | ✓ | - | All | |
| 63 | | **UserCPU** | Percentage | User-space CPU time | ✓ | ✓ | - | All | |
| 64 | | **SysCPU** | Percentage | Kernel-space CPU time | ✓ | ✓ | - | All | |
| 65 | | **GuestCPU** | Percentage | Guest VM CPU time (if available) | ✓ | ✓ | - | Linux only | |
| 66 | | **CUserCPU** | Percentage | Children user CPU (accumulated from exited children) | ✓ | ✓ | - | Linux, FreeBSD | |
| 67 | | **CSysCPU** | Percentage | Children system CPU (accumulated from exited children) | ✓ | ✓ | - | Linux, FreeBSD | |
| 68 | | **CGuestCPU** | Percentage | Children guest CPU (accumulated from exited children) | ✓ | ✓ | - | Linux only | |
| 69 | | **vCtxSwitch** | Rate | Voluntary context switches per second | ✓ | ✓ | - | Linux, macOS | |
| 70 | | **iCtxSwitch** | Rate | Involuntary context switches per second | ✓ | ✓ | - | Linux only | |
| 71 | | **Memory** | Percentage | Memory usage as percentage of total system RAM | ✓ | ✓ | - | All | |
| 72 | | **Resident** | MiB | Resident Set Size (physical memory) | ✓ | ✓ | - | All | |
| 73 | | **Estimated** | MiB | Estimated memory using PSS scaling (visible by default when enabled) | ✓ | ✓ | - | Linux 4.14+ (with PSS) | |
| 74 | | **Pss** | MiB | Proportional Set Size (hidden by default) | ✓ | ✓ | - | Linux 4.14+ (with PSS) | |
| 75 | | **PssAge** | Seconds | Time since last smaps sample (hidden by default) | ✓ | ✓ | - | Linux 4.14+ (with PSS) | |
| 76 | | **SharedRatio** | Percentage | Shared memory ratio from PSS (hidden by default) | ✓ | ✓ | - | Linux 4.14+ (with PSS) | |
| 77 | | **Shared** | MiB | Shared memory pages | ✓ | ✓ | - | Linux only | |
| 78 | | **Virtual** | MiB | Virtual memory size | ✓ | ✓ | - | All | |
| 79 | | **Swap** | MiB | Swap memory usage | ✓ | ✓ | - | Linux, Windows | |
| 80 | | **PReads** | KiB/s | Physical disk read rate | ✓ | ✓ | - | Linux only | |
| 81 | | **PWrites** | KiB/s | Physical disk write rate | ✓ | ✓ | - | Linux only | |
| 82 | | **LReads** | KiB/s | Logical I/O read rate (includes cache) | ✓ | ✓ | - | All | |
| 83 | | **LWrites** | KiB/s | Logical I/O write rate (includes cache) | ✓ | ✓ | - | All | |
| 84 | | **ROps** | ops/s | Read operations per second | ✓ | ✓ | - | Linux, Windows | |
| 85 | | **WOps** | ops/s | Write operations per second | ✓ | ✓ | - | Linux, Windows | |
| 86 | | **MinFlt** | pgflts/s | Minor page faults per second | ✓ | ✓ | - | All | |
| 87 | | **MajFlt** | pgflts/s | Major page faults per second | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 88 | | **CMinFlt** | pgflts/s | Children minor faults (accumulated) | ✓ | ✓ | - | Linux, FreeBSD | |
| 89 | | **CMajFlt** | pgflts/s | Children major faults (accumulated) | ✓ | ✓ | - | Linux, FreeBSD | |
| 90 | | **FDsLimitPercent** | Percentage | File descriptors usage vs limit | ✓ | ✓ | - | Linux only | |
| 91 | | **FDs** | Count | Total open file descriptors | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 92 | | **Files** | Count | Open regular files | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 93 | | **Pipes** | Count | Open pipes | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 94 | | **Sockets** | Count | Open network sockets | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 95 | | **iNotiFDs** | Count | iNotify file descriptors | ✓ | ✓ | - | Linux only | |
| 96 | | **EventFDs** | Count | Event file descriptors | ✓ | ✓ | - | Linux only | |
| 97 | | **TimerFDs** | Count | Timer file descriptors | ✓ | ✓ | - | Linux only | |
| 98 | | **SigFDs** | Count | Signal file descriptors | ✓ | ✓ | - | Linux only | |
| 99 | | **EvPollFDs** | Count | Event poll descriptors | ✓ | ✓ | - | Linux only | |
| 100 | | **OtherFDs** | Count | Other file descriptors | ✓ | ✓ | - | Linux, FreeBSD, macOS | |
| 101 | | **Handles** | Count | Open handles (Windows compatibility) | ✓ | ✓ | - | Windows only | |
| 102 | | **Processes** | Count | Number of processes (1 for single process, >1 for multi-process apps) | ✓ | ✓ | - | All | |
| 103 | | **Threads** | Count | Number of threads | ✓ | ✓ | - | All | |
| 104 | | **Uptime** | Seconds | Process uptime | ✓ | ✓ | - | All | |
| 105 | |
| 106 | ### Platform-Specific Field Notes |
| 107 | |
| 108 | - **Linux**: The most comprehensive data with all metrics including physical I/O, detailed file descriptors, child process accumulation, and resource limits |
| 109 | - **PSS Memory Estimation** (kernel 4.14+): When enabled (default), provides `Estimated`, `Pss`, `PssAge`, and `SharedRatio` fields for more accurate memory accounting in shared-memory workloads. The plugin uses adaptive sampling that prioritizes the largest memory consumers and processes with significant memory changes, refreshing them within seconds of detection. All processes are guaranteed to be refreshed within 2× the configured PSS refresh period (default: 600 seconds). Disable with `--pss 0` to remove these fields and use traditional RSS measurements. |
| 110 | - **macOS**: Full process data except physical I/O, children accumulation, and some advanced metrics |
| 111 | - **FreeBSD**: Similar to macOS but includes children CPU accumulation |
| 112 | - **Windows**: Different approach using handles instead of file descriptors, includes I/O operations but lacks user/group ownership and command line access |
| 113 | |
| 114 | ## Drill-Down Workflow |
| 115 | |
| 116 | The typical workflow for drilling down to individual processes looks like this: |
| 117 | |
| 118 | 1. **Observe Chart Anomaly**: Notice high resource usage in an `apps.plugin` chart category (e.g., "web" consuming 80% CPU) |
| 119 | 2. **Launch Processes Function**: Open the function to see all processes |
| 120 | 3. **Filter by Category**: Use `category:web` filter to see only processes in that category |
| 121 | 4. **Identify Culprit**: Sort by the relevant metric (CPU, Memory, etc.) to find the specific process |
| 122 | 5. **Analyze Process Tree**: Use PPID relationships to understand process spawning patterns |
| 123 | 6. **Group Analysis**: Group by User, Command, or other fields to understand patterns |
| 124 | |
| 125 | ## Use Cases |
| 126 | |
| 127 | ### 1. Break Down System Resources into Processes |
| 128 | |
| 129 | The processes function provides complete visibility into how system resources are distributed across all running processes, enabling comprehensive resource accounting and analysis. |
| 130 | |
| 131 | #### View resource distribution across all processes |
| 132 | Sort by `CPU`, `Memory`, or `I/O` metrics descending to see which processes consume the most resources. Group by `Category` to understand resource allocation across application groups. This provides a complete breakdown of system resource utilization at the process level. |
| 133 | |
| 134 | #### Understand category composition and aggregation |
| 135 | Filter by `category:[name]` to see all processes that contribute to a specific apps.plugin chart instance. Group by `Cmd` within a category to understand which different executables are grouped together. This reveals exactly how Netdata's intelligent grouping works and what's included in each category. |
| 136 | |
| 137 | #### Analyze resource usage by user or group |
| 138 | Group processes by `User` or `Group` to understand resource consumption patterns across different users and system accounts. Sort by aggregate CPU or memory within each group to identify which users are consuming the most resources. This helps with multi-tenant resource accounting and fair-share analysis. |
| 139 | |
| 140 | ### 2. Drill Down to Identify Specific Heavy Consumer Processes |
| 141 | |
| 142 | When apps.plugin charts show high resource usage in a category, the processes function enables precise identification of the specific processes responsible. |
| 143 | |
| 144 | #### Identify CPU-intensive processes within categories |
| 145 | Filter by `category:[name]` and sort by `CPU` descending to find the exact processes causing high CPU usage in a chart category. Look at both own CPU (`UserCPU`, `SysCPU`) and children CPU (`CUserCPU`, `CSysCPU`) to understand whether the load comes from the process itself or its children. |
| 146 | |
| 147 | #### Find memory-consuming processes in application groups |
| 148 | Filter by specific categories and sort by `Resident` or `Memory` percentage to identify which processes within an application group consume the most RAM. Compare `Virtual` vs `Resident` to understand memory allocation patterns and potential over-provisioning. |
| 149 | |
| 150 | On Linux 4.14+ with PSS enabled (default), use `Estimated` instead of `Resident` for more accurate memory accounting in shared-memory workloads (databases, cache servers, etc.). The `Estimated` field scales shared memory using PSS ratios to show true proportional memory usage. Check `SharedRatio` to see the scaling factor - values significantly below 100% indicate heavy shared memory usage where `Resident` would overstate consumption. The `PssAge` field shows seconds since the last PSS sample - expect low values (under 10s) for large memory consumers due to adaptive prioritization, while smaller processes may show higher ages (up to 600s by default) as they are refreshed less frequently. |
| 151 | |
| 152 | #### Locate I/O-heavy processes causing disk bottlenecks |
| 153 | Sort by `PReads + PWrites` for physical I/O or `LReads + LWrites` for logical I/O to find processes generating the most disk activity. Filter by category to drill down from chart-level I/O metrics to specific process-level I/O patterns. |
| 154 | |
| 155 | ### 3. Detect Leaks of Multiple Kinds |
| 156 | |
| 157 | The processes function excels at identifying various types of resource leaks by correlating resource usage with process uptime. |
| 158 | |
| 159 | #### Memory leak detection in long-running processes |
| 160 | Filter processes with `Uptime > 3600` (one hour) and sort by `Resident` (or `Estimated` on Linux with PSS enabled) memory descending. Look for processes where memory consumption is disproportionately high relative to their uptime. Track specific PIDs over time to observe continuously growing memory usage patterns. On shared-memory workloads, use `Estimated` to avoid false positives from shared pages that aren't actually leaking. Note that PSS samples for large memory consumers are refreshed within seconds, providing near real-time leak detection. |
| 161 | |
| 162 | #### File descriptor leak identification |
| 163 | Sort by `FDs` count or filter for `FDsLimitPercent > 50` to find processes approaching their file descriptor limits. Examine the breakdown of descriptor types (`Files`, `Sockets`, `Pipes`, etc.) to understand what type of resources are leaking. Correlate high FD counts with process uptime to identify gradual leaks. |
| 164 | |
| 165 | #### Socket and network connection leaks |
| 166 | Sort by `Sockets` count to identify processes with abnormally high network connections. Compare socket counts against expected application behavior and uptime to detect connection leaks. Group by `Category` to see if entire application groups are affected by socket exhaustion. |
| 167 | |
| 168 | #### Thread leak monitoring |
| 169 | Sort by `Threads` count and correlate with `Uptime` to find processes creating threads without proper cleanup. Look for processes where thread count grows continuously over time. Filter by category to identify applications with thread pool management issues. |
| 170 | |
| 171 | ### 4. Monitor Crashes or Abnormal Events via Uptime |
| 172 | |
| 173 | Process uptime tracking enables detection of crashes, restarts, and abnormal process lifecycle events. |
| 174 | |
| 175 | #### Detect recent process restarts and crashes |
| 176 | Sort by `Uptime` ascending to immediately see which processes have recently started or restarted. Filter by specific categories or command names to monitor critical services for unexpected restarts. Compare process start times with known maintenance windows to identify unplanned restarts. |
| 177 | |
| 178 | #### Identify unstable applications with frequent restarts |
| 179 | Group processes by `Cmd` and look for multiple PIDs with similar names but different uptimes, indicating repeated restarts. Track specific application categories over time to identify patterns of instability. Correlate low uptimes with high child CPU accumulation to detect crash loops. |
| 180 | |
| 181 | #### Monitor process lifecycle and stability patterns |
| 182 | Filter by category and examine uptime distribution to understand application stability. Look for processes that should be long-running but have short uptimes. Use PPID relationships to identify parent processes that frequently spawn short-lived children. |
| 183 | |
| 184 | ### 5. Security Monitoring |
| 185 | |
| 186 | The processes function provides critical security visibility by exposing process ownership, privileges, and behavior patterns. |
| 187 | |
| 188 | #### Detect unauthorized or suspicious processes |
| 189 | Filter by `Category:other` to find uncategorized processes that may be suspicious. Sort by `User` to identify processes running under unexpected accounts. Search for unusual command names or paths that don't match normal system behavior. |
| 190 | |
| 191 | #### Monitor privilege escalation and root processes |
| 192 | Filter by `Uid:0` or `User:root` to track all processes running with root privileges. Group root processes by `Cmd` to understand what's running with elevated permissions. Look for unexpected processes running as root that shouldn't require privileges. |
| 193 | |
| 194 | #### Analyze network activity and connection patterns |
| 195 | Sort by `Sockets` count to identify processes with unusual network activity. Filter by specific users or categories to detect abnormal network behavior patterns. Correlate high socket counts with process names to identify potential backdoors or data exfiltration. |
| 196 | |
| 197 | #### Track command line arguments for security forensics |
| 198 | Use full-text search in `CmdLine` to find processes launched with specific parameters or scripts. Group by command line patterns to identify potentially malicious execution patterns. Filter by user and examine command lines to detect privilege abuse or policy violations. |
| 199 | |
| 200 | ## Special Features |
| 201 | |
| 202 | - **Child Process Accumulation**: Uniquely captures resources from exited children - critical for accurate measurement of shell scripts and applications that spawn many short-lived processes (even 100+ commands/second) |
| 203 | - **PSS Memory Estimation** (Linux 4.14+): Provides accurate memory accounting for shared-memory workloads by using Proportional Set Size (PSS) to scale shared pages. Enabled by default with adaptive sampling to minimize overhead while ensuring rapid response to memory changes. The plugin alternates between two prioritization strategies each iteration: |
| 204 | - **Delta-based strategy**: Prioritizes processes with the largest memory changes, ensuring rapid detection and response to memory growth (typically within seconds) |
| 205 | - **Age-based strategy**: Prioritizes processes that haven't been updated longest, ensuring eventual consistency for all processes |
| 206 | |
| 207 | Both strategies sort candidates by priority and refresh the top N processes within the configured budget. This approach ensures that the biggest memory consumers (databases, cache servers, etc.) are refreshed within seconds of significant changes, while guaranteeing that even the smallest processes are refreshed within 2× the configured PSS refresh period (default: 600 seconds). Shows true memory consumption vs inflated RSS values for shared-memory workloads. |
| 208 | - **Category Correlation**: The `Category` field directly matches the instance names in `apps.plugin` charts, enabling drill-down from chart to process level |
| 209 | - **Intelligent Grouping**: Understands spawn managers (systemd, containerd, init) and groups by top-most parent to create manageable categories |
| 210 | - **Normalized Metrics**: All per-process usage is normalized to accurately match total system resource usage |
| 211 | - **Real-time Updates**: Data refreshes every few seconds showing current process state |
| 212 | - **Custom Grouping**: `apps_groups.conf` allows defining custom spawn managers and individual processes of interest |
| 213 | - **Comprehensive FD Breakdown**: Detailed categorization of all file descriptor types for leak detection |
| 214 | |
| 215 | ## Performance Considerations |
| 216 | |
| 217 | - Function executes with minimal overhead using efficient process enumeration |
| 218 | - Large process counts (>1000) may increase response time |
| 219 | - Command line access requires additional security permissions |
| 220 | - No historical data - shows current snapshot only |
| 221 | |
| 222 | ## Requirements and Limitations |
| 223 | |
| 224 | - **Operating System**: Linux, Windows, FreeBSD, macOS |
| 225 | - **Permissions**: Standard user can see basic data; elevated access needed for command lines |
| 226 | - **Data Type**: Real-time snapshot (no historical data) |
| 227 | - **Child Processes**: Only terminated children are accumulated; running children appear separately |
| 228 | - **Platform Variations**: Some fields may not be available on all operating systems (e.g., certain I/O metrics on macOS) |
| 229 | |
| 230 | ## Related Functions |
| 231 | |
| 232 | - `systemd-services`: Aggregated view of processes grouped by systemd service |
| 233 | - `containers-vms`: Container and VM-specific process information |
| 234 | - `network-connections`: Network connections per process |
| 235 | - `systemd-journal`: Process logs and events |