@cryptotaxi247 / netdata-1 / commits / 4491fda1e

Fix processes function: Add PPID grouping and fix WOps typo (#20902)

- Add PPID as a groupable field in the processes function Users can now group processes by their parent PID to see all child processes of each parent, providing a natural way to explore process hierarchies - Fix typo in IOCalls chart definition Changed 'WCalls' to 'WOps' to match the actual field name, fixing the I/O write operations display in the chart - Update documentation to reflect PPID is now groupable and correct OS-specific field availability based on code verification

Costa Tsaousis committed Aug 29, 2025 at 20:57 UTC 4491fda1e743b48aa1a2cd72255193c658592b12
2 files changed +237 -1
docs/functions/processes.md new
+223
@@ -0,0 +1,223 @@
1 +# Function: Top / Processes
2 +
3 +## Quick Info
4 +
5 +- **Plugin**: `apps.plugin`
6 +- **Type**: Simple Table (real-time snapshot)
7 +- **Availability**: Linux, Windows, FreeBSD, macOS
8 +- **Required Access**: View Agent Config for command line visibility
9 +
10 +### System Equivalents
11 +
12 +| Operating System | Traditional Tools |
13 +| ---------------- | ---------------------------------------------------- |
14 +| **Linux** | `top`, `htop`, `ps aux`, `pidstat` |
15 +| **Windows** | Task Manager, `tasklist`, `Get-Process` (PowerShell) |
16 +| **FreeBSD** | `top`, `ps aux`, `procstat` |
17 +| **macOS** | Activity Monitor, `top`, `ps aux` |
18 +
19 +The Netdata `processes` function provides several advantages over traditional tools:
20 +- More accurate resource accounting through child process accumulation (e.g., it can accurately provide the CPU utilization of shell scripts)
21 +- Unified cross-platform view with consistent metrics
22 +- Comprehensive I/O and file descriptor metrics per PID
23 +- Direct correlation with Netdata Apps dashboard section
24 +
25 +## Purpose
26 +
27 +The `processes` function is the drill-down companion to Apps (`apps.plugin`) charts, providing complete visibility into how system resources are broken down by individual processes and how they are aggregated into the categories shown in Netdata dashboards.
28 +
29 +`apps.plugin` intelligently groups processes into categories to avoid extreme cardinality issues (millions of potential PIDs). It identifies spawn managers (systemd, containerd, init, etc.) and groups process trees by their top-most parent - the direct children of these spawn managers. This creates a manageable set of categories with accumulated metrics from entire process trees, including exited children.
30 +
31 +When users see that "Application X" consumes significant resources in the charts, they need to understand:
32 +- Which specific processes are included in that category
33 +- How resources are distributed among those processes
34 +- Why certain processes are grouped together
35 +
36 +The `processes` function answers these questions by showing:
37 +- Every running PID with its assigned `Category` (matching the chart instances)
38 +- Complete resource breakdown per individual process
39 +- Accumulated metrics from exited children (unique capability)
40 +
41 +### Key Capabilities
42 +- **Accurate Resource Attribution**: More accurate than top/htop because it includes exited children and normalizes usage to match total system resources
43 +- **Process Tree Understanding**: Shows how processes are grouped into categories via the `Category` field
44 +- **Comprehensive Metrics**: Breaks down CPU (user/system), memory, I/O, file descriptors, threads, and more
45 +- **Leak Detection**: Identify memory leaks, file descriptor leaks, socket leaks, thread leaks
46 +- **Uptime Tracking**: Shows per-process uptime to spot restarts and long-running processes
47 +
48 +## Data Fields
49 +
50 +| Field | Type | Description | Filterable | Sortable | Groupable | OS Availability |
51 +| ------------------- | ---------- | --------------------------------------------------------------------- | ---------- | -------- | --------- | ---------------------- |
52 +| **PID** | Integer | Process ID | ✓ | ✓ | ✓ | All |
53 +| **Cmd** | String | Process command name | ✓ | ✓ | - | All |
54 +| **Name** | String | Process friendly name (if available) | ✓ | ✓ | - | Windows only |
55 +| **CmdLine** | String | Full command line with arguments (requires elevated access) | ✓ | ✓ | - | Linux, FreeBSD, macOS |
56 +| **PPID** | Integer | Parent process ID | ✓ | ✓ | ✓ | All |
57 +| **Category** | String | Process category from apps_groups.conf | ✓ | ✓ | ✓ | All |
58 +| **User** | String | User owner of the process | ✓ | ✓ | ✓ | All |
59 +| **Uid** | Integer | User ID | ✓ | ✓ | - | Linux, FreeBSD, macOS |
60 +| **Group** | String | Group owner | ✓ | ✓ | ✓ | Linux, FreeBSD, macOS |
61 +| **Gid** | Integer | Group ID | ✓ | ✓ | - | Linux, FreeBSD, macOS |
62 +| **CPU** | Percentage | Total CPU usage (100% = 1 core) | ✓ | ✓ | - | All |
63 +| **UserCPU** | Percentage | User-space CPU time | ✓ | ✓ | - | All |
64 +| **SysCPU** | Percentage | Kernel-space CPU time | ✓ | ✓ | - | All |
65 +| **GuestCPU** | Percentage | Guest VM CPU time (if available) | ✓ | ✓ | - | Linux only |
66 +| **CUserCPU** | Percentage | Children user CPU (accumulated from exited children) | ✓ | ✓ | - | Linux, FreeBSD |
67 +| **CSysCPU** | Percentage | Children system CPU (accumulated from exited children) | ✓ | ✓ | - | Linux, FreeBSD |
68 +| **CGuestCPU** | Percentage | Children guest CPU (accumulated from exited children) | ✓ | ✓ | - | Linux only |
69 +| **vCtxSwitch** | Rate | Voluntary context switches per second | ✓ | ✓ | - | Linux, macOS |
70 +| **iCtxSwitch** | Rate | Involuntary context switches per second | ✓ | ✓ | - | Linux only |
71 +| **Memory** | Percentage | Memory usage as percentage of total system RAM | ✓ | ✓ | - | All |
72 +| **Resident** | MiB | Resident Set Size (physical memory) | ✓ | ✓ | - | All |
73 +| **Shared** | MiB | Shared memory pages | ✓ | ✓ | - | Linux only |
74 +| **Virtual** | MiB | Virtual memory size | ✓ | ✓ | - | All |
75 +| **Swap** | MiB | Swap memory usage | ✓ | ✓ | - | Linux, Windows |
76 +| **PReads** | KiB/s | Physical disk read rate | ✓ | ✓ | - | Linux only |
77 +| **PWrites** | KiB/s | Physical disk write rate | ✓ | ✓ | - | Linux only |
78 +| **LReads** | KiB/s | Logical I/O read rate (includes cache) | ✓ | ✓ | - | All |
79 +| **LWrites** | KiB/s | Logical I/O write rate (includes cache) | ✓ | ✓ | - | All |
80 +| **ROps** | ops/s | Read operations per second | ✓ | ✓ | - | Linux, Windows |
81 +| **WOps** | ops/s | Write operations per second | ✓ | ✓ | - | Linux, Windows |
82 +| **MinFlt** | pgflts/s | Minor page faults per second | ✓ | ✓ | - | All |
83 +| **MajFlt** | pgflts/s | Major page faults per second | ✓ | ✓ | - | Linux, FreeBSD, macOS |
84 +| **CMinFlt** | pgflts/s | Children minor faults (accumulated) | ✓ | ✓ | - | Linux, FreeBSD |
85 +| **CMajFlt** | pgflts/s | Children major faults (accumulated) | ✓ | ✓ | - | Linux, FreeBSD |
86 +| **FDsLimitPercent** | Percentage | File descriptors usage vs limit | ✓ | ✓ | - | Linux only |
87 +| **FDs** | Count | Total open file descriptors | ✓ | ✓ | - | Linux, FreeBSD, macOS |
88 +| **Files** | Count | Open regular files | ✓ | ✓ | - | Linux, FreeBSD, macOS |
89 +| **Pipes** | Count | Open pipes | ✓ | ✓ | - | Linux, FreeBSD, macOS |
90 +| **Sockets** | Count | Open network sockets | ✓ | ✓ | - | Linux, FreeBSD, macOS |
91 +| **iNotiFDs** | Count | iNotify file descriptors | ✓ | ✓ | - | Linux only |
92 +| **EventFDs** | Count | Event file descriptors | ✓ | ✓ | - | Linux only |
93 +| **TimerFDs** | Count | Timer file descriptors | ✓ | ✓ | - | Linux only |
94 +| **SigFDs** | Count | Signal file descriptors | ✓ | ✓ | - | Linux only |
95 +| **EvPollFDs** | Count | Event poll descriptors | ✓ | ✓ | - | Linux only |
96 +| **OtherFDs** | Count | Other file descriptors | ✓ | ✓ | - | Linux, FreeBSD, macOS |
97 +| **Handles** | Count | Open handles (Windows compatibility) | ✓ | ✓ | - | Windows only |
98 +| **Processes** | Count | Number of processes (1 for single process, >1 for multi-process apps) | ✓ | ✓ | - | All |
99 +| **Threads** | Count | Number of threads | ✓ | ✓ | - | All |
100 +| **Uptime** | Seconds | Process uptime | ✓ | ✓ | - | All |
101 +
102 +### Platform-Specific Field Notes
103 +
104 +- **Linux**: The most comprehensive data with all metrics including physical I/O, detailed file descriptors, child process accumulation, and resource limits
105 +- **macOS**: Full process data except physical I/O, children accumulation, and some advanced metrics
106 +- **FreeBSD**: Similar to macOS but includes children CPU accumulation
107 +- **Windows**: Different approach using handles instead of file descriptors, includes I/O operations but lacks user/group ownership and command line access
108 +
109 +## Drill-Down Workflow
110 +
111 +The typical workflow for drilling down to individual processes looks like this:
112 +
113 +1. **Observe Chart Anomaly**: Notice high resource usage in an `apps.plugin` chart category (e.g., "web" consuming 80% CPU)
114 +2. **Launch Processes Function**: Open the function to see all processes
115 +3. **Filter by Category**: Use `category:web` filter to see only processes in that category
116 +4. **Identify Culprit**: Sort by the relevant metric (CPU, Memory, etc.) to find the specific process
117 +5. **Analyze Process Tree**: Use PPID relationships to understand process spawning patterns
118 +6. **Group Analysis**: Group by User, Command, or other fields to understand patterns
119 +
120 +## Use Cases
121 +
122 +### 1. Break Down System Resources into Processes
123 +
124 +The processes function provides complete visibility into how system resources are distributed across all running processes, enabling comprehensive resource accounting and analysis.
125 +
126 +#### View resource distribution across all processes
127 +Sort by `CPU`, `Memory`, or `I/O` metrics descending to see which processes consume the most resources. Group by `Category` to understand resource allocation across application groups. This provides a complete breakdown of system resource utilization at the process level.
128 +
129 +#### Understand category composition and aggregation
130 +Filter by `category:[name]` to see all processes that contribute to a specific apps.plugin chart instance. Group by `Cmd` within a category to understand which different executables are grouped together. This reveals exactly how Netdata's intelligent grouping works and what's included in each category.
131 +
132 +#### Analyze resource usage by user or group
133 +Group processes by `User` or `Group` to understand resource consumption patterns across different users and system accounts. Sort by aggregate CPU or memory within each group to identify which users are consuming the most resources. This helps with multi-tenant resource accounting and fair-share analysis.
134 +
135 +### 2. Drill Down to Identify Specific Heavy Consumer Processes
136 +
137 +When apps.plugin charts show high resource usage in a category, the processes function enables precise identification of the specific processes responsible.
138 +
139 +#### Identify CPU-intensive processes within categories
140 +Filter by `category:[name]` and sort by `CPU` descending to find the exact processes causing high CPU usage in a chart category. Look at both own CPU (`UserCPU`, `SysCPU`) and children CPU (`CUserCPU`, `CSysCPU`) to understand whether the load comes from the process itself or its children.
141 +
142 +#### Find memory-consuming processes in application groups
143 +Filter by specific categories and sort by `Resident` or `Memory` percentage to identify which processes within an application group consume the most RAM. Compare `Virtual` vs `Resident` to understand memory allocation patterns and potential over-provisioning.
144 +
145 +#### Locate I/O-heavy processes causing disk bottlenecks
146 +Sort by `PReads + PWrites` for physical I/O or `LReads + LWrites` for logical I/O to find processes generating the most disk activity. Filter by category to drill down from chart-level I/O metrics to specific process-level I/O patterns.
147 +
148 +### 3. Detect Leaks of Multiple Kinds
149 +
150 +The processes function excels at identifying various types of resource leaks by correlating resource usage with process uptime.
151 +
152 +#### Memory leak detection in long-running processes
153 +Filter processes with `Uptime > 3600` (one hour) and sort by `Resident` memory descending. Look for processes where memory consumption is disproportionately high relative to their uptime. Track specific PIDs over time to observe continuously growing memory usage patterns.
154 +
155 +#### File descriptor leak identification
156 +Sort by `FDs` count or filter for `FDsLimitPercent > 50` to find processes approaching their file descriptor limits. Examine the breakdown of descriptor types (`Files`, `Sockets`, `Pipes`, etc.) to understand what type of resources are leaking. Correlate high FD counts with process uptime to identify gradual leaks.
157 +
158 +#### Socket and network connection leaks
159 +Sort by `Sockets` count to identify processes with abnormally high network connections. Compare socket counts against expected application behavior and uptime to detect connection leaks. Group by `Category` to see if entire application groups are affected by socket exhaustion.
160 +
161 +#### Thread leak monitoring
162 +Sort by `Threads` count and correlate with `Uptime` to find processes creating threads without proper cleanup. Look for processes where thread count grows continuously over time. Filter by category to identify applications with thread pool management issues.
163 +
164 +### 4. Monitor Crashes or Abnormal Events via Uptime
165 +
166 +Process uptime tracking enables detection of crashes, restarts, and abnormal process lifecycle events.
167 +
168 +#### Detect recent process restarts and crashes
169 +Sort by `Uptime` ascending to immediately see which processes have recently started or restarted. Filter by specific categories or command names to monitor critical services for unexpected restarts. Compare process start times with known maintenance windows to identify unplanned restarts.
170 +
171 +#### Identify unstable applications with frequent restarts
172 +Group processes by `Cmd` and look for multiple PIDs with similar names but different uptimes, indicating repeated restarts. Track specific application categories over time to identify patterns of instability. Correlate low uptimes with high child CPU accumulation to detect crash loops.
173 +
174 +#### Monitor process lifecycle and stability patterns
175 +Filter by category and examine uptime distribution to understand application stability. Look for processes that should be long-running but have short uptimes. Use PPID relationships to identify parent processes that frequently spawn short-lived children.
176 +
177 +### 5. Security Monitoring
178 +
179 +The processes function provides critical security visibility by exposing process ownership, privileges, and behavior patterns.
180 +
181 +#### Detect unauthorized or suspicious processes
182 +Filter by `Category:other` to find uncategorized processes that may be suspicious. Sort by `User` to identify processes running under unexpected accounts. Search for unusual command names or paths that don't match normal system behavior.
183 +
184 +#### Monitor privilege escalation and root processes
185 +Filter by `Uid:0` or `User:root` to track all processes running with root privileges. Group root processes by `Cmd` to understand what's running with elevated permissions. Look for unexpected processes running as root that shouldn't require privileges.
186 +
187 +#### Analyze network activity and connection patterns
188 +Sort by `Sockets` count to identify processes with unusual network activity. Filter by specific users or categories to detect abnormal network behavior patterns. Correlate high socket counts with process names to identify potential backdoors or data exfiltration.
189 +
190 +#### Track command line arguments for security forensics
191 +Use full-text search in `CmdLine` to find processes launched with specific parameters or scripts. Group by command line patterns to identify potentially malicious execution patterns. Filter by user and examine command lines to detect privilege abuse or policy violations.
192 +
193 +## Special Features
194 +
195 +- **Child Process Accumulation**: Uniquely captures resources from exited children - critical for accurate measurement of shell scripts and applications that spawn many short-lived processes (even 100+ commands/second)
196 +- **Category Correlation**: The `Category` field directly matches the instance names in `apps.plugin` charts, enabling drill-down from chart to process level
197 +- **Intelligent Grouping**: Understands spawn managers (systemd, containerd, init) and groups by top-most parent to create manageable categories
198 +- **Normalized Metrics**: All per-process usage is normalized to accurately match total system resource usage
199 +- **Real-time Updates**: Data refreshes every few seconds showing current process state
200 +- **Custom Grouping**: `apps_groups.conf` allows defining custom spawn managers and individual processes of interest
201 +- **Comprehensive FD Breakdown**: Detailed categorization of all file descriptor types for leak detection
202 +
203 +## Performance Considerations
204 +
205 +- Function executes with minimal overhead using efficient process enumeration
206 +- Large process counts (>1000) may increase response time
207 +- Command line access requires additional security permissions
208 +- No historical data - shows current snapshot only
209 +
210 +## Requirements and Limitations
211 +
212 +- **Operating System**: Linux, Windows, FreeBSD, macOS
213 +- **Permissions**: Standard user can see basic data; elevated access needed for command lines
214 +- **Data Type**: Real-time snapshot (no historical data)
215 +- **Child Processes**: Only terminated children are accumulated; running children appear separately
216 +- **Platform Variations**: Some fields may not be available on all operating systems (e.g., certain I/O metrics on macOS)
217 +
218 +## Related Functions
219 +
220 +- `systemd-services`: Aggregated view of processes grouped by systemd service
221 +- `containers-vms`: Container and VM-specific process information
222 +- `network-connections`: Network connections per process
223 +- `systemd-journal`: Process logs and events
\ No newline at end of file
src/collectors/apps.plugin/apps_functions.c
+14 -1
@@ -977,7 +977,7 @@ void function_processes(const char *transaction, char *function,
977 buffer_json_member_add_array(wb, "columns");
978 {
979 buffer_json_add_array_item_string(wb, "ROps");
980 - buffer_json_add_array_item_string(wb, "WCalls");
980 + buffer_json_add_array_item_string(wb, "WOps");
981 }
982 buffer_json_array_close(wb);
983 }
@@ -1102,6 +1102,19 @@ void function_processes(const char *transaction, char *function,
1102 }
1103 buffer_json_object_close(wb);
1104
1105 + // group by PPID
1106 + buffer_json_member_add_object(wb, "PPID");
1107 + {
1108 + buffer_json_member_add_string(wb, "name", "Process Tree by Parent PID");
1109 + buffer_json_member_add_array(wb, "columns");
1110 + {
1111 + buffer_json_add_array_item_string(wb, "PPID");
1112 + buffer_json_add_array_item_string(wb, "PID");
1113 + }
1114 + buffer_json_array_close(wb);
1115 + }
1116 + buffer_json_object_close(wb);
1117 +
1118 #if (PROCESSES_HAVE_UID == 1) || (PROCESSES_HAVE_SID == 1)
1119 // group by User
1120 buffer_json_member_add_object(wb, "User");