| 1 | <!--startmeta |
| 2 | custom_edit_url: "https://github.com/netdata/netdata/edit/master/src/collectors/proc.plugin/integrations/system_statistics.md" |
| 3 | meta_yaml: "https://github.com/netdata/netdata/edit/master/src/collectors/proc.plugin/metadata.yaml" |
| 4 | sidebar_label: "System statistics" |
| 5 | learn_status: "Published" |
| 6 | learn_rel_path: "Collecting Metrics/Collectors/Operating Systems" |
| 7 | keywords: ['cpu utilization', 'process counts'] |
| 8 | message: "DO NOT EDIT THIS FILE DIRECTLY, IT IS GENERATED BY THE COLLECTOR'S metadata.yaml FILE" |
| 9 | endmeta--> |
| 10 | |
| 11 | # System statistics |
| 12 | |
| 13 | |
| 14 | <img src="https://netdata.cloud/img/linuxserver.svg" width="150"/> |
| 15 | |
| 16 | |
| 17 | Plugin: proc.plugin |
| 18 | Module: /proc/stat |
| 19 | |
| 20 | <img src="https://img.shields.io/badge/maintained%20by-Netdata-%2300ab44" /> |
| 21 | |
| 22 | ## Overview |
| 23 | |
| 24 | CPU utilization, states and frequencies and key Linux system performance metrics. |
| 25 | |
| 26 | The `/proc/stat` file provides various types of system statistics: |
| 27 | |
| 28 | - The overall system CPU usage statistics |
| 29 | - Per CPU core statistics |
| 30 | - The total context switching of the system |
| 31 | - The total number of processes running |
| 32 | - The total CPU interrupts |
| 33 | - The total CPU softirqs |
| 34 | |
| 35 | The collector also reads: |
| 36 | |
| 37 | - `/proc/schedstat` for statistics about the process scheduler in the Linux kernel. |
| 38 | - `/sys/devices/system/cpu/[X]/thermal_throttle/core_throttle_count` to get the count of thermal throttling events for a specific CPU core on Linux systems. |
| 39 | - `/sys/devices/system/cpu/[X]/thermal_throttle/package_throttle_count` to get the count of thermal throttling events for a specific CPU package on a Linux system. |
| 40 | - `/sys/devices/system/cpu/[X]/cpufreq/scaling_cur_freq` to get the current operating frequency of a specific CPU core. |
| 41 | - `/sys/devices/system/cpu/[X]/cpufreq/stats/time_in_state` to get the amount of time the CPU has spent in each of its available frequency states. |
| 42 | - `/sys/devices/system/cpu/[X]/cpuidle/state[X]/name` to get the names of the idle states for each CPU core in a Linux system. |
| 43 | - `/sys/devices/system/cpu/[X]/cpuidle/state[X]/time` to get the total time each specific CPU core has spent in each idle state since the system was started. |
| 44 | |
| 45 | |
| 46 | |
| 47 | |
| 48 | This collector is only supported on the following platforms: |
| 49 | |
| 50 | - linux |
| 51 | |
| 52 | This collector only supports collecting metrics from a single instance of this integration. |
| 53 | |
| 54 | |
| 55 | ### Default Behavior |
| 56 | |
| 57 | #### Auto-Detection |
| 58 | |
| 59 | The collector auto-detects all metrics. No configuration is needed. |
| 60 | |
| 61 | |
| 62 | #### Limits |
| 63 | |
| 64 | The default configuration for this integration does not impose any limits on data collection. |
| 65 | |
| 66 | #### Performance Impact |
| 67 | |
| 68 | The collector disables cpu frequency and idle state monitoring when there are more than 128 CPU cores available. |
| 69 | |
| 70 | |
| 71 | ## Setup |
| 72 | |
| 73 | |
| 74 | ### Prerequisites |
| 75 | |
| 76 | No action required. |
| 77 | |
| 78 | ### Configuration |
| 79 | |
| 80 | #### Options |
| 81 | |
| 82 | |
| 83 | |
| 84 | <details open><summary>Config options</summary> |
| 85 | |
| 86 | |
| 87 | |
| 88 | | Option | Description | Default | Required | |
| 89 | |:-----|:------------|:--------|:---------:| |
| 90 | | per cpu core utilization | Collects CPU usage metrics for each individual core, in addition to the system-wide averages. | no | no | |
| 91 | | cpu idle states | Collects CPU idle state residency metrics for each individual core, showing how much time each core spends in different idle states (C-states). | no | no | |
| 92 | |
| 93 | |
| 94 | </details> |
| 95 | |
| 96 | |
| 97 | |
| 98 | #### via File |
| 99 | |
| 100 | The configuration file name for this integration is `netdata.conf`. |
| 101 | Configuration for this specific integration is located in the `plugin:proc:/proc/stat` section within that file. |
| 102 | |
| 103 | The file format is a modified INI syntax. The general structure is: |
| 104 | |
| 105 | ```ini |
| 106 | [section1] |
| 107 | option1 = some value |
| 108 | option2 = some other value |
| 109 | |
| 110 | [section2] |
| 111 | option3 = some third value |
| 112 | ``` |
| 113 | You can edit the configuration file using the [`edit-config`](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#edit-configuration-files) script from the |
| 114 | Netdata [config directory](https://github.com/netdata/netdata/blob/master/docs/netdata-agent/configuration/README.md#locate-your-config-directory). |
| 115 | |
| 116 | ```bash |
| 117 | cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata |
| 118 | sudo ./edit-config netdata.conf |
| 119 | ``` |
| 120 | |
| 121 | ##### Examples |
| 122 | There are no configuration examples. |
| 123 | |
| 124 | |
| 125 | |
| 126 | ## Alerts |
| 127 | |
| 128 | |
| 129 | The following alerts are available: |
| 130 | |
| 131 | | Alert name | On metric | Description | |
| 132 | |:------------|:----------|:------------| |
| 133 | | [ 10min_cpu_usage ](https://github.com/netdata/netdata/blob/master/src/health/health.d/cpu.conf) | system.cpu | average CPU utilization over the last 10 minutes (excluding iowait, nice and steal) | |
| 134 | | [ 10min_cpu_iowait ](https://github.com/netdata/netdata/blob/master/src/health/health.d/cpu.conf) | system.cpu | average CPU iowait time over the last 10 minutes | |
| 135 | | [ 20min_steal_cpu ](https://github.com/netdata/netdata/blob/master/src/health/health.d/cpu.conf) | system.cpu | average CPU steal time over the last 20 minutes | |
| 136 | |
| 137 | |
| 138 | ## Metrics |
| 139 | |
| 140 | Metrics grouped by *scope*. |
| 141 | |
| 142 | The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels. |
| 143 | |
| 144 | |
| 145 | |
| 146 | ### Per System statistics instance |
| 147 | |
| 148 | |
| 149 | |
| 150 | This scope has no labels. |
| 151 | |
| 152 | Metrics: |
| 153 | |
| 154 | | Metric | Dimensions | Unit | |
| 155 | |:------|:----------|:----| |
| 156 | | system.cpu | guest_nice, guest, steal, softirq, irq, user, system, nice, iowait, idle | percentage | |
| 157 | | system.intr | interrupts | interrupts/s | |
| 158 | | system.ctxt | switches | context switches/s | |
| 159 | | system.forks | started | processes/s | |
| 160 | | system.processes | running, blocked | processes | |
| 161 | | cpu.core_throttling | a dimension per cpu core | events/s | |
| 162 | | cpu.package_throttling | a dimension per package | events/s | |
| 163 | | cpu.cpufreq | a dimension per cpu core | MHz | |
| 164 | |
| 165 | ### Per cpu core |
| 166 | |
| 167 | Per-core CPU metrics. Disabled by default, can be enabled in the [configuration options](#configuration). |
| 168 | |
| 169 | Labels: |
| 170 | |
| 171 | | Label | Description | |
| 172 | |:-----------|:----------------| |
| 173 | | cpu | Identifier of the CPU core (e.g., core0, core1, core2). | |
| 174 | |
| 175 | Metrics: |
| 176 | |
| 177 | | Metric | Dimensions | Unit | |
| 178 | |:------|:----------|:----| |
| 179 | | cpu.cpu | guest_nice, guest, steal, softirq, irq, user, system, nice, iowait, idle | percentage | |
| 180 | | cpuidle.cpu_cstate_residency_time | a dimension per c-state | percentage | |
| 181 | |
| 182 |