| 1 | # Daemon |
| 2 | |
| 3 | The Netdata Daemon, often referred to as the Netdata Agent, controls the entire operation of the monitoring system. This document provides an overview of command-line options, debugging, and troubleshooting. |
| 4 | |
| 5 | ## Command-Line Options |
| 6 | |
| 7 | While Netdata typically runs with default settings, you can override configurations using command-line options. For a complete list of options and detailed descriptions, run: |
| 8 | |
| 9 | ```sh |
| 10 | netdata -h |
| 11 | ``` |
| 12 | |
| 13 | **Common Options**: |
| 14 | |
| 15 | | Option | Description | Default | |
| 16 | |---------------|-----------------------------------|-----------------------------| |
| 17 | | `-c filename` | Specify configuration file | `/etc/netdata/netdata.conf` | |
| 18 | | `-D` | Run in foreground (do not fork) | Run in background | |
| 19 | | `-d` | Run in background (fork) | Run in background | |
| 20 | | `-P filename` | Save PID to file | No PID file | |
| 21 | | `-i IP` | Set listening IP address | All IPv4 and IPv6 addresses | |
| 22 | | `-p port` | Set API/Web port | `19999` | |
| 23 | | `-s path` | Set prefix for `/proc` and `/sys` | No prefix | |
| 24 | | `-t seconds` | Set internal clock interval | `1` | |
| 25 | | `-u username` | Set running user | `netdata` | |
| 26 | | `-v`, `-V` | Display version and exit | - | |
| 27 | | `-W options` | Advanced options (see below) | - | |
| 28 | |
| 29 | ## Logging |
| 30 | |
| 31 | For details about Netdata's logging system and configuration, see [Netdata Logging](/src/libnetdata/log/README.md). |
| 32 | |
| 33 | ## Process Scheduling Policy (Unix only) |
| 34 | |
| 35 | Netdata uses the `batch` scheduling policy by default, which helps eliminate gaps in charts on busy systems while maintaining low system impact. |
| 36 | |
| 37 | |
| 38 | <details> |
| 39 | <summary>Change (Systemd)</summary> |
| 40 | |
| 41 | When Netdata runs under systemd as the `netdata` user, it can’t directly modify its scheduling policy and priority. Instead, configure these settings through systemd. |
| 42 | |
| 43 | 1. Use the following command to edit the systemd service (requires root privileges): |
| 44 | |
| 45 | ```bash |
| 46 | systemctl edit netdata |
| 47 | ``` |
| 48 | |
| 49 | 2. Below are the available scheduling options. Uncomment and adjust the values according to your needs: |
| 50 | |
| 51 | ```bash |
| 52 | [Service] |
| 53 | ## CPU Scheduling Policy |
| 54 | ## Options: other (system default) | batch | idle | fifo | rr |
| 55 | #CPUSchedulingPolicy=other |
| 56 | |
| 57 | ## CPU Scheduling Priority (for fifo and rr policies) |
| 58 | ## Range: 1 (lowest) to 99 (highest) |
| 59 | ## Note: Netdata can only reduce this value via netdata.conf |
| 60 | #CPUSchedulingPriority=1 |
| 61 | |
| 62 | ## Process Nice Level (for other and batch policies) |
| 63 | ## Range: -20 (highest) to 19 (lowest) |
| 64 | ## Note: Netdata can only increase this value via netdata.conf |
| 65 | #Nice=0 |
| 66 | ``` |
| 67 | |
| 68 | 3. Configure Netdata to preserve systemd settings by editing `netdata.conf`: |
| 69 | ```text |
| 70 | [global] |
| 71 | process scheduling policy = keep |
| 72 | ``` |
| 73 | |
| 74 | 4. [Restart](/docs/netdata-agent/start-stop-restart.md) netdata service. |
| 75 | |
| 76 | </details> |
| 77 | |
| 78 | |
| 79 | <details> |
| 80 | <summary>Change (Non-Systemd)</summary> |
| 81 | |
| 82 | To modify the scheduling policy, [edit](/docs/netdata-agent/configuration/README.md#edit-configuration-files) `netdata.conf`: |
| 83 | |
| 84 | ```text |
| 85 | [global] |
| 86 | process scheduling policy = idle |
| 87 | ``` |
| 88 | |
| 89 | **Available Policies**: |
| 90 | |
| 91 | | Policy | Description | |
| 92 | |----------------|---------------------------------------------------------------------------------------------------------------------------| |
| 93 | | `batch` | Similar to `other` but treats the thread as CPU-intensive, applying a mild scheduling penalty. This is Netdata's default. | |
| 94 | | `idle` | Uses CPU only when available (lower than nice 19). Under extreme system load, may cause 1-2 second gaps in charts. | |
| 95 | | `other`/`nice` | Linux's default process policy. Uses dynamic priorities based on the process's `nice` level. | |
| 96 | | `fifo` | Requires static priorities above 0. Immediately preempts `other`, `batch`, or `idle` threads. No time slicing. | |
| 97 | | `rr` | Enhanced `fifo` with maximum time quantum for each thread. | |
| 98 | | `keep`/`none` | Maintains existing scheduling policy and priority settings. | |
| 99 | |
| 100 | For additional details about process scheduling, see [man sched](https://man7.org/linux/man-pages/man7/sched.7.html). |
| 101 | |
| 102 | **FIFO and RR Priority**: |
| 103 | |
| 104 | When using `fifo` or `rr` policies, you can set the process priority in `netdata.conf`: |
| 105 | |
| 106 | ```text |
| 107 | [global] |
| 108 | process scheduling priority = 0 |
| 109 | ``` |
| 110 | |
| 111 | Priority values range from 0 to 99, with higher values indicating higher process importance. |
| 112 | |
| 113 | **Nice Level** |
| 114 | |
| 115 | For `other`, `nice`, or `batch` policies, you can adjust the nice level: |
| 116 | |
| 117 | ```text |
| 118 | [global] |
| 119 | process nice level = 19 |
| 120 | ``` |
| 121 | |
| 122 | The nice level ranges from -20 (the highest priority) to 19 (the lowest priority). A higher value means the process is "nicer" to other processes by using fewer CPU resources. |
| 123 | |
| 124 | </details> |
| 125 | |
| 126 | ## Debugging |
| 127 | |
| 128 | When Netdata is compiled with debugging enabled: |
| 129 | |
| 130 | - **Performance Impact**: Compiler optimizations are disabled, which may result in slightly reduced performance. |
| 131 | - **Debug Logging**: Disabled by default. To enable logging for specific components: |
| 132 | - Open `netdata.conf`. |
| 133 | - Set the `debug flags` option to a hex value that corresponds to the components you want to trace. |
| 134 | - Debug flag options are defined in [log.h](https://raw.githubusercontent.com/netdata/netdata/master/src/libnetdata/log/log.h) as `D_*` values. Use `0xffffffffffffffff` to enable all debug flags. |
| 135 | |
| 136 | > **Important** |
| 137 | > |
| 138 | > Remember to disable debug logging (`debug flags = 0`) after you finish troubleshooting. Debug logs can grow rapidly and consume significant disk space. |
| 139 | |
| 140 | ### Compiling Netdata with debugging |
| 141 | |
| 142 | To compile Netdata with debugging capabilities: |
| 143 | |
| 144 | ```sh |
| 145 | # Navigate to Netdata source directory |
| 146 | cd /usr/src/netdata.git |
| 147 | |
| 148 | # Install with debugging enabled |
| 149 | CFLAGS="-O1 -ggdb -DNETDATA_INTERNAL_CHECKS=1" ./netdata-installer.sh |
| 150 | ``` |
| 151 | |
| 152 | After installation, use the `debug flags` setting in your configuration to specify which components to trace. |
| 153 | |
| 154 | This compilation method includes debugging information in the binary and enables internal checks. **This is recommended only for development or troubleshooting purposes**. |
| 155 | |
| 156 | ### Debugging Crashes |
| 157 | |
| 158 | While Netdata is designed to be highly stable, if you encounter a crash, providing stack traces greatly helps in identifying the root cause. |
| 159 | |
| 160 | To generate useful debugging information, ensure you have Netdata [compiled with debugging enabled](#debugging). |
| 161 | |
| 162 | #### Method 1: Analyzing a Core Dump |
| 163 | |
| 164 | If you have a core dump from the crash, run: |
| 165 | |
| 166 | ```sh |
| 167 | gdb $(which netdata) /path/to/core/dump |
| 168 | ``` |
| 169 | |
| 170 | #### Method 2: Using Valgrind for Reproducible Crashes |
| 171 | |
| 172 | If you can reproduce the crash consistently: |
| 173 | |
| 174 | 1. Install the `valgrind` package |
| 175 | 2. Run Netdata under Valgrind: |
| 176 | |
| 177 | ```sh |
| 178 | valgrind $(which netdata) -D |
| 179 | ``` |
| 180 | |
| 181 | Netdata will run significantly slower under Valgrind. When the crash occurs, Valgrind will output the stack trace to your console. |
| 182 | |
| 183 | #### Reporting the Issue |
| 184 | |
| 185 | For either method: |
| 186 | |
| 187 | - Create a [new GitHub issue](https://github.com/netdata/netdata/issues/new/choose). |
| 188 | - Include the complete output from gdb or Valgrind. |
| 189 | - Add any relevant details about the circumstances of the crash. |