5e19c2525 · cryptotaxi247/netdata-1

@@ -1,103 +1,139 @@

1 # How to optimize the Netdata Agent's performance

2

3 We designed the Netdata Agent to be incredibly lightweight, even when it's collecting a few thousand dimensions every

4 -second and visualizing that data into hundreds of charts. However, the default settings of the Netdata Agent are not

5 -optimized for performance, but for a simple, standalone setup. We want the first install to give you something you can

6 -run without any configuration. Most of the settings and options are enabled, since we want you to experience the full thing.

4 +second and visualizing that data into hundreds of charts. However, the default settings of the Netdata Agent are not

5 +optimized for performance, but for a simple, standalone setup. We want the first install to give you something you can

6 +run without any configuration. Most of the settings and options are enabled, since we want you to experience the full

7 +thing.

8

8 -By default, Netdata will automatically detect applications running on the node it is installed to start collecting metrics in

9 -real-time, has health monitoring enabled to evaluate alerts and trains Machine Learning (ML) models for each metric, to detect anomalies.

9 +By default, Netdata will automatically detect applications running on the node it is installed to start collecting

10 +metrics in real-time, has health monitoring enabled to evaluate alerts and trains Machine Learning (ML) models for each

11 +metric, to detect anomalies.

12

11 -This document describes the resources required for the various default capabilities and the strategies to optimize Netdata for production use.

13 +This document describes the resources required for the various default capabilities and the strategies to optimize

14 +Netdata for production use.

15

16 ## Summary of performance optimizations

17

18 The following table summarizes the effect of each optimization on the CPU, RAM and Disk IO utilization in production.

19

17 -Optimization | CPU | RAM | Disk IO

18 -[Use streaming and replication](#use-streaming-and-replication) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:

19 -[Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:

20 -[Reduce data collection frequency](#reduce-collection-frequency) | :heavy_check_mark: | | :heavy_check_mark:

21 -[Change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md) | | :heavy_check_mark: | :heavy_check_mark:

22 -[Use a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md) | | :heavy_check_mark: | :heavy_check_mark:

23 -[Disable machine learning](#disable-machine-learning) | :heavy_check_mark: | |

24 -[Use a reverse proxy](#run-netdata-behind-a-proxy) | :heavy_check_mark: | |

25 -[Disable/lower gzip compression for the agent dashboard](#disablelower-gzip-compression-for-the-dashboard) | :heavy_check_mark: | |

20 +| Optimization | CPU | RAM | Disk IO |

21 +|-------------------------------------------------------------------------------------------------------------------------------|--------------------|--------------------|--------------------|

25 +| [Change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md) | | :heavy_check_mark: | :heavy_check_mark: |

29 +| [Disable/lower gzip compression for the agent dashboard](#disablelower-gzip-compression-for-the-dashboard) | :heavy_check_mark: | | |

30

31 ## Resources required by a default Netdata installation

32

29 -Netdata's performance is primarily affected by **data collection/retention** and **clients accessing data**.

33 +Netdata's performance is primarily affected by **data collection/retention** and **clients accessing data**.

34

31 -You can configure almost all aspects of data collection/retention, and certain aspects of clients accessing data.

35 +You can configure almost all aspects of data collection/retention, and certain aspects of clients accessing data.

36

37 ### CPU consumption

38

39 Expect about:

36 - - 1-3% of a single core for the netdata core

37 - - 1-3% of a single core for the various collectors (e.g. go.d.plugin, apps.plugin)

38 - - 5-10% of a single core, when ML training runs

40

40 -Your experience may vary depending on the number of metrics collected, the collectors enabled and the specific environment they

41 -run on, i.e. the work they have to do to collect these metrics.

41 +- 1-3% of a single core for the netdata core

42 +- 1-3% of a single core for the various collectors (e.g. go.d.plugin, apps.plugin)

43 +- 5-10% of a single core, when ML training runs

44

43 -As a general rule, for modern hardware and VMs, the total CPU consumption of a standalone Netdata installation, including all its components,

44 -should be below 5 - 15% of a single core. For example, on 8 core server it will use only 0.6% - 1.8% of a total CPU capacity, depending on

45 -the CPU characteristics.

45 +Your experience may vary depending on the number of metrics collected, the collectors enabled and the specific

46 +environment they run on, i.e. the work they have to do to collect these metrics.

47

47 -The Netdata Agent runs with the lowest possible [process scheduling policy](https://github.com/netdata/netdata/blob/master/daemon/README.md#netdata-process-scheduling-policy), which is `nice 19`, and uses the `idle` process scheduler.

48 -Together, these settings ensure that the Agent only gets CPU resources when the node has CPU resources to space. If the

49 -node reaches 100% CPU utilization, the Agent is stopped first to ensure your applications get any available resources.

48 +As a general rule, for modern hardware and VMs, the total CPU consumption of a standalone Netdata installation,

49 +including all its components, should be below 5 - 15% of a single core. For example, on 8 core server it will use only

50 +0.6% - 1.8% of a total CPU capacity, depending on the CPU characteristics.

51

51 -To reduce CPU usage you can [disable machine learning](#disable-machine-learning),

52 -[use streaming and replication](#use-streaming-and-replication),

53 -[reduce the data collection frequency](#reduce-collection-frequency), [disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors), [use a reverse proxy](#run-netdata-behind-a-proxy), and [disable/lower gzip compression for the agent dashboard](#disablelower-gzip-compression-for-the-dashboard).

52 +The Netdata Agent runs with the lowest

53 +possible [process scheduling policy](https://github.com/netdata/netdata/blob/master/daemon/README.md#netdata-process-scheduling-policy),

54 +which is `nice 19`, and uses the `idle` process scheduler. Together, these settings ensure that the Agent only gets CPU

55 +resources when the node has CPU resources to space. If the node reaches 100% CPU utilization, the Agent is stopped first

56 +to ensure your applications get any available resources.

57 +

58 +To reduce CPU usage you can (either one or a combination of the following actions):

59 +

60 +1. [Disable machine learning](#disable-machine-learning),

61 +2. [Use streaming and replication](#use-streaming-and-replication),

62 +3. [Reduce the data collection frequency](#reduce-collection-frequency)

63 +4. [Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors)

64 +5. [Use a reverse proxy](#run-netdata-behind-a-proxy),

65 +6. [Disable/lower gzip compression for the agent dashboard](#disablelower-gzip-compression-for-the-dashboard).

66

67 ### Memory consumption

68

57 -The memory footprint of Netdata is mainly influenced by the number of metrics concurrently being collected. Expect about 150MB of RAM for a typical 64-bit server collecting about 2000 to 3000 metrics.

69 +The memory footprint of Netdata is mainly influenced by the number of metrics concurrently being collected. Expect about

70 +150MB of RAM for a typical 64-bit server collecting about 2000 to 3000 metrics.

71

59 -To estimate and control memory consumption, you can [disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors), [change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md), or [use a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md).

72 +To estimate and control memory consumption, you can (either one or a combination of the following actions):

73

74 +1. [Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors)

75 +2. [Change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md)

76 +3. [Use a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md).

77

78 ### Disk footprint and I/O

79

64 -By default, Netdata should not use more than 1GB of disk space, most of which is dedicated for storing metric data and metadata. For typical installations collecting 2000 - 3000 metrics, this storage should provide a few days of high-resolution retention (per second), about a month of mid-resolution retention (per minute) and more than a year of low-resolution retention (per hour).

80 +By default, Netdata should not use more than 1GB of disk space, most of which is dedicated for storing metric data and

81 +metadata. For typical installations collecting 2000 - 3000 metrics, this storage should provide a few days of

82 +high-resolution retention (per second), about a month of mid-resolution retention (per minute) and more than a year of

83 +low-resolution retention (per hour).

84 +

85 +Netdata spreads I/O operations across time. For typical standalone installations there should be a few write operations

86 +every 5-10 seconds of a few kilobytes each, occasionally up to 1MB. In addition, under heavy load, collectors that

87 +require disk I/O may stop and show gaps in charts.

88 +

89 +To optimize your disk footprint in any aspect described below you can:

90 +

91 +

92 +To configure retention, you can:

93

66 -Netdata spreads I/O operations across time. For typical standalone installations there should be a few write operations every 5-10 seconds of a few kilobytes each, occasionally up to 1MB. In addition, under heavy load, collectors that require disk I/O may stop and show gaps in charts.

94 +1. [Change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md).

95

68 -To configure retention, you can [change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md).

69 -To control disk I/O [use a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md), avoid querying the

70 -production system [using streaming and replication](#use-streaming-and-replication), [reduce the data collection frequency](#reduce-collection-frequency), and [disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors).

96 +To control disk I/O:

97 +

98 +1. [Use a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md),

99 +

100 +

101 +Minimize deployment impact on the production system by optimizing disk footprint:

102 +

103 +1. [Using streaming and replication](#use-streaming-and-replication)

104 +2. [Reduce the data collection frequency](#reduce-collection-frequency)

105 +3. [Disable unneeded plugins or collectors](#disable-unneeded-plugins-or-collectors).

106

107 ## Use streaming and replication

108

74 -For all production environments, parent Netdata nodes outside the production infrastructure should be receiving all

75 -collected data from children Netdata nodes running on the production infrastructure, using [streaming and replication](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/enable-streaming.md).

109 +For all production environments, parent Netdata nodes outside the production infrastructure should be receiving all

110 +collected data from children Netdata nodes running on the production infrastructure,

111 +using [streaming and replication](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/enable-streaming.md).

112

113 ### Disable health checks on the child nodes

114

79 -When you set up streaming, we recommend you run your health checks on the parent. This saves resources on the children

115 +When you set up streaming, we recommend you run your health checks on the parent. This saves resources on the children

116 and makes it easier to configure or disable alerts and agent notifications.

117

82 -The parents by default run health checks for each child, as long as the child is connected (the details are in `stream.conf`).

83 -On the child nodes you should add to `netdata.conf` the following:

118 +The parents by default run health checks for each child, as long as the child is connected (the details are

119 +in `stream.conf`). On the child nodes you should add to `netdata.conf` the following:

120

121 ```conf

122 [health]

123 enabled = no

124 ```

125

90 -### Use memory mode ram or save for the child nodes

126 +### Use memory mode ram for the child nodes

127

128 See [using a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md).

129

130 ## Disable unneeded plugins or collectors

131

132 If you know that you don't need an [entire plugin or a specific

97 -collector](https://github.com/netdata/netdata/blob/master/collectors/README.md#collector-architecture-and-terminology), you can disable any of them.

98 -Keep in mind that if a plugin/collector has nothing to do, it simply shuts down and does not consume system resources.

99 -You will only improve the Agent's performance by disabling plugins/collectors that are actively collecting metrics.

133 +collector](https://github.com/netdata/netdata/blob/master/collectors/README.md#collector-architecture-and-terminology),

134 +you can disable any of them. Keep in mind that if a plugin/collector has nothing to do, it simply shuts down and does

135 +not consume system resources. You will only improve the Agent's performance by disabling plugins/collectors that are

136 +actively collecting metrics.

137

138 Open `netdata.conf` and scroll down to the `[plugins]` section. To disable any plugin, uncomment it and set the value to

139 `no`. For example, to explicitly keep the `proc` and `go.d` plugins enabled while disabling `python.d` and `charts.d`.

@@ -135,7 +171,8 @@ The fastest way to improve the Agent's resource utilization is to reduce how oft

171 ### Global

172

173 If you don't need per-second metrics, or if the Netdata Agent uses a lot of CPU even when no one is viewing that node's

138 -dashboard, [configure the Agent](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md) to collect metrics less often.

174 +dashboard, [configure the Agent](https://github.com/netdata/netdata/blob/master/docs/configure/nodes.md) to collect

175 +metrics less often.

176

177 Open `netdata.conf` and edit the `update every` setting. The default is `1`, meaning that the Agent collects metrics

178 every second.

@@ -153,21 +190,23 @@ seconds, respectively.

190

191 Every collector and plugin has its own `update every` setting, which you can also change in the `go.d.conf`,

192 `python.d.conf`, or `charts.d.conf` files, or in individual collector configuration files. If the `update

156 -every` for an individual collector is less than the global, the Netdata Agent uses the global setting. See the [collectors configuration reference](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md) for details.

193 +every` for an individual collector is less than the global, the Netdata Agent uses the global setting. See

194 +the [collectors configuration reference](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md) for

195 +details.

196

158 -To reduce the frequency of an [internal

159 -plugin/collector](https://github.com/netdata/netdata/blob/master/collectors/README.md#collector-architecture-and-terminology), open `netdata.conf` and

160 -find the appropriate section. For example, to reduce the frequency of the `apps` plugin, which collects and visualizes

161 -metrics on application resource utilization:

197 +To reduce the frequency of

198 +an [internal_plugin/collector](https://github.com/netdata/netdata/blob/master/collectors/README.md#collector-architecture-and-terminology),

199 +open `netdata.conf` and find the appropriate section. For example, to reduce the frequency of the `apps` plugin, which

200 +collects and visualizes metrics on application resource utilization:

201

202 ```conf

203 [plugin:apps]

204 update every = 5

205 ```

206

168 -To [configure an individual collector](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md#configure-a-collector), open its specific configuration file with

169 -`edit-config` and look for the `update_every` setting. For example, to reduce the frequency of the `nginx` collector,

170 -run `sudo ./edit-config go.d/nginx.conf`:

207 +To [configure an individual collector](https://github.com/netdata/netdata/blob/master/collectors/REFERENCE.md#configure-a-collector),

208 +open its specific configuration file with `edit-config` and look for the `update_every` setting. For example, to reduce

209 +the frequency of the `nginx` collector, run `sudo ./edit-config go.d/nginx.conf`:

210

211 ```conf

212 # [ GLOBAL ]

@@ -176,30 +215,33 @@ update_every: 10

215

216 ## Lower memory usage for metrics retention

217

179 -See how to [change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md).

218 +See how

219 +to [change how long Netdata stores metrics](https://github.com/netdata/netdata/blob/master/docs/store/change-metrics-storage.md).

220

221 ## Use a different metric storage database

222

183 -Consider [using a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md) when running Netdata on IoT devices,

184 -and for children in a parent-child set up based on [streaming and replication](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/enable-streaming.md).

223 +Consider [using a different metric storage database](https://github.com/netdata/netdata/blob/master/src/database/README.md)

224 +when running Netdata on IoT devices, and for children in a parent-child set up based

225 +on [streaming and replication](https://github.com/netdata/netdata/blob/master/docs/metrics-storage-management/enable-streaming.md).

226

227 ## Disable machine learning

228

188 -Automated anomaly detection may be a powerful tool, but we recommend it to only be enabled on Netdata parents

189 -that sit outside your production infrastructure, or if you have cpu and memory to spare. You can disable ML

190 -with the following:

229 +Automated anomaly detection may be a powerful tool, but we recommend it to only be enabled on Netdata parents that sit

230 +outside your production infrastructure, or if you have cpu and memory to spare. You can disable ML with the following:

231

232 ```conf

233 [ml]

234 enabled = no

235 ```

196 -

236 +

237 ## Run Netdata behind a proxy

238

199 -A dedicated web server like nginx provides more robustness than the Agent's internal [web server](https://github.com/netdata/netdata/blob/master/web/README.md).

239 +A dedicated web server like nginx provides more robustness than the Agent's

240 +internal [web server](https://github.com/netdata/netdata/blob/master/web/README.md).

241 Nginx can handle more concurrent connections, reuse idle connections, and use fast gzip compression to reduce payloads.

242

202 -For details on installing another web server as a proxy for the local Agent dashboard, see [reverse proxies](https://github.com/netdata/netdata/blob/master/docs/category-overview-pages/reverse-proxies.md).

243 +For details on installing another web server as a proxy for the local Agent dashboard,

244 +see [reverse proxies](https://github.com/netdata/netdata/blob/master/docs/category-overview-pages/reverse-proxies.md).

245

246 ## Disable/lower gzip compression for the dashboard

247

Minor rework on document (#16925)