| 1 | # Resource utilization |
| 2 | |
| 3 | Netdata is designed to automatically adjust its resource consumption based on the specific workload. |
| 4 | |
| 5 | This table shows the specific system resources affected by different Netdata features: |
| 6 | |
| 7 | | Feature | CPU | RAM | Disk I/O | Disk Space | Network Traffic | |
| 8 | |------------------------:|:---:|:---:|:--------:|:----------:|:---------------:| |
| 9 | | Collected metrics | ✓ | ✓ | ✓ | ✓ | - | |
| 10 | | Sample frequency | ✓ | - | ✓ | ✓ | - | |
| 11 | | Database mode and tiers | - | ✓ | ✓ | ✓ | - | |
| 12 | | Machine learning | ✓ | ✓ | - | - | - | |
| 13 | | Streaming | ✓ | ✓ | - | - | ✓ | |
| 14 | |
| 15 | 1. **Collected metrics** |
| 16 | |
| 17 | - **Impact**: More metrics mean higher CPU, RAM, disk I/O, and disk space usage. |
| 18 | - **Optimization**: To reduce resource consumption, consider lowering the number of collected metrics by disabling unnecessary data collectors. |
| 19 | |
| 20 | 2. **Sample frequency** |
| 21 | |
| 22 | - **Impact**: Netdata collects most metrics with 1-second granularity. This high frequency impacts CPU usage. |
| 23 | - **Optimization**: Lowering the sampling frequency (e.g., 1-second to 2-second intervals) can halve CPU usage. Balance the need for detailed data with resource efficiency. |
| 24 | |
| 25 | 3. **Database Mode** |
| 26 | |
| 27 | - **Impact**: The default database mode, `dbengine`, compresses data and writes it to disk. |
| 28 | - **Optimization**: In a Parent-Child setup, switch the Child's database mode to `ram`. This eliminates disk I/O for the Child. |
| 29 | |
| 30 | 4. **Database Tiers** |
| 31 | |
| 32 | - **Impact**: The number of database tiers directly affects memory consumption. More tiers mean higher memory usage. |
| 33 | - **Optimization**: The default number of tiers is 3. Choose the appropriate number of tiers based on data retention requirements. |
| 34 | |
| 35 | 5. **Machine Learning** |
| 36 | |
| 37 | - **Impact**: Machine learning model training is CPU-intensive, affecting overall CPU usage. |
| 38 | - **Optimization**: Consider disabling machine learning for less critical metrics or adjusting model training frequency. |
| 39 | |
| 40 | 6. **Streaming Compression** |
| 41 | |
| 42 | - **Impact**: Compression algorithm choice affects CPU usage and network traffic. |
| 43 | - **Optimization**: Select an algorithm that balances CPU efficiency with network bandwidth requirements (e.g., zstd for a good balance). |
| 44 | |
| 45 | ## Minimizing the resources used by Netdata Agents |
| 46 | |
| 47 | To optimize resource utilization, consider using a **Parent-Child** setup. |
| 48 | |
| 49 | This approach involves centralizing the collection and processing of metrics on Parent nodes while running lightweight Children Agents on edge devices. |
| 50 | |
| 51 | ## Maximizing the scale of Parent Agents |
| 52 | |
| 53 | Parents dynamically adjust their resource usage based on the volume of metrics received. However, for optimal query performance, you may need to dedicate more RAM. |
| 54 | |
| 55 | Check [RAM Requirements](/docs/netdata-agent/sizing-netdata-agents/ram-requirements.md) for more information. |
| 56 | |
| 57 | ## Netdata's performance and scalability optimization techniques |
| 58 | |
| 59 | 1. **Minimal Disk I/O** |
| 60 | |
| 61 | Netdata directly writes metric data to disk, bypassing system caches and reducing I/O overhead. Additionally, its optimized data structures minimize disk space and memory usage through efficient compression and timestamping. |
| 62 | |
| 63 | 2. **Compact Storage Engine** |
| 64 | |
| 65 | Netdata uses a custom 32-bit floating-point format tailored for efficient storage of time-series data, along with an anomaly bit. This, combined with a fixed-step database design, enables efficient storage and retrieval of data. |
| 66 | |
| 67 | | Tier | Approximate Sample Size (bytes) | |
| 68 | |-----------------------------------|---------------------------------| |
| 69 | | High-resolution tier (per-second) | 0.6 | |
| 70 | | Mid-resolution tier (per-minute) | 6 | |
| 71 | | Low-resolution tier (per-hour) | 18 | |
| 72 | |
| 73 | Timestamp optimization further reduces storage overhead by storing timestamps at regular intervals. |
| 74 | |
| 75 | 3. **Intelligent Query Engine** |
| 76 | |
| 77 | Netdata prioritizes interactive queries over background tasks like machine learning and replication, ensuring optimal user experience, especially under heavy load. |
| 78 | |
| 79 | 4. **Efficient Label Storage** |
| 80 | |
| 81 | Netdata uses pointers to reference shared label key-value pairs, minimizing memory usage, especially in highly dynamic environments. |
| 82 | |
| 83 | 5. **Scalable Streaming Protocol** |
| 84 | |
| 85 | Netdata's streaming protocol enables the creation of distributed monitoring setups, where Children offload data processing to Parents, optimizing resource utilization. |