e04abb927 · cryptotaxi247/netdata-1

@@ -1,29 +1,43 @@

1 # Netdata Logging

2

3 -This document describes how Netdata generates its own logs, not how Netdata manages and queries logs databases.

3 +:::note

4 +This document describes how Netdata generates its own logs

5 +:::

6 +

7 +Netdata provides enterprise-grade structured logging with full observability of all system events. The logging system is designed to be:

8 +

9 +- **Structured** - All logs include rich contextual fields for filtering and analysis

10 +- **Performant** - Minimal overhead with built-in flood protection

11 +- **Flexible** - Multiple output formats and destinations

12 +- **Secure** - Integrated with platform-native security features

13 +- **Standards-compliant** - Compatible with syslog, journald, ETW, and JSON standards

14

15 ## Log sources

16

17 Netdata supports the following log sources:

18

9 -1. **daemon**, logs generated by Netdata daemon.

10 -2. **collector**, logs generated by Netdata collectors, including internal and external ones.

11 -3. **access**, API requests received by Netdata

12 -4. **health**, all alert transitions and notifications

19 +1. **daemon** - Core service lifecycle events, startup/shutdown, configuration changes, and fatal errors

20 +2. **collector** - Data collection events from both internal and external collectors, including errors and warnings

21 +3. **access** - Complete API access logs with request/response details, useful for security auditing

22 +4. **health** - Alert state transitions, notifications, and health monitoring events

23 +

24 +Each source can be independently configured with different outputs, formats, and verbosity levels.

25

26 ## Log outputs

27

28 For each log source, Netdata supports the following output methods:

29

18 -- **off**, to disable this log source

19 -- **journal**, to send the logs to systemd-journal.

20 -- **etw**, to send the logs to Event Tracing for Windows (ETW).

21 -- **wel**, to send the logs to the Windows Event Log (WEL).

22 -- **syslog**, to send the logs to syslog.

23 -- **system**, to send the output to `stderr` or `stdout` depending on the log source.

24 -- **stdout**, to write the logs to Netdata's `stdout`.

25 -- **stderr**, to write the logs to Netdata's `stderr`.

26 -- **filename**, to send the logs to a file.

31 +|--------------|----------|------------------------------------------------|------------------------------------------------------------------|

32 +| **off** | All | Disable this log source | Reduce log volume for specific sources |

41

42 On Linux, when systemd-journal is available, the default is `journal` for `daemon` and `collector` and `filename` for the rest. To decide if systemd-journal is available, Netdata checks:

43

@@ -37,13 +51,39 @@ On Windows, the default is `etw` and if that is not available it falls back to `

51

52 ## Log formats

53

40 -| Format | Description |

41 -|---------|--------------------------------------------------------------------------------------------------------|

42 -| journal | journald-specific log format. Automatically selected when logging to systemd-journal. |

43 -| etw | Event Tracing for Windows specific format. Structured logging in Event Viewer. |

44 -| wel | Windows Event Log specific format. Basic field-based logging in Event Viewer. |

45 -| logfmt | logs data as a series of key/value pairs. The default when logging to any output other than `journal`. |

46 -| json | logs data in JSON format. |

54 +Netdata supports multiple log formats to integrate with different systems:

55 +

57 +|-------------|---------------------------------------------------------|------------------------------------------------------------|---------------------------------|

61 +| **json** | Structured JSON with all fields as key-value pairs | `{"time":1234567890000000,"level":"info","msg":"Started"}` | Modern log aggregation systems |

63 +

64 +The format is automatically selected based on the output destination, but can be manually specified in the configuration.

65 +

66 +### Field Transformations (Annotators)

67 +

68 +The LOGFMT, ETW, and WEL formats apply special transformations (annotators) to certain fields for better human readability:

69 +

71 +|--------------------------------|-------------------------|------------------------------------|---------------------------------------------------|

77 +

78 +**Formats using these transformations:**

79 +

80 +- **LOGFMT** - All annotated fields are transformed for readability

81 +- **ETW** (Event Tracing for Windows) - Uses the same transformations

82 +- **WEL** (Windows Event Logs) - Uses the same transformations

83 +

84 +**Formats NOT using these transformations:**

85 +

86 +- **JSON** - Outputs raw values for all fields (no transformations applied)

87

88 ## Log levels

89

@@ -65,49 +105,87 @@ For `wel` these are mapped to `Informational`, `Warning`, `Error`.

105

106 ## Logs Configuration

107

68 -In `netdata.conf`, there are the following settings:

108 +[Configuration](/docs/netdata-agent/configuration/README.md) is done in the `[logs]` section of `netdata.conf`:

109

70 -```

110 +```ini

111 [logs]

72 - # logs to trigger flood protection = 1000

73 - # logs flood protection period = 1m

74 - # facility = daemon

75 - # level = info

76 - # daemon = journal

77 - # collector = journal

78 - # access = /var/log/netdata/access.log

79 - # health = /var/log/netdata/health.log

112 + # Global settings

113 + logs to trigger flood protection = 1000 # Number of logs to trigger protection

114 + logs flood protection period = 1m # Time window for flood protection

115 + facility = daemon # Syslog facility (when using syslog)

116 + level = info # Minimum log level (daemon/collector only)

117 +

118 + # Per-source configuration

119 + daemon = journal # Daemon logs to systemd journal

120 + collector = journal # Collector logs to systemd journal

121 + access = /var/log/netdata/access.log # Access logs to file

122 + health = /var/log/netdata/health.log # Health logs to file

123 ```

124

82 -- `logs to trigger flood protection` and `logs flood protection period` enable logs flood protection for `daemon` and `collector` sources. It can also be configured per log source.

83 -- `facility` is used only when Netdata logs to syslog.

84 -- `level` defines the minimum [log level](#log-levels) of logs that will be logged. This setting is applied only to `daemon` and `collector` sources. It can also be configured per source.

125 +### Key configuration options:

126

86 -### Configuring log sources

127 +- **Flood Protection**: Prevents log storms from overwhelming the system. When triggered, logs are suppressed with a summary message.

128 +- **Log Level**: Controls verbosity. Only messages at or above this level are logged.

129 +- **Facility**: Used for syslog categorization (local0-local7, daemon, user, etc.)

130 +- **Per-Source Control**: Each source can have independent settings for maximum flexibility.

131

88 -Each for the sources (`daemon`, `collector`, `access`, `health`), accepts the following:

132 +### Advanced per-source configuration

133 +

134 +Each source (`daemon`, `collector`, `access`, `health`) accepts this syntax:

135

136 ```

91 -source = {FORMAT},level={LEVEL},protection={LOG}/{PERIOD}@{OUTPUT}

137 +source = {FORMAT},level={LEVEL},protection={LOGS}/{PERIOD}@{OUTPUT}

138 ```

139

140 Where:

141

96 -- `{FORMAT}`, is one of the [log formats](#log-formats),

97 -- `{LEVEL}`, is the minimum [log level](#log-levels) to be logged,

98 -- `{LOGS}` is the number of `logs to trigger flood protection` configured per output,

99 -- `{PERIOD}` is the equivalent of `logs flood protection period` configured per output,

100 -- `{OUTPUT}` is one of the `[log outputs](#log-outputs),

142 +- `{FORMAT}` - One of the [log formats](#log-formats) (json, logfmt, etc.)

143 +- `{LEVEL}` - Minimum [log level](#log-levels) to be logged

144 +- `{LOGS}` - Number of logs to trigger flood protection for this source

145 +- `{PERIOD}` - Time period for flood protection (e.g., 1m, 30s, 5m)

146 +- `{OUTPUT}` - One of the [log outputs](#log-outputs) (journal, filename, etc.)

147 +

148 +All parameters except `{OUTPUT}` are optional. The `@` can be omitted if only specifying output.

149

102 -All parameters can be omitted, except `{OUTPUT}`. If `{OUTPUT}` is the only given parameter, `@` can be omitted.

150 +#### Examples:

151 +

152 +```ini

153 +# JSON format to file with debug level

154 +daemon = json,level=debug@/var/log/netdata/daemon.json

155 +

156 +# High-volume access logs with aggressive flood protection

157 +access = logfmt,protection=10000/5m@/var/log/netdata/access.log

158 +

159 +# Critical-only health alerts to syslog

160 +health = level=critical@syslog

161 +

162 +# Simple output specification

163 +collector = journal

164 +```

165

166 ### Logs rotation

167

106 -Netdata comes with `logrotate` configuration to rotate its log files periodically.

168 +Netdata includes automatic log rotation support:

169

108 -The default is usually found in `/etc/logrotate.d/netdata`.

170 +1. **Built-in logrotate configuration** at `/etc/logrotate.d/netdata`

171 +2. **Signal handling**: Send `SIGHUP` to Netdata to reopen all log files

172 +3. **Automatic handling** for journal and ETW outputs (managed by the OS)

173

110 -Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.

174 +Example logrotate configuration:

175 +

176 +```

177 +/var/log/netdata/*.log {

178 + daily

179 + rotate 7

180 + compress

181 + delaycompress

182 + missingok

183 + notifempty

184 + postrotate

185 + killall -USR2 netdata 2>/dev/null || true

186 + endscript

187 +}

188 +```

189

190 ## Log Fields

191

@@ -116,11 +194,12 @@ Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.

194

195 | `journal` | `logfmt` and `json` | `etw` | `wel` | Description |

196 |:--------------------------------------:|:------------------------------:|:-----------------------------:|:-----:|:----------------------------------------------------------------------------------------------------------|

202 +| - | `winerror` | `WindowsError` | 6 | Windows GetLastError() code |

203 | `INVOCATION_ID` | - | `InvocationID` | 7 | a unique UUID of the Netdata session, reset on every Netdata restart, inherited by systemd when available |

204 | `CODE_LINE` | - | `CodeLine` | 8 | the line number of of the source code logging this event |

205 | `CODE_FILE` | - | `CodeFile` | 9 | the filename of the source code logging this event |

@@ -134,6 +213,10 @@ Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.

@@ -166,7 +249,8 @@ Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.

@@ -174,6 +258,7 @@ Sending a `SIGHUP` to Netdata, will instruct it to re-open all its log files.

260 | `MESSAGE` | `msg` | `Message` | 64 | the event message |

262

263 For `wel` (Windows Event Logs), all logs have an array of 64 fields strings, and their index number provides their meaning.

264 For `etw` (Event Tracing for Windows), Netdata logs in a structured way, and field names are available.

@@ -182,49 +267,72 @@ For `etw` (Event Tracing for Windows), Netdata logs in a structured way, and fie

267

268 ### Message IDs

269

185 -Netdata assigns specific message IDs to certain events:

186 -

187 -- `ed4cdb8f1beb4ad3b57cb3cae2d162fa` when a Netdata child connects to this Netdata

188 -- `6e2e3839067648968b646045dbf28d66` when this Netdata connects to a Netdata parent

189 -- `9ce0cb58ab8b44df82c4bf1ad9ee22de` when alerts change state

190 -- `6db0018e83e34320ae2a659d78019fb7` when notifications are sent

270 +Netdata assigns unique UUIDs to specific event types for easy filtering and correlation:

271 +

272 +| Message ID | Event Type | Description |

273 +|----------------------------------------|----------------------|-----------------------------------------------|

274 +| `ed4cdb8f-1beb-4ad3-b57c-b3cae2d162fa` | Child Connection | A Netdata child connects to this parent |

275 +| `6e2e3839-0676-4896-8b64-6045dbf28d66` | Parent Connection | This Netdata connects to a parent |

276 +| `9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de` | Alert Transition | Alert changes state (CLEAR/WARNING/CRITICAL) |

277 +| `6db0018e-83e3-4320-ae2a-659d78019fb7` | Alert Notification | Notification sent to external system |

278 +| `1e6061a9-fbd4-4501-b3cc-c368119f2b69` | Service Start | Netdata service started |

279 +| `02f47d35-0af5-4491-97bf-7a95b605a468` | Service Stop | Netdata service stopped |

280 +| `23e93dfc-cbf6-4e11-aac8-58b9410d8a82` | Fatal Error | Critical error requiring attention |

281 +| `acb33cb9-5778-476b-aac7-02eb7e4e151d` | ACLK Connection | Netdata Cloud (ACLK) connection state changed |

282 +| `8daf5ba3-3a74-078b-6092-50db1e951f3` | Sensor State Change | Hardware sensor state transition |

283 +| `ec87a561-20d5-431b-ace5-1e2fb8bba243` | Log Flood Protection | Log flooding detected and suppressed |

284 +| `d1f59606-dd4d-41e3-b217-a0cfcae8e632` | Extreme Cardinality | Metric cardinality exceeds safe limits |

285 +| `4fdf4081-6c12-4623-a032-b7fe73beacb8` | User Configuration | Dynamic configuration changed by user |

286

287 You can view these events using the Netdata systemd-journal.plugin at the `MESSAGE_ID` filter,

288 or using `journalctl` like this:

289

290 ```bash

196 -# query children connection

197 -journalctl MESSAGE_ID=ed4cdb8f1beb4ad3b57cb3cae2d162fa

198 -

199 -# query parent connection

200 -journalctl MESSAGE_ID=6e2e3839067648968b646045dbf28d66

291 +# Query specific event types

292 +journalctl MESSAGE_ID=ed4cdb8f-1beb-4ad3-b57c-b3cae2d162fa # Child connections

293 +journalctl MESSAGE_ID=9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de # Alert transitions

294

202 -# query alert transitions

203 -journalctl MESSAGE_ID=9ce0cb58ab8b44df82c4bf1ad9ee22de

295 +# Query multiple event types

296 +journalctl MESSAGE_ID=9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de + MESSAGE_ID=6db0018e-83e3-4320-ae2a-659d78019fb7

297

205 -# query alert notifications

206 -journalctl MESSAGE_ID=6db0018e83e34320ae2a659d78019fb7

298 +# Query with time range

299 +journalctl MESSAGE_ID=9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de --since "1 hour ago"

300 ```

301

209 -## Using journalctl to query Netdata logs

302 +## Platform-Specific Log Access

303

211 -The Netdata service's processes execute within the `netdata` journal namespace. To view the Netdata logs, you should

212 -specify the `--namespace=netdata` option.

304 +### Linux: Using journalctl to query Netdata logs

305 +

306 +The Netdata service's processes execute within the `netdata` journal namespace. Common queries:

307

308 ```bash

215 -# Netdata logs since the last time the service was started

216 -journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata

309 +# Real-time log monitoring

310 +journalctl -u netdata --namespace=netdata -f

311

218 -# All netdata logs, the oldest entries are displayed first

219 -journalctl -u netdata --namespace=netdata

312 +# Logs since last restart

313 +journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata

314

221 -# All netdata logs, the newest entries are displayed first

315 +# All logs, newest first

316 journalctl -u netdata --namespace=netdata -r

317 +

318 +# Export logs as JSON for processing

319 +journalctl -u netdata --namespace=netdata -o json --since "1 hour ago" > netdata-logs.json

320 +

321 +# Filter by severity

322 +journalctl -u netdata --namespace=netdata -p warning # Warnings and above

323 +

324 +# Complex queries with field filters

325 +journalctl -u netdata --namespace=netdata \

326 + ND_ALERT_STATUS=CRITICAL \

327 + ND_LOG_SOURCE=health \

328 + --since "2024-01-01"

329 ```

330

225 -## Using Windows Event Viewer to View Netdata Logs

331 +### Windows: Using Event Viewer to View Netdata Logs

332 +

333 +The Netdata service on Windows systems automatically logs events to the Windows Event Viewer.

334

227 -The Netdata service on Windows systems automatically logs events to the Windows Event Viewer. To access these logs:

335 +#### Accessing logs via GUI:

336

337 1. Click the **Start** menu

338 2. Type **Event Viewer** and select **Run as Administrator**

@@ -233,6 +341,24 @@ The Netdata service on Windows systems automatically logs events to the Windows

341

342 The Netdata section contains all available log categories [listed above](#log-sources).

343

344 +#### Accessing logs via PowerShell:

345 +

346 +```powershell

347 +# Get recent Netdata events

348 +Get-WinEvent -LogName "Netdata/Health" -MaxEvents 100

349 +

350 +# Filter by severity

351 +Get-WinEvent -FilterHashtable @{ LogName = "Netdata/Health"; Level = 2 } # Errors only

352 +

353 +# Export to CSV

354 +Get-WinEvent -LogName "Netdata/Health" | Export-Csv netdata-logs.csv

355 +

356 +# Real-time monitoring

357 +Get-WinEvent -LogName "Netdata/Health" -MaxEvents 1 |

358 + ForEach-Object { $_ } |

359 + Out-GridView -Title "Netdata Health Events"

360 +```

361 +

362 ## Using Event Tracing for Windows (ETW)

363

364 ETW requires the publisher `Netdata` to be registered. Our Windows installer does this automatically.

@@ -330,7 +456,7 @@ they are not similar. In ETW a Publisher is a collection on the publisher's Prov

456 a Channel may include independent WEL Providers (e.g. the "Applications" Channel). Additionally,

457 WEL Channels cannot include ETW Providers.

458

333 -### Retention

459 +### Log Retention

460

461 Retention is always defined per Stream.

462

@@ -350,3 +476,56 @@ Retention is always defined per Stream.

476 what each field means and its datatype.

477 - WEL uses a simple string table for fields, and consumers need to map these string fields based on

478 their index.

479 +

480 +## SIEM Integration

481 +

482 +Netdata's structured logging system is designed for seamless integration with all major Security Information and Event Management (SIEM) platforms. Logs are emitted in **standards-compliant formats** — systemd-journal, JSON, logfmt, syslog (RFC5424), Windows Event Log (WEL), and Event Tracing for Windows (ETW).

483 +

484 +This guarantees compatibility with SIEMs including (but not limited to):

485 +**Splunk, Elastic Security (ELK Stack / OpenSearch), IBM QRadar, Microsoft Sentinel, Wazuh, CrowdStrike Falcon LogScale, Datadog Security Monitoring, Sumo Logic, LogRhythm, Securonix, ArcSight, Graylog, Chronicle SIEM, AlienVault OSSIM, Devo, Exabeam, Rapid7 InsightIDR, McAfee Enterprise Security Manager (ESM), Fortinet FortiSIEM, SolarWinds SEM, AT&T Cybersecurity USM, RSA NetWitness.**

486 +

487 +### Supported Log Formats and SIEM Compatibility

488 +

489 +| Format / Output | Description | Commonly Used By |

490 +|-----------------------------|--------------------------------------------------------------------|----------------------------------------------------------------------------------|

491 +| **systemd-journal** | Native Linux logging with structured fields, tamper-proof with FSS | Splunk (journald input), Elastic Filebeat/Journalbeat, Wazuh, QRadar, Sentinel |

492 +| **ETW (Event Tracing)** | Rich structured events in Event Viewer (Windows native) | Splunk UF (Win), Sentinel, QRadar, LogRhythm, ArcSight, Elastic Winlogbeat |

493 +| **WEL (Windows Event Log)** | Legacy Windows Event Log array-based fields | All Windows SIEM agents (Splunk UF, Sentinel, QRadar, Wazuh, Elastic Winlogbeat) |

494 +| **JSON** | Structured JSON objects with key-value pairs | Elastic/Logstash, Splunk (indexed extractions), Datadog, Sumo Logic, Graylog |

495 +| **logfmt** | Human-readable `key=value` logs | Traditional syslog pipelines, Graylog, SolarWinds SEM, legacy SIEM integrations |

496 +| **syslog (RFC5424)** | Standard syslog protocol | QRadar, ArcSight, LogRhythm, FortiSIEM, AlienVault OSSIM, RSA NetWitness, Devo |

497 +

498 +### Key Integration Features

499 +

500 +1. **Structured Logs** – All Netdata events contain contextual fields, no regex parsing required.

501 +2. **Message IDs (UUIDs)** – Unique identifiers for alert transitions, service lifecycle events, configuration changes, and network connections. Enables precise rule building without pattern matching.

502 +3. **Multiple Output Options** – Select the best integration path for your SIEM: journald (Linux), ETW/WEL (Windows), JSON/logfmt (cross-platform), syslog (legacy).

503 +4. **Security-Relevant Events** – Alert transitions, anomalous resource use, configuration changes, service errors, API access attempts.

504 +5. **Compliance Support** – Journald Forward Secure Sealing (FSS) and Windows Event Log immutability controls support PCI DSS, ISO 27001, SOC 2, HIPAA, and other frameworks.

505 +

506 +### Recommended Outputs

507 +

508 +If you want **zero-configuration ingestion**, choose **systemd-journal** on Linux and **ETW** on Windows.

509 +JSON/logfmt are universally portable but require custom field mapping inside your SIEM.

510 +Syslog is provided for legacy collectors.

511 +

512 +| Platform | Best Output | Why |

513 +|--------------------|-----------------|--------------------------------------------------------------------|

514 +| **Linux** | systemd-journal | Zero-config ingestion, structured fields, tamper-proofing with FSS |

515 +| **Windows** | ETW | Structured named fields in Event Viewer, native SIEM support |

516 +| **Cross-Platform** | JSON | Universally portable, works everywhere, requires mapping rules |

517 +| **Legacy Unix** | syslog | Compatibility with traditional SIEM collectors (RFC5424) |

518 +

519 +### Integration Workflow

520 +

521 +1. **Select log format** appropriate to your environment.

522 +2. **Enable relevant sources** (`health` for alerts, `access` for audit trails, `daemon` for lifecycle events).

523 +3. **Configure SIEM collection**:

524 + - Journald → SIEM agent (Splunk UF, Filebeat, Wazuh agent, QRadar DSM)

525 + - ETW/WEL → Windows Event Forwarding, Winlogbeat, Splunk UF, Sentinel Connector

526 + - JSON/logfmt → Filebeat, Logstash, Fluent Bit, Graylog input, Sumo Logic agent

527 + - Syslog → Direct to SIEM collector (QRadar, ArcSight, LogRhythm, FortiSIEM)

528 +4. **Use Message IDs** to build reliable detection rules:

529 + - Alert storms → `9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de`

530 + - Service restarts → startup/shutdown IDs

531 + - Unexpected parent/child connections → connection IDs

updated logging documentation and added natural siem integration (#20829)