| 1 | # Netdata Logging |
| 2 | |
| 3 | :::note |
| 4 | This document describes how Netdata generates its own logs |
| 5 | ::: |
| 6 | |
| 7 | Netdata provides enterprise-grade structured logging with full observability of all system events. The logging system is designed to be: |
| 8 | |
| 9 | - **Structured** - All logs include rich contextual fields for filtering and analysis |
| 10 | - **Performant** - Minimal overhead with built-in flood protection |
| 11 | - **Flexible** - Multiple output formats and destinations |
| 12 | - **Secure** - Integrated with platform-native security features |
| 13 | - **Standards-compliant** - Compatible with syslog, journald, ETW, and JSON standards |
| 14 | |
| 15 | ## Log sources |
| 16 | |
| 17 | Netdata supports the following log sources: |
| 18 | |
| 19 | 1. **daemon** - Core service lifecycle events, startup/shutdown, configuration changes, and fatal errors |
| 20 | 2. **collector** - Data collection events from both internal and external collectors, including errors and warnings |
| 21 | 3. **access** - Complete API access logs with request/response details, useful for security auditing |
| 22 | 4. **health** - Alert state transitions, notifications, and health monitoring events |
| 23 | |
| 24 | Each source can be independently configured with different outputs, formats, and verbosity levels. |
| 25 | |
| 26 | ## Log outputs |
| 27 | |
| 28 | For each log source, Netdata supports the following output methods: |
| 29 | |
| 30 | | Output | Platform | Description | Use Case | |
| 31 | |--------------|----------|------------------------------------------------|------------------------------------------------------------------| |
| 32 | | **off** | All | Disable this log source | Reduce log volume for specific sources | |
| 33 | | **journal** | Linux | systemd-journal with full structured fields | **Recommended for Linux** - Native integration with journald | |
| 34 | | **etw** | Windows | Event Tracing for Windows with structured data | **Recommended for Windows** - Rich field support in Event Viewer | |
| 35 | | **wel** | Windows | Windows Event Log with basic fields | Fallback when ETW is unavailable | |
| 36 | | **syslog** | Unix | Traditional syslog protocol | Legacy system compatibility | |
| 37 | | **system** | All | Platform's default stderr/stdout | Container environments | |
| 38 | | **stdout** | All | Direct to Netdata's stdout | Debugging, containers | |
| 39 | | **stderr** | All | Direct to Netdata's stderr | Debugging, containers | |
| 40 | | **filename** | All | Write to specified file path | Custom log management | |
| 41 | |
| 42 | On Linux, when systemd-journal is available, the default is `journal` for `daemon` and `collector` and `filename` for the rest. To decide if systemd-journal is available, Netdata checks: |
| 43 | |
| 44 | 1. `stderr` is connected to systemd-journald |
| 45 | 2. `/run/systemd/journal/socket` exists |
| 46 | 3. `/host/run/systemd/journal/socket` exists (`/host` is configurable in containers) |
| 47 | |
| 48 | If any of the above is detected, Netdata will select `journal` for `daemon` and `collector` sources. |
| 49 | |
| 50 | On Windows, the default is `etw` and if that is not available it falls back to `wel`. The availability of `etw` is decided at compile time. |
| 51 | |
| 52 | ## Log formats |
| 53 | |
| 54 | Netdata supports multiple log formats to integrate with different systems: |
| 55 | |
| 56 | | Format | Description | Example | Best For | |
| 57 | |-------------|---------------------------------------------------------|------------------------------------------------------------|---------------------------------| |
| 58 | | **journal** | Native systemd-journal format with all fields preserved | Binary format with 65+ structured fields | Linux systems with journald | |
| 59 | | **etw** | Event Tracing for Windows structured format | Structured events in Windows Event Viewer | Windows monitoring and analysis | |
| 60 | | **wel** | Windows Event Log format with indexed fields | String array format in Event Viewer | Windows legacy compatibility | |
| 61 | | **json** | Structured JSON with all fields as key-value pairs | `{"time":1234567890000000,"level":"info","msg":"Started"}` | Modern log aggregation systems | |
| 62 | | **logfmt** | Space-separated key=value pairs | `time="2024-01-15T10:30:00.123Z" level=info msg="Started"` | Traditional log processors | |
| 63 | |
| 64 | The format is automatically selected based on the output destination, but can be manually specified in the configuration. |
| 65 | |
| 66 | ### Field Transformations (Annotators) |
| 67 | |
| 68 | The LOGFMT, ETW, and WEL formats apply special transformations (annotators) to certain fields for better human readability: |
| 69 | |
| 70 | | Field | Raw Value | Transformation | Example | |
| 71 | |--------------------------------|-------------------------|------------------------------------|---------------------------------------------------| |
| 72 | | `time` | Unix epoch microseconds | RFC3339 with microsecond precision | `1737302400000000` → `"2025-01-19T16:00:00.000Z"` | |
| 73 | | `alert_notification_timestamp` | Unix epoch microseconds | RFC3339 with microsecond precision | `1737302400000000` → `"2025-01-19T16:00:00.000Z"` | |
| 74 | | `level` | Priority number (0-7) | Text representation | `6` → `info` | |
| 75 | | `errno` | Error number | Number + error string | `2` → `2, No such file or directory` | |
| 76 | | `winerror` | Windows error code | Number + error message | `5` → `5, Access is denied` | |
| 77 | |
| 78 | **Formats using these transformations:** |
| 79 | |
| 80 | - **LOGFMT** - All annotated fields are transformed for readability |
| 81 | - **ETW** (Event Tracing for Windows) - Uses the same transformations |
| 82 | - **WEL** (Windows Event Logs) - Uses the same transformations |
| 83 | |
| 84 | **Formats NOT using these transformations:** |
| 85 | |
| 86 | - **JSON** - Outputs raw values for all fields (no transformations applied) |
| 87 | |
| 88 | ## Log levels |
| 89 | |
| 90 | Each time Netdata logs, it assigns a priority to the log. It can be one of this (in order of importance): |
| 91 | |
| 92 | | Level | Description | |
| 93 | |-----------|----------------------------------------------------------------------------------------| |
| 94 | | emergency | a fatal condition, Netdata will most likely exit immediately after. | |
| 95 | | alert | a very important issue that may affect how Netdata operates. | |
| 96 | | critical | a very important issue the user should know which, Netdata thinks it can survive. | |
| 97 | | error | an error condition indicating that Netdata is trying to do something, but it fails. | |
| 98 | | warning | something unexpected has happened that may or may not affect the operation of Netdata. | |
| 99 | | notice | something that does not affect the operation of Netdata, but the user should notice. | |
| 100 | | info | the default log level about information the user should know. | |
| 101 | | debug | these are more verbose logs that can be ignored. | |
| 102 | |
| 103 | For `etw` these are mapped to `Verbose`, `Informational`, `Warning`, `Error` and `Critical`. |
| 104 | For `wel` these are mapped to `Informational`, `Warning`, `Error`. |
| 105 | |
| 106 | ## Logs Configuration |
| 107 | |
| 108 | [Configuration](/docs/netdata-agent/configuration/README.md) is done in the `[logs]` section of `netdata.conf`: |
| 109 | |
| 110 | ```ini |
| 111 | [logs] |
| 112 | # Global settings |
| 113 | logs to trigger flood protection = 1000 # Number of logs to trigger protection |
| 114 | logs flood protection period = 1m # Time window for flood protection |
| 115 | facility = daemon # Syslog facility (when using syslog) |
| 116 | level = info # Minimum log level (daemon/collector only) |
| 117 | |
| 118 | # Per-source configuration |
| 119 | daemon = journal # Daemon logs to systemd journal |
| 120 | collector = journal # Collector logs to systemd journal |
| 121 | access = /var/log/netdata/access.log # Access logs to file |
| 122 | health = /var/log/netdata/health.log # Health logs to file |
| 123 | ``` |
| 124 | |
| 125 | ### Key configuration options: |
| 126 | |
| 127 | - **Flood Protection**: Prevents log storms from overwhelming the system. When triggered, logs are suppressed with a summary message. |
| 128 | - **Log Level**: Controls verbosity. Only messages at or above this level are logged. |
| 129 | - **Facility**: Used for syslog categorization (local0-local7, daemon, user, etc.) |
| 130 | - **Per-Source Control**: Each source can have independent settings for maximum flexibility. |
| 131 | |
| 132 | ### Advanced per-source configuration |
| 133 | |
| 134 | Each source (`daemon`, `collector`, `access`, `health`) accepts this syntax: |
| 135 | |
| 136 | ``` |
| 137 | source = {FORMAT},level={LEVEL},protection={LOGS}/{PERIOD}@{OUTPUT} |
| 138 | ``` |
| 139 | |
| 140 | Where: |
| 141 | |
| 142 | - `{FORMAT}` - One of the [log formats](#log-formats) (json, logfmt, etc.) |
| 143 | - `{LEVEL}` - Minimum [log level](#log-levels) to be logged |
| 144 | - `{LOGS}` - Number of logs to trigger flood protection for this source |
| 145 | - `{PERIOD}` - Time period for flood protection (e.g., 1m, 30s, 5m) |
| 146 | - `{OUTPUT}` - One of the [log outputs](#log-outputs) (journal, filename, etc.) |
| 147 | |
| 148 | All parameters except `{OUTPUT}` are optional. The `@` can be omitted if only specifying output. |
| 149 | |
| 150 | #### Examples: |
| 151 | |
| 152 | ```ini |
| 153 | # JSON format to file with debug level |
| 154 | daemon = json,level=debug@/var/log/netdata/daemon.json |
| 155 | |
| 156 | # High-volume access logs with aggressive flood protection |
| 157 | access = logfmt,protection=10000/5m@/var/log/netdata/access.log |
| 158 | |
| 159 | # Critical-only health alerts to syslog |
| 160 | health = level=critical@syslog |
| 161 | |
| 162 | # Simple output specification |
| 163 | collector = journal |
| 164 | ``` |
| 165 | |
| 166 | ### Logs rotation |
| 167 | |
| 168 | Netdata includes automatic log rotation support: |
| 169 | |
| 170 | 1. **Built-in logrotate configuration** at `/etc/logrotate.d/netdata` |
| 171 | 2. **Signal handling**: Send `SIGHUP` to Netdata to reopen all log files |
| 172 | 3. **Automatic handling** for journal and ETW outputs (managed by the OS) |
| 173 | |
| 174 | Example logrotate configuration: |
| 175 | |
| 176 | ``` |
| 177 | /var/log/netdata/*.log { |
| 178 | daily |
| 179 | rotate 7 |
| 180 | compress |
| 181 | delaycompress |
| 182 | missingok |
| 183 | notifempty |
| 184 | postrotate |
| 185 | killall -USR2 netdata 2>/dev/null || true |
| 186 | endscript |
| 187 | } |
| 188 | ``` |
| 189 | |
| 190 | ## Log Fields |
| 191 | |
| 192 | <details> |
| 193 | <summary>All fields exposed by Netdata</summary> |
| 194 | |
| 195 | | `journal` | `logfmt` and `json` | `etw` | `wel` | Description | |
| 196 | |:--------------------------------------:|:------------------------------:|:-----------------------------:|:-----:|:----------------------------------------------------------------------------------------------------------| |
| 197 | | `_SOURCE_REALTIME_TIMESTAMP` | `time` | `Timestamp` | 1 | the timestamp of the event (logfmt: RFC3339, json: Unix epoch microseconds) | |
| 198 | | `SYSLOG_IDENTIFIER` | `comm` | `Program` | 2 | the program logging the event | |
| 199 | | `ND_LOG_SOURCE` | `source` | `NetdataLogSource` | 3 | one of the [log sources](#log-sources) | |
| 200 | | `PRIORITY`<br/>numeric | `level`<br/>text | `Level`<br/>text | 4 | one of the [log levels](#log-levels) | |
| 201 | | `ERRNO` | `errno` | `UnixErrno` | 5 | the numeric value of `errno` | |
| 202 | | - | `winerror` | `WindowsError` | 6 | Windows GetLastError() code | |
| 203 | | `INVOCATION_ID` | - | `InvocationID` | 7 | a unique UUID of the Netdata session, reset on every Netdata restart, inherited by systemd when available | |
| 204 | | `CODE_LINE` | - | `CodeLine` | 8 | the line number of of the source code logging this event | |
| 205 | | `CODE_FILE` | - | `CodeFile` | 9 | the filename of the source code logging this event | |
| 206 | | `CODE_FUNCTION` | - | `CodeFunction` | 10 | the function name of the source code logging this event | |
| 207 | | `TID` | `tid` | `ThreadID` | 11 | the thread id of the thread logging this event | |
| 208 | | `THREAD_TAG` | `thread` | `ThreadName` | 12 | the name of the thread logging this event | |
| 209 | | `MESSAGE_ID` | `msg_id` | `MessageID` | 13 | see [message IDs](#message-ids) | |
| 210 | | `ND_MODULE` | `module` | `Module` | 14 | the Netdata module logging this event | |
| 211 | | `ND_NIDL_NODE` | `node` | `Node` | 15 | the hostname of the node the event is related to | |
| 212 | | `ND_NIDL_INSTANCE` | `instance` | `Instance` | 16 | the instance of the node the event is related to | |
| 213 | | `ND_NIDL_CONTEXT` | `context` | `Context` | 17 | the context the event is related to (this is usually the chart name, as shown on netdata dashboards | |
| 214 | | `ND_NIDL_DIMENSION` | `dimension` | `Dimension` | 18 | the dimension the event is related to | |
| 215 | | `ND_SRC_TRANSPORT` | `src_transport` | `SourceTransport` | 19 | when the event happened during a request, this is the request transport | |
| 216 | | `ND_ACCOUNT_ID` | `account_id` | `AccountID` | 20 | Netdata Cloud account identifier | |
| 217 | | `ND_USER_NAME` | `user_name` | `UserName` | 21 | username making the request | |
| 218 | | `ND_USER_ROLE` | `user_role` | `UserRole` | 22 | user's role in the space | |
| 219 | | `ND_USER_ACCESS` | `user_access` | `UserAccess` | 23 | user's access permissions | |
| 220 | | `ND_SRC_IP` | `src_ip` | `SourceIP` | 24 | when the event happened during an inbound request, this is the IP the request came from | |
| 221 | | `ND_SRC_PORT` | `src_port` | `SourcePort` | 25 | when the event happened during an inbound request, this is the port the request came from | |
| 222 | | `ND_SRC_FORWARDED_HOST` | `src_forwarded_host` | `SourceForwardedHost` | 26 | the contents of the HTTP header `X-Forwarded-Host` | |
| 223 | | `ND_SRC_FORWARDED_FOR` | `src_forwarded_for` | `SourceForwardedFor` | 27 | the contents of the HTTP header `X-Forwarded-For` | |
| 224 | | `ND_SRC_CAPABILITIES` | `src_capabilities` | `SourceCapabilities` | 28 | when the request came from a child, this is the communication capabilities of the child | |
| 225 | | `ND_DST_TRANSPORT` | `dst_transport` | `DestinationTransport` | 29 | when the event happened during an outbound request, this is the outbound request transport | |
| 226 | | `ND_DST_IP` | `dst_ip` | `DestinationIP` | 30 | when the event happened during an outbound request, this is the IP the request destination | |
| 227 | | `ND_DST_PORT` | `dst_port` | `DestinationPort` | 31 | when the event happened during an outbound request, this is the port the request destination | |
| 228 | | `ND_DST_CAPABILITIES` | `dst_capabilities` | `DestinationCapabilities` | 32 | when the request goes to a parent, this is the communication capabilities of the parent | |
| 229 | | `ND_REQUEST_METHOD` | `req_method` | `RequestMethod` | 33 | when the event happened during an inbound request, this is the method the request was received | |
| 230 | | `ND_RESPONSE_CODE` | `code` | `ResponseCode` | 34 | when responding to a request, this this the response code | |
| 231 | | `ND_CONNECTION_ID` | `conn` | `ConnectionID` | 35 | when there is a connection id for an inbound connection, this is the connection id | |
| 232 | | `ND_TRANSACTION_ID` | `transaction` | `TransactionID` | 36 | the transaction id (UUID) of all API requests | |
| 233 | | `ND_RESPONSE_SENT_BYTES` | `sent_bytes` | `ResponseSentBytes` | 37 | the bytes we sent to API responses | |
| 234 | | `ND_RESPONSE_SIZE_BYTES` | `size_bytes` | `ResponseSizeBytes` | 38 | the uncompressed bytes of the API responses | |
| 235 | | `ND_RESPONSE_PREP_TIME_USEC` | `prep_ut` | `ResponsePreparationTimeUsec` | 39 | the time needed to prepare a response | |
| 236 | | `ND_RESPONSE_SENT_TIME_USEC` | `sent_ut` | `ResponseSentTimeUsec` | 40 | the time needed to send a response | |
| 237 | | `ND_RESPONSE_TOTAL_TIME_USEC` | `total_ut` | `ResponseTotalTimeUsec` | 41 | the total time needed to complete a response | |
| 238 | | `ND_ALERT_ID` | `alert_id` | `AlertID` | 42 | the alert id this event is related to | |
| 239 | | `ND_ALERT_EVENT_ID` | `alert_event_id` | `AlertEventID` | 44 | a sequential number of the alert transition (per host) | |
| 240 | | `ND_ALERT_UNIQUE_ID` | `alert_unique_id` | `AlertUniqueID` | 43 | a sequential number of the alert transition (per alert) | |
| 241 | | `ND_ALERT_TRANSITION_ID` | `alert_transition_id` | `AlertTransitionID` | 45 | the unique UUID of this alert transition | |
| 242 | | `ND_ALERT_CONFIG` | `alert_config` | `AlertConfig` | 46 | the alert configuration hash (UUID) | |
| 243 | | `ND_ALERT_NAME` | `alert` | `AlertName` | 47 | the alert name | |
| 244 | | `ND_ALERT_CLASS` | `alert_class` | `AlertClass` | 48 | the alert classification | |
| 245 | | `ND_ALERT_COMPONENT` | `alert_component` | `AlertComponent` | 49 | the alert component | |
| 246 | | `ND_ALERT_TYPE` | `alert_type` | `AlertType` | 50 | the alert type | |
| 247 | | `ND_ALERT_EXEC` | `alert_exec` | `AlertExec` | 51 | the alert notification program | |
| 248 | | `ND_ALERT_RECIPIENT` | `alert_recipient` | `AlertRecipient` | 52 | the alert recipient(s) | |
| 249 | | `ND_ALERT_VALUE` | `alert_value` | `AlertValue` | 54 | the current alert value | |
| 250 | | `ND_ALERT_VALUE_OLD` | `alert_value_old` | `AlertOldValue` | 55 | the previous alert value | |
| 251 | | `ND_ALERT_STATUS` | `alert_status` | `AlertStatus` | 56 | the current alert status | |
| 252 | | `ND_ALERT_STATUS_OLD` | `alert_status_old` | `AlertOldStatus` | 57 | the previous alert status | |
| 253 | | `ND_ALERT_SOURCE` | `alert_source` | `AlertSource` | 58 | the source of the alert | |
| 254 | | `ND_ALERT_UNITS` | `alert_units` | `AlertUnits` | 59 | the units of the alert | |
| 255 | | `ND_ALERT_SUMMARY` | `alert_summary` | `AlertSummary` | 60 | the summary text of the alert | |
| 256 | | `ND_ALERT_INFO` | `alert_info` | `AlertInfo` | 61 | the info text of the alert | |
| 257 | | `ND_ALERT_DURATION` | `alert_duration` | `AlertDuration` | 53 | the duration the alert was in its previous state | |
| 258 | | `ND_ALERT_NOTIFICATION_TIMESTAMP_USEC` | `alert_notification_timestamp` | `AlertNotificationTimeUsec` | 62 | the timestamp the notification delivery is scheduled | |
| 259 | | `ND_REQUEST` | `request` | `Request` | 63 | the full request during which the event happened | |
| 260 | | `MESSAGE` | `msg` | `Message` | 64 | the event message | |
| 261 | | `ND_STACK_TRACE` | `stack_trace` | `StackTrace` | 65 | stack trace at time of logging (on fatal errors) | |
| 262 | |
| 263 | For `wel` (Windows Event Logs), all logs have an array of 64 fields strings, and their index number provides their meaning. |
| 264 | For `etw` (Event Tracing for Windows), Netdata logs in a structured way, and field names are available. |
| 265 | |
| 266 | </details> |
| 267 | |
| 268 | ### Message IDs |
| 269 | |
| 270 | Netdata assigns unique UUIDs to specific event types for easy filtering and correlation: |
| 271 | |
| 272 | | Message ID | Event Type | Description | |
| 273 | |----------------------------------------|----------------------|-----------------------------------------------| |
| 274 | | `ed4cdb8f-1beb-4ad3-b57c-b3cae2d162fa` | Child Connection | A Netdata child connects to this parent | |
| 275 | | `6e2e3839-0676-4896-8b64-6045dbf28d66` | Parent Connection | This Netdata connects to a parent | |
| 276 | | `9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de` | Alert Transition | Alert changes state (CLEAR/WARNING/CRITICAL) | |
| 277 | | `6db0018e-83e3-4320-ae2a-659d78019fb7` | Alert Notification | Notification sent to external system | |
| 278 | | `1e6061a9-fbd4-4501-b3cc-c368119f2b69` | Service Start | Netdata service started | |
| 279 | | `02f47d35-0af5-4491-97bf-7a95b605a468` | Service Stop | Netdata service stopped | |
| 280 | | `23e93dfc-cbf6-4e11-aac8-58b9410d8a82` | Fatal Error | Critical error requiring attention | |
| 281 | | `acb33cb9-5778-476b-aac7-02eb7e4e151d` | ACLK Connection | Netdata Cloud (ACLK) connection state changed | |
| 282 | | `8daf5ba3-3a74-078b-6092-50db1e951f3` | Sensor State Change | Hardware sensor state transition | |
| 283 | | `ec87a561-20d5-431b-ace5-1e2fb8bba243` | Log Flood Protection | Log flooding detected and suppressed | |
| 284 | | `d1f59606-dd4d-41e3-b217-a0cfcae8e632` | Extreme Cardinality | Metric cardinality exceeds safe limits | |
| 285 | | `4fdf4081-6c12-4623-a032-b7fe73beacb8` | User Configuration | Dynamic configuration changed by user | |
| 286 | |
| 287 | You can view these events using the Netdata systemd-journal.plugin at the `MESSAGE_ID` filter, |
| 288 | or using `journalctl` like this: |
| 289 | |
| 290 | ```bash |
| 291 | # Query specific event types |
| 292 | journalctl MESSAGE_ID=ed4cdb8f-1beb-4ad3-b57c-b3cae2d162fa # Child connections |
| 293 | journalctl MESSAGE_ID=9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de # Alert transitions |
| 294 | |
| 295 | # Query multiple event types |
| 296 | journalctl MESSAGE_ID=9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de + MESSAGE_ID=6db0018e-83e3-4320-ae2a-659d78019fb7 |
| 297 | |
| 298 | # Query with time range |
| 299 | journalctl MESSAGE_ID=9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de --since "1 hour ago" |
| 300 | ``` |
| 301 | |
| 302 | ## Platform-Specific Log Access |
| 303 | |
| 304 | ### Linux: Using journalctl to query Netdata logs |
| 305 | |
| 306 | The Netdata service's processes execute within the `netdata` journal namespace. Common queries: |
| 307 | |
| 308 | ```bash |
| 309 | # Real-time log monitoring |
| 310 | journalctl -u netdata --namespace=netdata -f |
| 311 | |
| 312 | # Logs since last restart |
| 313 | journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata |
| 314 | |
| 315 | # All logs, newest first |
| 316 | journalctl -u netdata --namespace=netdata -r |
| 317 | |
| 318 | # Export logs as JSON for processing |
| 319 | journalctl -u netdata --namespace=netdata -o json --since "1 hour ago" > netdata-logs.json |
| 320 | |
| 321 | # Filter by severity |
| 322 | journalctl -u netdata --namespace=netdata -p warning # Warnings and above |
| 323 | |
| 324 | # Complex queries with field filters |
| 325 | journalctl -u netdata --namespace=netdata \ |
| 326 | ND_ALERT_STATUS=CRITICAL \ |
| 327 | ND_LOG_SOURCE=health \ |
| 328 | --since "2024-01-01" |
| 329 | ``` |
| 330 | |
| 331 | ### Windows: Using Event Viewer to View Netdata Logs |
| 332 | |
| 333 | The Netdata service on Windows systems automatically logs events to the Windows Event Viewer. |
| 334 | |
| 335 | #### Accessing logs via GUI: |
| 336 | |
| 337 | 1. Click the **Start** menu |
| 338 | 2. Type **Event Viewer** and select **Run as Administrator** |
| 339 | 3. In the Event Viewer window, expand **Applications and Services Logs** |
| 340 | 4. Click **Netdata** |
| 341 | |
| 342 | The Netdata section contains all available log categories [listed above](#log-sources). |
| 343 | |
| 344 | #### Accessing logs via PowerShell: |
| 345 | |
| 346 | ```powershell |
| 347 | # Get recent Netdata events |
| 348 | Get-WinEvent -LogName "Netdata/Health" -MaxEvents 100 |
| 349 | |
| 350 | # Filter by severity |
| 351 | Get-WinEvent -FilterHashtable @{ LogName = "Netdata/Health"; Level = 2 } # Errors only |
| 352 | |
| 353 | # Export to CSV |
| 354 | Get-WinEvent -LogName "Netdata/Health" | Export-Csv netdata-logs.csv |
| 355 | |
| 356 | # Real-time monitoring |
| 357 | Get-WinEvent -LogName "Netdata/Health" -MaxEvents 1 | |
| 358 | ForEach-Object { $_ } | |
| 359 | Out-GridView -Title "Netdata Health Events" |
| 360 | ``` |
| 361 | |
| 362 | ## Using Event Tracing for Windows (ETW) |
| 363 | |
| 364 | ETW requires the publisher `Netdata` to be registered. Our Windows installer does this automatically. |
| 365 | |
| 366 | Registering the publisher is done via a manifest (`%SystemRoot%\System32\wevt_netdata_manifest.xml`) |
| 367 | and its messages resources DLL (`%SystemRoot%\System32\wevt_netdata.dll`). |
| 368 | |
| 369 | If needed, the publisher can be registered and unregistered manually using these commands: |
| 370 | |
| 371 | ```bat |
| 372 | REM register the Netdata publisher |
| 373 | wevtutil im "%SystemRoot%\System32\wevt_netdata_manifest.xml" "/mf:%SystemRoot%\System32\wevt_netdata.dll" "/rf:%SystemRoot%\System32\wevt_netdata.dll" |
| 374 | |
| 375 | REM unregister the Netdata publisher |
| 376 | wevtutil um "%SystemRoot%\System32\wevt_netdata_manifest.xml" |
| 377 | ``` |
| 378 | |
| 379 | The structure of the logs are as follows: |
| 380 | |
| 381 | - Publisher `Netdata` |
| 382 | - Channel `Netdata/Daemon`: general messages about the Netdata service |
| 383 | - Channel `Netdata/Collector`: general messages about Netdata external plugins |
| 384 | - Channel `Netdata/Health`: alert transitions and general messages generated by Netdata's health engine |
| 385 | - Channel `Netdata/Access`: all accesses to Netdata APIs |
| 386 | - Channel `Netdata/Aclk`: for Cloud connectivity tracing (disabled by default) |
| 387 | |
| 388 | Retention can be configured per Channel via the Event Viewer. Netdata does not set a default, so the system default is used. |
| 389 | |
| 390 | > **IMPORTANT**<br/> |
| 391 | > Event Tracing for Windows (ETW) does not allow logging the percentage character `%`. |
| 392 | > The `%` followed by a number, is recursively used for fields expansion and ETW has not |
| 393 | > provided any way to escape the character for preventing further expansion.<br/> |
| 394 | > <br/> |
| 395 | > To work around this limitation, Netdata replaces all `%` which are followed by a number, with `℅` |
| 396 | > (the Unicode character `care of`). Visually, they look similar, but when copying IPv6 addresses |
| 397 | > or URLs from the logs, you have to be careful to manually replace `℅` with `%` before using them. |
| 398 | |
| 399 | ## Using Windows Event Logs (WEL) |
| 400 | |
| 401 | WEL has a different logs structure and unfortunately WEL and ETW need to use different names if they are to be used |
| 402 | concurrently. |
| 403 | |
| 404 | For WEL, Netdata logs as follows: |
| 405 | |
| 406 | - Channel `NetdataWEL` (unfortunately `Netdata` cannot be used, it conflicts with the ETW Publisher name) |
| 407 | - Publisher `NetdataDaemon`: general messages about the Netdata service |
| 408 | - Publisher `NetdataCollector`: general messages about Netdata external plugins |
| 409 | - Publisher `NetdataHealth`: alert transitions and general messages generated by Netdata's health engine |
| 410 | - Publisher `NetdataAccess`: all accesses to Netdata APIs |
| 411 | - Publisher `NetdataAclk`: for Cloud connectivity tracing (disabled by default) |
| 412 | |
| 413 | Publishers must have unique names system-wide, so we had to prefix them with `Netdata`. |
| 414 | |
| 415 | Retention can be configured per Publisher via the Event Viewer or the Registry. |
| 416 | Netdata sets by default 20MiB for all of them, except `NetdataAclk` (5MiB) and `NetdataAccess` (35MiB), |
| 417 | for a total of 100MiB. |
| 418 | |
| 419 | For WEL some registry entries are needed. Netdata automatically takes care of them when it starts. |
| 420 | |
| 421 | WEL does not have the problem ETW has with the percent character `%`, so Netdata logs it as-is. |
| 422 | |
| 423 | ## Differences between ETW and WEL |
| 424 | |
| 425 | There are key differences between ETW and WEL. |
| 426 | |
| 427 | ### Publishers and Providers |
| 428 | |
| 429 | **Publishers** are collections of ETW Providers. A Publisher is implied by a manifest file, |
| 430 | each of which is considered a Publisher, and each manifest file can define multiple **Providers** in it. |
| 431 | Other than that there is no entity related to **Publishers** in the system. |
| 432 | |
| 433 | **Publishers** are not defined for WEL. |
| 434 | |
| 435 | **Providers** are the applications or modules logging. Provider names must be unique across the system, |
| 436 | for ETW and WEL together. |
| 437 | |
| 438 | To define a **Provider**: |
| 439 | |
| 440 | - ETW requires a **Publisher** manifest coupled with resources DLLs and must be registered |
| 441 | via `wevtutil` (handled by the Netdata Windows installer automatically). |
| 442 | - WEL requires some registry entries and a message resources DLL (handled by Netdata automatically on startup). |
| 443 | |
| 444 | The Provider appears as `Source` in the Event Viewer, for both WEL and ETW. |
| 445 | |
| 446 | ### Channels |
| 447 | |
| 448 | - **Channels** for WEL are collections of WEL Providers, (each WEL Provider is a single Stream of logs). |
| 449 | - **Channels** for ETW slice the logs of each Provider into multiple Streams. |
| 450 | |
| 451 | WEL Channels cannot have the same name as ETW Providers. This is why Netdata's ETW provider is |
| 452 | called `Netdata`, and WEL channel is called `NetdataWEL`. |
| 453 | |
| 454 | Despite the fact that ETW **Publishers** and WEL **Channels** are both collections of Providers, |
| 455 | they are not similar. In ETW a Publisher is a collection on the publisher's Providers, but in WEL |
| 456 | a Channel may include independent WEL Providers (e.g. the "Applications" Channel). Additionally, |
| 457 | WEL Channels cannot include ETW Providers. |
| 458 | |
| 459 | ### Log Retention |
| 460 | |
| 461 | Retention is always defined per Stream. |
| 462 | |
| 463 | - Retention in ETW is defined per ETW Channel (ETW Provider Stream). |
| 464 | - Retention in WEL is defined per WEL Provider (each WEL Provider is a single Stream). |
| 465 | |
| 466 | ### Messages Formatting |
| 467 | |
| 468 | - ETW supports recursive fields expansion, and therefore `%N` in fields is expanded recursively |
| 469 | (or replaced with an error message if expansion fails). Netdata replaces `%N` with `℅N` to stop |
| 470 | recursive expansion (since `%N` cannot be logged otherwise). |
| 471 | - WEL performs a single field expansion, and therefore the `%` character in fields is never expanded. |
| 472 | |
| 473 | ### Usability |
| 474 | |
| 475 | - ETW names all the fields and allows multiple datatypes per field, enabling log consumers to know |
| 476 | what each field means and its datatype. |
| 477 | - WEL uses a simple string table for fields, and consumers need to map these string fields based on |
| 478 | their index. |
| 479 | |
| 480 | ## SIEM Integration |
| 481 | |
| 482 | Netdata's structured logging system is designed for seamless integration with all major Security Information and Event Management (SIEM) platforms. Logs are emitted in **standards-compliant formats** — systemd-journal, JSON, logfmt, syslog (RFC5424), Windows Event Log (WEL), and Event Tracing for Windows (ETW). |
| 483 | |
| 484 | This guarantees compatibility with SIEMs including (but not limited to): |
| 485 | **Splunk, Elastic Security (ELK Stack / OpenSearch), IBM QRadar, Microsoft Sentinel, Wazuh, CrowdStrike Falcon LogScale, Datadog Security Monitoring, Sumo Logic, LogRhythm, Securonix, ArcSight, Graylog, Chronicle SIEM, AlienVault OSSIM, Devo, Exabeam, Rapid7 InsightIDR, McAfee Enterprise Security Manager (ESM), Fortinet FortiSIEM, SolarWinds SEM, AT&T Cybersecurity USM, RSA NetWitness.** |
| 486 | |
| 487 | ### Supported Log Formats and SIEM Compatibility |
| 488 | |
| 489 | | Format / Output | Description | Commonly Used By | |
| 490 | |-----------------------------|--------------------------------------------------------------------|----------------------------------------------------------------------------------| |
| 491 | | **systemd-journal** | Native Linux logging with structured fields, tamper-proof with FSS | Splunk (journald input), Elastic Filebeat/Journalbeat, Wazuh, QRadar, Sentinel | |
| 492 | | **ETW (Event Tracing)** | Rich structured events in Event Viewer (Windows native) | Splunk UF (Win), Sentinel, QRadar, LogRhythm, ArcSight, Elastic Winlogbeat | |
| 493 | | **WEL (Windows Event Log)** | Legacy Windows Event Log array-based fields | All Windows SIEM agents (Splunk UF, Sentinel, QRadar, Wazuh, Elastic Winlogbeat) | |
| 494 | | **JSON** | Structured JSON objects with key-value pairs | Elastic/Logstash, Splunk (indexed extractions), Datadog, Sumo Logic, Graylog | |
| 495 | | **logfmt** | Human-readable `key=value` logs | Traditional syslog pipelines, Graylog, SolarWinds SEM, legacy SIEM integrations | |
| 496 | | **syslog (RFC5424)** | Standard syslog protocol | QRadar, ArcSight, LogRhythm, FortiSIEM, AlienVault OSSIM, RSA NetWitness, Devo | |
| 497 | |
| 498 | ### Key Integration Features |
| 499 | |
| 500 | 1. **Structured Logs** – All Netdata events contain contextual fields, no regex parsing required. |
| 501 | 2. **Message IDs (UUIDs)** – Unique identifiers for alert transitions, service lifecycle events, configuration changes, and network connections. Enables precise rule building without pattern matching. |
| 502 | 3. **Multiple Output Options** – Select the best integration path for your SIEM: journald (Linux), ETW/WEL (Windows), JSON/logfmt (cross-platform), syslog (legacy). |
| 503 | 4. **Security-Relevant Events** – Alert transitions, anomalous resource use, configuration changes, service errors, API access attempts. |
| 504 | 5. **Compliance Support** – Journald Forward Secure Sealing (FSS) and Windows Event Log immutability controls support PCI DSS, ISO 27001, SOC 2, HIPAA, and other frameworks. |
| 505 | |
| 506 | ### Recommended Outputs |
| 507 | |
| 508 | If you want **zero-configuration ingestion**, choose **systemd-journal** on Linux and **ETW** on Windows. |
| 509 | JSON/logfmt are universally portable but require custom field mapping inside your SIEM. |
| 510 | Syslog is provided for legacy collectors. |
| 511 | |
| 512 | | Platform | Best Output | Why | |
| 513 | |--------------------|-----------------|--------------------------------------------------------------------| |
| 514 | | **Linux** | systemd-journal | Zero-config ingestion, structured fields, tamper-proofing with FSS | |
| 515 | | **Windows** | ETW | Structured named fields in Event Viewer, native SIEM support | |
| 516 | | **Cross-Platform** | JSON | Universally portable, works everywhere, requires mapping rules | |
| 517 | | **Legacy Unix** | syslog | Compatibility with traditional SIEM collectors (RFC5424) | |
| 518 | |
| 519 | ### Integration Workflow |
| 520 | |
| 521 | 1. **Select log format** appropriate to your environment. |
| 522 | 2. **Enable relevant sources** (`health` for alerts, `access` for audit trails, `daemon` for lifecycle events). |
| 523 | 3. **Configure SIEM collection**: |
| 524 | - Journald → SIEM agent (Splunk UF, Filebeat, Wazuh agent, QRadar DSM) |
| 525 | - ETW/WEL → Windows Event Forwarding, Winlogbeat, Splunk UF, Sentinel Connector |
| 526 | - JSON/logfmt → Filebeat, Logstash, Fluent Bit, Graylog input, Sumo Logic agent |
| 527 | - Syslog → Direct to SIEM collector (QRadar, ArcSight, LogRhythm, FortiSIEM) |
| 528 | 4. **Use Message IDs** to build reliable detection rules: |
| 529 | - Alert storms → `9ce0cb58-ab8b-44df-82c4-bf1ad9ee22de` |
| 530 | - Service restarts → startup/shutdown IDs |
| 531 | - Unexpected parent/child connections → connection IDs |