| 1 | # Facets and Logs Query System |
| 2 | |
| 3 | The facets engine provides a unified interface for querying, searching, and analyzing log data from multiple sources including systemd journal (Linux) and Windows Event logs. |
| 4 | |
| 5 | ## Overview |
| 6 | |
| 7 | The logs query system uses a two-phase approach: |
| 8 | 1. **Discovery Phase**: Get information about available log sources |
| 9 | 2. **Query Phase**: Query the actual log data |
| 10 | |
| 11 | ## Discovery Phase (info=true) |
| 12 | |
| 13 | When called with `info=true`, the system returns metadata about available log sources and accepted parameters. |
| 14 | |
| 15 | ### Response Structure |
| 16 | |
| 17 | ```json |
| 18 | { |
| 19 | "_request": { |
| 20 | // Echo of the request parameters |
| 21 | }, |
| 22 | "versions": { |
| 23 | "sources": 1748491820000467 // Version timestamp for source list |
| 24 | }, |
| 25 | "v": 3, // API version |
| 26 | "accepted_params": [ |
| 27 | // List of all parameters the API accepts |
| 28 | ], |
| 29 | "required_params": [ |
| 30 | // Array of required parameter definitions |
| 31 | ], |
| 32 | "show_ids": false, |
| 33 | "has_history": true, |
| 34 | "pagination": { |
| 35 | "enabled": true, |
| 36 | "key": "anchor", |
| 37 | "column": "timestamp", |
| 38 | "units": "timestamp_usec" |
| 39 | }, |
| 40 | "status": 200, |
| 41 | "type": "table", |
| 42 | "help": "..." |
| 43 | } |
| 44 | ``` |
| 45 | |
| 46 | ### Platform-Specific Sources |
| 47 | |
| 48 | #### Linux (systemd journal) |
| 49 | Sources are hierarchical and represent different views of journal data: |
| 50 | - `all` - All available logs |
| 51 | - `all-local-logs` - All local logs |
| 52 | - `all-local-namespaces` - Namespace-specific logs |
| 53 | - `all-local-system-logs` - System logs only |
| 54 | - `all-local-user-logs` - User logs only |
| 55 | - `namespace-*` - Specific namespace logs |
| 56 | - `remote-*` - Remote logs (if available) |
| 57 | |
| 58 | Each source includes: |
| 59 | - File count |
| 60 | - Total size |
| 61 | - Time coverage (e.g., "1y 6mo 8d 16h 21m 18s") |
| 62 | |
| 63 | #### Windows (Event logs) |
| 64 | Sources are Windows Event channels organized by provider: |
| 65 | - `All` - All event channels |
| 66 | - `All-Admin` - Administrative channels |
| 67 | - `All-Classic` - Classic event logs (Application, Security, System) |
| 68 | - `All-Operational` - Operational channels |
| 69 | - Provider-specific channels (e.g., `Microsoft-Windows-*/Operational`) |
| 70 | - Application channels (e.g., `Netdata/Health`, `Netdata/Daemon`) |
| 71 | |
| 72 | Each source includes: |
| 73 | - Channel count |
| 74 | - Total size |
| 75 | - Time coverage |
| 76 | - **Entry count** (unique to Windows) |
| 77 | - `default_selected` flag (indicates if selected by default) |
| 78 | |
| 79 | **Windows Events Performance Optimization:** |
| 80 | - Windows Events have few native fields (Level, TimeCreated, EventID, etc.) |
| 81 | - Additional data is stored in XML format within each event |
| 82 | - XML parsing is expensive, so windows-events.plugin uses **lazy loading**: |
| 83 | - XML is parsed only for rows that will be returned to the user |
| 84 | - For full-text search, XML is fetched and searched but not parsed |
| 85 | - Field extraction from XML happens only for visible rows |
| 86 | - This approach balances search capability with performance |
| 87 | |
| 88 | ## Query Parameters |
| 89 | |
| 90 | ### Core Parameters |
| 91 | |
| 92 | #### Required Parameters |
| 93 | - `__logs_sources` - Multiselect field for choosing log sources to query |
| 94 | |
| 95 | #### Time Filtering |
| 96 | - `after` - Start timestamp (Unix seconds) |
| 97 | - `before` - End timestamp (Unix seconds) |
| 98 | |
| 99 | #### Pagination |
| 100 | - `anchor` - Pagination anchor (timestamp in microseconds) |
| 101 | - `direction` - "backward" (newest first) or "forward" |
| 102 | - `last` - Number of entries to return (default: 200) |
| 103 | |
| 104 | #### Search and Analysis |
| 105 | - `query` - Full-text search query |
| 106 | - `facets` - Array of field names to analyze and return facet counts |
| 107 | - `histogram` - Field name for generating time-based histogram (default: `_PRIORITY` for Linux, `Level` for Windows) |
| 108 | |
| 109 | #### Options |
| 110 | - `data_only` - Return only log data without facet analysis |
| 111 | - `delta` - Return incremental updates |
| 112 | - `tail` - Follow mode for real-time updates |
| 113 | - `sampling` - Sample rate for large datasets (default: 1000000) |
| 114 | - `slice` - Time slicing for analysis |
| 115 | - `if_modified_since` - Conditional requests |
| 116 | |
| 117 | ### Default Values |
| 118 | When parameters are not specified: |
| 119 | - `source_type`: 1 (platform default) |
| 120 | - `direction`: "backward" (newest entries first) |
| 121 | - `last`: 200 entries |
| 122 | - `sampling`: 1000000 |
| 123 | - Time range: Last 15 minutes (900 seconds) |
| 124 | |
| 125 | ## Auto-Selection Behavior |
| 126 | |
| 127 | The system is designed to work without user intervention: |
| 128 | |
| 129 | 1. **Source Auto-Selection**: |
| 130 | - If no sources are selected, the UI/client should select the first available source |
| 131 | - On Windows, sources with `default_selected: true` should be pre-selected |
| 132 | |
| 133 | 2. **Immediate Data Fetch**: |
| 134 | - After the `info` call, clients should immediately fetch data |
| 135 | - Use default or auto-selected sources |
| 136 | - Apply default time range and pagination |
| 137 | |
| 138 | ## Query Response Format |
| 139 | |
| 140 | The query response includes: |
| 141 | - Faceted results with counts (unless `data_only=true`) |
| 142 | - Log entries matching the query |
| 143 | - Histogram data with breakdown per facet value (unless `data_only=true`) |
| 144 | - Pagination information for fetching more results |
| 145 | - Source failures return partial data (no explicit error indication) |
| 146 | |
| 147 | ## Usage Example Flow |
| 148 | |
| 149 | 1. **Get available sources**: |
| 150 | ``` |
| 151 | GET /api/v1/logs?info=true |
| 152 | ``` |
| 153 | |
| 154 | 2. **Parse response and auto-select sources**: |
| 155 | - Use sources with `default_selected: true` (Windows) |
| 156 | - Or select first source (Linux) |
| 157 | |
| 158 | 3. **Query logs**: |
| 159 | ``` |
| 160 | POST /api/v1/logs |
| 161 | { |
| 162 | "__logs_sources": ["Application", "System"], |
| 163 | "after": 1748527000, |
| 164 | "before": 1748528000, |
| 165 | "query": "error", |
| 166 | "last": 100 |
| 167 | } |
| 168 | ``` |
| 169 | |
| 170 | ## Modes of Operation |
| 171 | |
| 172 | ### Fast Data Query (`data_only=true`) |
| 173 | The fastest query mode that seeks directly to the anchor point (or time boundary) and returns the next `last` entries in the specified `direction`: |
| 174 | - Does not scan the entire time window |
| 175 | - No facet calculation unless `delta=true` is specified |
| 176 | - Uses learned out-of-order deltas to minimize data scanning |
| 177 | - Ideal for pagination and real-time updates |
| 178 | |
| 179 | ### Full Analysis Query (`data_only=false`) - Default |
| 180 | Scans the entire time window to calculate complete facet counts and histogram data: |
| 181 | - Returns the same `last` entries as data_only mode |
| 182 | - Provides comprehensive statistics for the entire time range |
| 183 | - Required for initial queries to understand data distribution |
| 184 | - Must complete full scan due to potential out-of-order entries |
| 185 | |
| 186 | ### Real-time Following (`tail=true`) |
| 187 | Combined with `data_only=true` and `delta=true` for efficient log following: |
| 188 | - Scans only new entries since last anchor |
| 189 | - When used with `if_modified_since`, returns HTTP 304 if no changes |
| 190 | - Change detection: |
| 191 | - Linux: Uses inotify for file watching |
| 192 | - Windows: Polls providers for latest timestamps |
| 193 | - Equivalent to `tail -f` or `journalctl -f` |
| 194 | - Clients typically poll once per second for updates |
| 195 | |
| 196 | ## Advanced Parameters |
| 197 | |
| 198 | ### Sampling (`sampling`) |
| 199 | Controls when statistical sampling begins: |
| 200 | - Default: 1,000,000 entries (unusually high for accurate results) |
| 201 | - Sampling algorithm (systemd-journal only): |
| 202 | - Estimates volume per journal file based on time window |
| 203 | - Distributes sampling proportionally across files |
| 204 | - Maintains temporal representation across the dataset |
| 205 | - Provides accurate counts up to the sampling threshold |
| 206 | - Above threshold, provides statistically representative estimates |
| 207 | - Sampling stages: |
| 208 | - First stage: Skip facet processing, continue row counting |
| 209 | - Second stage: Skip rows, estimate counts |
| 210 | - Histogram shows additional dimensions: `unsampled` and `estimated` |
| 211 | |
| 212 | ### Slicing (`slice`) |
| 213 | Database-level filtering optimization (Linux only): |
| 214 | - `slice=false` (default): |
| 215 | - Facets library reads all data |
| 216 | - Knows counts for all facet values (selected and non-selected) |
| 217 | - Shows all possible filter options to users |
| 218 | - `slice=true` (when backend supports it): |
| 219 | - Database uses indexes to filter data |
| 220 | - Faster queries for filtered datasets |
| 221 | - Non-selected facet values may show as zero |
| 222 | - Backend provides list of all possible values separately |
| 223 | |
| 224 | Windows Events note: Does not support slicing; Netdata maintains internal cache of possible facet values. |
| 225 | |
| 226 | ## Performance Characteristics |
| 227 | |
| 228 | ### Processing Speed |
| 229 | - Typical: ~200,000 rows/second on modern hardware |
| 230 | - Factors affecting speed: |
| 231 | - Query complexity |
| 232 | - Number of sources |
| 233 | - Time window size |
| 234 | - Filtering and facets |
| 235 | |
| 236 | ### Progress Reporting |
| 237 | For long-running queries: |
| 238 | - UI can request progress updates |
| 239 | - Each progress check extends the timeout |
| 240 | - Immediate cancellation on user action |
| 241 | - Prevents timeout during active monitoring |
| 242 | |
| 243 | ### Out-of-Order Data Handling |
| 244 | Log databases often contain out-of-order entries: |
| 245 | - Plugins learn maximum out-of-order deltas per source |
| 246 | - Linux: Per journal file |
| 247 | - Windows: Per event provider |
| 248 | - Enables efficient minimal scanning for `data_only=true` queries |
| 249 | |
| 250 | ## Implementation Notes |
| 251 | |
| 252 | - The backend automatically detects the platform and returns appropriate sources |
| 253 | - The same query interface works for both systemd journal and Windows Events |
| 254 | - Sources can change over time (tracked by `versions.sources`) |
| 255 | - Pagination uses microsecond timestamps for precise positioning |
| 256 | - The system supports real-time following with `tail=true` |
| 257 | - All caching and optimization is transparent to the caller |
| 258 | - Authentication handled via USER_AUTH structure from Netdata Cloud SSO |
| 259 | - Plugins run with root/Administrator privileges for full data access |
| 260 | - User preferences (filters, time windows, facets) persist at dashboard level |
| 261 | - All queries are logged for audit purposes (standard system logging) |
| 262 | |
| 263 | ## Integration with Netdata Architecture |
| 264 | |
| 265 | ### Plugin Communication |
| 266 | - The facets library runs within plugins (systemd-journal.plugin, windows-events.plugin) |
| 267 | - Plugins can run anywhere in the Netdata ecosystem (parent nodes, child nodes, etc.) |
| 268 | - Communication with plugins happens via **rrdfunctions** - Netdata's function execution framework |
| 269 | - rrdfunctions provide the transport layer to send requests to plugins and receive responses |
| 270 | |
| 271 | ### MCP (Model Context Protocol) Integration |
| 272 | - Netdata has an MCP server implementation for LLM interactions |
| 273 | - Functions that return `has_history=true` (like logs) are currently excluded from MCP's table processing |
| 274 | - Regular functions return simple table format: `{"type": "table", "data": [...], "columns": {...}}` |
| 275 | - Logs functions return a different JSON structure with additional fields for: |
| 276 | - Faceted search results |
| 277 | - Histogram data |
| 278 | - Pagination information (anchors) |
| 279 | - Time-based navigation |
| 280 | - Dynamic field discovery |
| 281 | |
| 282 | #### JSON Structure Differences: Regular Tables vs Logs |
| 283 | |
| 284 | **Regular Table Functions** (e.g., processes, network connections): |
| 285 | ```json |
| 286 | { |
| 287 | "status": 200, |
| 288 | "type": "table", |
| 289 | "has_history": false, |
| 290 | "columns": { /* column definitions */ }, |
| 291 | "data": [ /* simple arrays of values */ ], |
| 292 | "charts": { /* optional chart configs */ } |
| 293 | } |
| 294 | ``` |
| 295 | |
| 296 | **Logs Functions** (e.g., systemd-journal, Windows events): |
| 297 | ```json |
| 298 | { |
| 299 | "status": 200, |
| 300 | "type": "table", |
| 301 | "has_history": true, |
| 302 | "_request": { /* complete request parameters */ }, |
| 303 | "columns": { /* column definitions with facet support */ }, |
| 304 | "data": [ |
| 305 | /* arrays starting with timestamp, rowOptions, then values */ |
| 306 | ], |
| 307 | "facets": { /* available filters with counts */ }, |
| 308 | "histogram": { /* time-series visualization */ }, |
| 309 | "pagination": { /* anchor-based navigation */ }, |
| 310 | "_journal_files": { /* source metadata */ }, |
| 311 | "_sampling": { /* sampling statistics */ }, |
| 312 | "items": 12345, |
| 313 | "last_modified": 1234567890, |
| 314 | /* many more metadata fields */ |
| 315 | } |
| 316 | ``` |
| 317 | |
| 318 | Key differences: |
| 319 | - Logs have 20+ additional top-level fields |
| 320 | - Data rows include timestamps and metadata |
| 321 | - Built-in support for faceted filtering and time navigation |
| 322 | - Rich metadata about data sources and query performance |
| 323 | |
| 324 | ### Dynamic Schema Challenge |
| 325 | - systemd-journal can store any structured data with custom fields |
| 326 | - Plugins discover new fields dynamically as they process data |
| 327 | - LLMs need a way to discover available fields to build intelligent queries |
| 328 | - Current MCP implementation doesn't handle this dynamic schema discovery |
| 329 | |
| 330 | #### Example: Schema Variability in systemd-journal |
| 331 | The same systemd-journal can contain completely different datasets with different schemas: |
| 332 | |
| 333 | 1. **Standard System Logs** - Traditional journal fields: |
| 334 | - System fields: `_HOSTNAME`, `_UID`, `_GID`, `_PID`, `_COMM`, `_EXE` |
| 335 | - Message fields: `MESSAGE`, `PRIORITY`, `SYSLOG_FACILITY`, `SYSLOG_IDENTIFIER` |
| 336 | - Systemd fields: `_SYSTEMD_UNIT`, `_SYSTEMD_CGROUP`, `_SYSTEMD_SLICE` |
| 337 | - Boot/runtime fields: `_BOOT_ID`, `_MACHINE_ID`, `_RUNTIME_SCOPE` |
| 338 | |
| 339 | 2. **Netdata Agent Events** - Custom application data with `AE_` prefix: |
| 340 | - Agent metadata: `AE_AGENT_ID`, `AE_AGENT_VERSION`, `AE_AGENT_STATUS` |
| 341 | - Hardware info: `AE_HW_BOARD_NAME`, `AE_HW_CHASSIS_TYPE`, `AE_HW_SYS_VENDOR` |
| 342 | - Cloud/container: `AE_HOST_CLOUD_PROVIDER`, `AE_HOST_CONTAINER`, `AE_AGENT_KUBERNETES` |
| 343 | - Crash analytics: `AE_AGENT_CRASHES`, `AE_FATAL_FAULT_ADDRESS`, `AE_FATAL_THREAD` |
| 344 | - Performance: `AE_AGENT_UPTIME`, `AE_AGENT_TIMINGS_INIT`, `AE_AGENT_TIMINGS_EXIT` |
| 345 | |
| 346 | 3. **Other Structured Data** - Any application can log structured data: |
| 347 | - Netdata alerts: `ND_ALERT_NAME`, `ND_ALERT_STATUS`, `ND_ALERT_CLASS` |
| 348 | - Custom applications: Arbitrary fields specific to each application |
| 349 | - IoT devices: Sensor readings, device states, telemetry data |
| 350 | - Business applications: Transaction IDs, user actions, audit trails |
| 351 | |
| 352 | This variability means: |
| 353 | - The same logs query API must handle completely different schemas |
| 354 | - Fields available for filtering/faceting vary by dataset |
| 355 | - LLMs need to discover what fields exist before building meaningful queries |
| 356 | - Traditional fixed-schema approaches don't work |
| 357 | |
| 358 | #### Platform-Specific Schema Characteristics |
| 359 | |
| 360 | **Linux (systemd-journal)**: |
| 361 | - All fields are native journal fields - no lazy loading needed |
| 362 | - Can have **thousands** of different fields in a single dataset |
| 363 | - Multiple datasets can be queried in parallel (multiplexed, interleaved) |
| 364 | - Fields are discovered dynamically as data is processed |
| 365 | - Fast field access and filtering |
| 366 | - This massive scalability and flexibility makes systemd-journal extremely powerful as a structured data store |
| 367 | |
| 368 | **Windows (Event logs)**: |
| 369 | - Limited native fields (Level, TimeCreated, EventID, Provider, etc.) |
| 370 | - Rich data stored in XML format within each event |
| 371 | - Lazy XML parsing for performance: |
| 372 | - Full-text search scans XML without parsing |
| 373 | - XML parsing happens only for returned rows |
| 374 | - Balances search capability with performance constraints |
| 375 | - Field discovery requires XML inspection |
| 376 | |
| 377 | ### Future Direction |
| 378 | - Evolve MCP's function processing to handle both regular tables and logs uniformly |
| 379 | - Create MCP tools that: |
| 380 | - Support dynamic field discovery |
| 381 | - Enable faceted search and analysis |
| 382 | - Provide intelligent query building based on discovered schema |
| 383 | - Handle pagination and time-based navigation |
| 384 | - Leverage the full power of the facets engine |