| 1 | # Netdata Functions v3 Protocol - Technical Reference |
| 2 | |
| 3 | > **Note**: This is the technical specification. For a practical guide to implementing functions, see [Functions Developer Guide](/src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md). Topology Functions use the dedicated [Topology Function Schema](/src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md). |
| 4 | |
| 5 | ## Overview |
| 6 | |
| 7 | This document provides the complete technical reference for Netdata Functions protocol v3, combining all information about simple tables (has_history=false) and log explorers (has_history=true). This is the authoritative internal documentation for maintaining and extending Netdata functions. |
| 8 | |
| 9 | ## Table of Contents |
| 10 | |
| 11 | 1. [Protocol Overview](#protocol-overview) |
| 12 | 2. [Request Flow](#request-flow) |
| 13 | 3. [Simple Table Format](#simple-table-format) |
| 14 | 4. [Log Explorer Format](#log-explorer-format) |
| 15 | 5. [Field Types and Enumerations](#field-types-and-enumerations) |
| 16 | 6. [UI Implementation](#ui-implementation) |
| 17 | 7. [Backend Implementation](#backend-implementation) |
| 18 | 8. [Best Practices](#best-practices) |
| 19 | 9. [Known Functions](#known-functions) |
| 20 | 10. [Development Checklist](#development-checklist) |
| 21 | |
| 22 | ## Protocol Overview |
| 23 | |
| 24 | Netdata Functions allow collectors/plugins to expose interactive data through a streaming protocol. The protocol has evolved from GET-based CLI parameters (legacy) to POST-based JSON payloads (modern v3). |
| 25 | |
| 26 | ### Function Types |
| 27 | |
| 28 | 1. **Simple Table View** (`has_history: false`) |
| 29 | - Basic tabular data display with frontend-side filtering and search |
| 30 | - Examples: `processes`, `network-connections`, `block-devices` |
| 31 | |
| 32 | 2. **Log Explorer Format** (`has_history: true`) |
| 33 | - Advanced table with backend-powered faceted search, histograms, and infinite scroll |
| 34 | - Examples: `systemd-journal`, `windows-events` |
| 35 | |
| 36 | 3. **Topology Format** (`type: "topology"`) |
| 37 | - Compact graph payload with actors, graph links, relationship evidence, custom actor detail tables, correlation rules, and overlay references |
| 38 | - Uses [FUNCTION_TOPOLOGY_SCHEMA.json](/src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json) |
| 39 | - Examples: `topology:network-connections`, `topology:streaming`, `topology:snmp` |
| 40 | |
| 41 | ### Critical Implementation Differences |
| 42 | |
| 43 | | Aspect | Simple Tables | Log Explorers | |
| 44 | |--------|---------------|---------------| |
| 45 | | **Data Processing** | All data sent to frontend | Backend filters before sending | |
| 46 | | **Facet Counts** | Frontend counts occurrences in received data | Backend computes counts during query execution | |
| 47 | | **Full-Text Search** | Frontend substring search across visible data | Backend pattern matching with facets library | |
| 48 | | **Histograms** | Not supported - no time-based visualization | Optional - backend generates Netdata chart format | |
| 49 | | **Performance** | Limited by browser memory and processing | Scales to millions of records with sampling | |
| 50 | | **Query Parameter** | Ignored by backend (frontend only) | Processed by backend using simple patterns | |
| 51 | |
| 52 | ## Request Flow |
| 53 | |
| 54 | ### Modern Flow (v:3) |
| 55 | |
| 56 | 1. **Info Request** (Always GET) |
| 57 | ``` |
| 58 | GET /api/v3/function?function=systemd-journal info after:1234567890 before:1234567890 |
| 59 | ``` |
| 60 | |
| 61 | 2. **Info Response** |
| 62 | ```json |
| 63 | { |
| 64 | "v": 3, // Indicates POST should be used for data requests |
| 65 | "accepted_params": [...], |
| 66 | "required_params": [...], |
| 67 | "status": 200, |
| 68 | "type": "table", |
| 69 | "has_history": false |
| 70 | } |
| 71 | ``` |
| 72 | |
| 73 | 3. **Data Request** (POST when v=3) |
| 74 | ```json |
| 75 | { |
| 76 | "query": "*error* !*debug*", |
| 77 | "selections": { |
| 78 | "priority": ["error", "warning"], |
| 79 | "unit": ["nginx.service"] |
| 80 | }, |
| 81 | "after": 1234567890, |
| 82 | "before": 1234567890, |
| 83 | "last": 100 |
| 84 | } |
| 85 | ``` |
| 86 | |
| 87 | ### Preflight Info Request Details |
| 88 | |
| 89 | **Request Format:** |
| 90 | ``` |
| 91 | GET /api/v3/function?function=systemd-journal info after:1234567890 before:1234567890 |
| 92 | ``` |
| 93 | |
| 94 | **Required Response Fields:** |
| 95 | ```json |
| 96 | { |
| 97 | "v": 3, |
| 98 | "status": 200, |
| 99 | "type": "table", |
| 100 | "has_history": false, |
| 101 | "accepted_params": ["info", "after", "before", "direction", "last"], |
| 102 | "required_params": [ |
| 103 | { |
| 104 | "id": "priority", |
| 105 | "name": "Log Level", |
| 106 | "type": "select", |
| 107 | "options": [ |
| 108 | {"id": "error", "name": "Error", "defaultSelected": true}, |
| 109 | {"id": "warn", "name": "Warning"} |
| 110 | ] |
| 111 | } |
| 112 | ], |
| 113 | "help": "Function description" |
| 114 | } |
| 115 | ``` |
| 116 | |
| 117 | **Frontend Processing:** |
| 118 | - `accepted_params`: **drives outgoing payload** (only params in this list are sent). When facets are selected, filters are restricted to `accepted_params` plus required param IDs. |
| 119 | - `required_params`: Generates filter UI, prevents execution if missing |
| 120 | - `v: 3`: Enables POST requests with JSON payloads |
| 121 | - Missing required parameters show user-friendly error messages |
| 122 | |
| 123 | **Parameter shapes (wire format):** |
| 124 | - `accepted_params`: array of **strings** (parameter IDs). Example: `["sockets"]`, `["group"]`. |
| 125 | - `required_params`: array of **objects** that define UI selectors. |
| 126 | - Required fields: `id`, `name`, `type`, `options` |
| 127 | - Common fields: `help`, `unique_view` |
| 128 | - `options[]`: `id`, `name`, optional `defaultSelected`, `disabled`, `sort` |
| 129 | |
| 130 | **Cloud-frontend UI notes (verified):** |
| 131 | - `type: "select"` renders as a **single-select**. |
| 132 | - If no `defaultSelected`, UI selects the **first** option by default. |
| 133 | |
| 134 | ### Backend Implementation |
| 135 | |
| 136 | ```c |
| 137 | // Simplified from logs_query_status.h |
| 138 | if(payload) { |
| 139 | // POST request - parse JSON payload |
| 140 | facets_use_hashes_for_ids(facets, false); // Use plain field names |
| 141 | rq->fields_are_ids = false; |
| 142 | } else { |
| 143 | // GET request - parse CLI parameters (legacy) |
| 144 | facets_use_hashes_for_ids(facets, true); // Use hash IDs |
| 145 | rq->fields_are_ids = true; |
| 146 | } |
| 147 | ``` |
| 148 | |
| 149 | ### Key Differences |
| 150 | |
| 151 | | Aspect | GET (Legacy) | POST (Modern v3) | |
| 152 | |--------|--------------|------------------| |
| 153 | | Version | v \< 3 | v = 3 | |
| 154 | | Field IDs | 11-char hashes | Plain names | |
| 155 | | Parameters | URL encoded | JSON body | |
| 156 | | Facet filters | `field_hash:value1,value2` | `{"field": ["value1", "value2"]}` | |
| 157 | | Full-text search | `query:search terms` | `{"query": "search terms"}` | |
| 158 | |
| 159 | ### Standard Accepted Parameters |
| 160 | |
| 161 | **Core Parameters (All Functions):** |
| 162 | - `"info"` - Function info requests (always supported) |
| 163 | - `"after"` - Time range start (seconds epoch) |
| 164 | - `"before"` - Time range end (seconds epoch) |
| 165 | |
| 166 | **Log Explorer Parameters (has_history=true):** |
| 167 | - `"direction"` - Query direction: `"backward"` | `"forward"` |
| 168 | - `"last"` - Result limit (default: 200) |
| 169 | - `"anchor"` - Pagination cursor (timestamp or row identifier) |
| 170 | - `"query"` - Full-text search using Netdata simple patterns |
| 171 | - `"facets"` - Facet selection: `"field1,field2"` |
| 172 | - `"histogram"` - Histogram field selection |
| 173 | - `"if_modified_since"` - Conditional updates (microseconds epoch) |
| 174 | - `"data_only"` - Skip metadata in response (boolean) |
| 175 | - `"delta"` - Incremental responses (boolean) |
| 176 | - `"tail"` - Streaming mode (boolean) |
| 177 | - `"sampling"` - Data sampling control |
| 178 | |
| 179 | **UI Feature Parameters:** |
| 180 | - `"slice"` - Enables "Full data queries" toggle in UI |
| 181 | |
| 182 | **Frontend Behavior:** |
| 183 | ```javascript |
| 184 | // Only accepted parameters are sent to functions |
| 185 | const allowedFilterIds = [...selectedFacets, ...requiredParamIds, ...acceptedParams] |
| 186 | filtersToSend = allowedFilterIds.reduce((acc, filterId) => { |
| 187 | if (filterId in filters) acc[filterId] = filters[filterId] |
| 188 | return acc |
| 189 | }, {}) |
| 190 | ``` |
| 191 | |
| 192 | ### Query Parameter (Netdata Simple Patterns) |
| 193 | |
| 194 | The `query` parameter uses Netdata's simple pattern matching for full-text search across all fields: |
| 195 | |
| 196 | **Pattern Syntax:** |
| 197 | - `|` - Pattern separator (OR logic between patterns) |
| 198 | - `*` - Wildcard matching any number of characters |
| 199 | - `!` - Negates the pattern (exclude matches) |
| 200 | - Spaces are matched literally (not pattern separators) |
| 201 | - Case-insensitive matching |
| 202 | - Default behavior is substring matching (no wildcards needed) |
| 203 | |
| 204 | **Matching Modes:** |
| 205 | - `pattern` - Substring match (default) - finds "pattern" anywhere |
| 206 | - `*pattern` - Suffix match - finds strings ending with "pattern" |
| 207 | - `pattern*` - Prefix match - finds strings starting with "pattern" |
| 208 | - `*pattern*` - Substring match (explicit) - same as default |
| 209 | |
| 210 | **Examples:** |
| 211 | ```json |
| 212 | { |
| 213 | "query": "error" // Finds "error" anywhere (substring) |
| 214 | "query": "error|warning" // Finds "error" OR "warning" |
| 215 | "query": "error|warning|critical" // Multiple OR patterns |
| 216 | "query": "!debug" // Exclude ALL rows containing "debug" |
| 217 | "query": "!*debugging*|*debug*" // Include "debug" but exclude "debugging" |
| 218 | "query": "connection failed" // Find exact phrase (spaces included) |
| 219 | "query": "*error" // Find strings ending with "error" |
| 220 | "query": "nginx*" // Find strings starting with "nginx" |
| 221 | } |
| 222 | ``` |
| 223 | |
| 224 | **Pattern Evaluation Rules:** |
| 225 | |
| 226 | 1. **Within a field**: Left-to-right, first match wins |
| 227 | - `"!*debugging*|*debug*"` - If text contains "debugging", it's negative match. Otherwise, if contains "debug", it's positive match. |
| 228 | |
| 229 | 2. **Across all fields**: ALL fields are evaluated (no short-circuit) |
| 230 | - Every field with FTS enabled is checked against the pattern |
| 231 | - Positive and negative matches are counted separately |
| 232 | - Field evaluation order doesn't affect the outcome |
| 233 | |
| 234 | 3. **Row decision (after all fields evaluated)**: |
| 235 | - **Excluded if**: ANY field has a negative match (regardless of positive matches) |
| 236 | - **Included if**: At least one positive match AND zero negative matches |
| 237 | - **Excluded if**: No positive matches found |
| 238 | |
| 239 | **Example**: Query `"!*debugging*|*debug*"` |
| 240 | ``` |
| 241 | Row 1: message="debug info", category="debugging tips" |
| 242 | → message: positive match (debug) |
| 243 | → category: negative match (debugging) |
| 244 | → Result: EXCLUDED (has negative match) |
| 245 | |
| 246 | Row 2: message="debug info", category="testing" |
| 247 | → message: positive match (debug) |
| 248 | → category: no match |
| 249 | → Result: INCLUDED (positive match, no negative) |
| 250 | |
| 251 | Row 3: message="error info", category="testing" |
| 252 | → message: no match |
| 253 | → category: no match |
| 254 | → Result: EXCLUDED (no positive matches) |
| 255 | ``` |
| 256 | |
| 257 | **Key Point**: The order fields are evaluated doesn't matter - the same counters are updated and the same decision is made regardless of whether positive or negative matches are found first. |
| 258 | |
| 259 | **Common Use Cases:** |
| 260 | - `"timeout|failed|refused"` - Find various connection issues |
| 261 | - `"!trace|!debug|nginx|apache"` - Find web server logs, but exclude debug/trace |
| 262 | - `"error code 500"` - Find exact phrase with spaces |
| 263 | - `"critical|fatal|emergency"` - Find severe log levels |
| 264 | - `"!*test*|!*debug*|*"` - Include everything except test and debug content |
| 265 | |
| 266 | **Implementation Details:** |
| 267 | - Uses `simple_pattern_create(query, "|", SIMPLE_PATTERN_SUBSTRING, false)` |
| 268 | - Searches all fields marked with `FACET_KEY_OPTION_FTS` or when `FACETS_OPTION_ALL_KEYS_FTS` is set |
| 269 | - No support for `?` single-character wildcards |
| 270 | - Escaping with `\` is supported for literal matches |
| 271 | |
| 272 | **Important**: Hash IDs (like `priority_hash`) are obsolete and only exist for backward compatibility with GET requests. All new functions should use v:3 with plain field names. |
| 273 | |
| 274 | ## Simple Table Format |
| 275 | |
| 276 | ### Complete Structure |
| 277 | |
| 278 | ```json |
| 279 | { |
| 280 | // Required fields |
| 281 | "status": 200, |
| 282 | "type": "table", |
| 283 | "has_history": false, |
| 284 | "data": [ |
| 285 | [value1, value2, value3, ...], // Row 1 |
| 286 | [value1, value2, value3, ...], // Row 2 |
| 287 | // Special rowOptions as last element |
| 288 | [..., {"rowOptions": {"severity": "warning|error|notice|normal"}}] |
| 289 | ], |
| 290 | "columns": { |
| 291 | "column_name": { |
| 292 | // Required |
| 293 | "index": 0, // Position in data array |
| 294 | "name": "Display Name", // Column header |
| 295 | "type": "string", // Field type |
| 296 | |
| 297 | // Optional |
| 298 | "unique_key": false, // Row identifier (exactly one required) |
| 299 | "visible": true, // Default visibility |
| 300 | "sticky": false, // Pin when scrolling |
| 301 | "visualization": "value", // How to render |
| 302 | "value_options": { // Value formatting options |
| 303 | "units": "bytes", |
| 304 | "transform": "none", |
| 305 | "decimal_points": 2, |
| 306 | "default_value": "" |
| 307 | }, |
| 308 | "max": 100, // For bar types |
| 309 | "pointer_to": "col_id", // Optional reference target |
| 310 | "sort": "descending", // Default sort |
| 311 | "sortable": true, // User can sort |
| 312 | "filter": "multiselect", // Filter type |
| 313 | "full_width": false, // Expand to fill |
| 314 | "wrap": false, // Text wrapping |
| 315 | "default_expanded_filter": false, |
| 316 | "summary": "sum", // Aggregation (backend only) |
| 317 | "dummy": false // True for hidden/internal columns |
| 318 | } |
| 319 | }, |
| 320 | |
| 321 | // Optional extensions |
| 322 | "help": "Function description", |
| 323 | "update_every": 1, |
| 324 | "expires": 1234567890, |
| 325 | "default_sort_column": "column_name", |
| 326 | "group_by": { |
| 327 | "aggregated": [{ |
| 328 | "id": "group_id", |
| 329 | "name": "Group Name", |
| 330 | "column": "column_to_group_by" |
| 331 | }] |
| 332 | }, |
| 333 | "charts": { |
| 334 | "chart_id": { |
| 335 | "name": "Chart Name", |
| 336 | "type": "stacked|bar", |
| 337 | "columns": ["col1", "col2"] |
| 338 | } |
| 339 | }, |
| 340 | "accepted_params": ["param_id"], |
| 341 | "required_params": [ |
| 342 | { |
| 343 | "id": "param_id", |
| 344 | "name": "Parameter Name", |
| 345 | "type": "select", |
| 346 | "unique_view": true, |
| 347 | "options": [ |
| 348 | {"id": "opt1", "name": "Option 1", "defaultSelected": true}, |
| 349 | {"id": "opt2", "name": "Option 2"} |
| 350 | ] |
| 351 | } |
| 352 | ] |
| 353 | } |
| 354 | ``` |
| 355 | |
| 356 | ### Row Options |
| 357 | |
| 358 | Special last element in data array for row styling: |
| 359 | |
| 360 | ```json |
| 361 | [..., {"rowOptions": {"severity": "error"}}] // Red background |
| 362 | [..., {"rowOptions": {"severity": "warning"}}] // Yellow background |
| 363 | [..., {"rowOptions": {"severity": "notice"}}] // Blue background |
| 364 | [..., {"rowOptions": {"severity": "normal"}}] // Default appearance |
| 365 | ``` |
| 366 | |
| 367 | ## Log Explorer Format |
| 368 | |
| 369 | ### Complete Structure |
| 370 | |
| 371 | ```json |
| 372 | { |
| 373 | // Basic fields (same as simple table) |
| 374 | "status": 200, |
| 375 | "type": "table", |
| 376 | "has_history": true, // REQUIRED: Enables log explorer UI |
| 377 | "help": "System log explorer", |
| 378 | "update_every": 1, |
| 379 | |
| 380 | // Table metadata |
| 381 | "table": { |
| 382 | "id": "logs", |
| 383 | "has_history": true, |
| 384 | "pin_alert": false |
| 385 | }, |
| 386 | |
| 387 | // Faceted filters (dynamic with counts) |
| 388 | "facets": [ |
| 389 | { |
| 390 | "id": "priority", // Plain field name (not hash) |
| 391 | "name": "Priority", |
| 392 | "order": 1, |
| 393 | "options": [ |
| 394 | { |
| 395 | "id": "ERROR", |
| 396 | "name": "ERROR", |
| 397 | "count": 45, // Real-time count |
| 398 | "order": 1 |
| 399 | } |
| 400 | ] |
| 401 | } |
| 402 | ], |
| 403 | |
| 404 | // Enhanced columns |
| 405 | "columns": { |
| 406 | "timestamp": { |
| 407 | "index": 0, |
| 408 | "id": "timestamp", |
| 409 | "name": "Time", |
| 410 | "type": "timestamp", |
| 411 | "transform": "datetime_usec", |
| 412 | "sort": "descending|fixed", |
| 413 | "sortable": false, |
| 414 | "sticky": true |
| 415 | }, |
| 416 | "level": { |
| 417 | "index": 1, |
| 418 | "id": "priority", // Links to facet |
| 419 | "name": "Level", |
| 420 | "type": "string", |
| 421 | "visualization": "pill", |
| 422 | "filter": "facet", // Not multiselect! |
| 423 | "options": ["facet", "visible", "sticky"] |
| 424 | }, |
| 425 | "message": { |
| 426 | "index": 3, |
| 427 | "id": "message", |
| 428 | "name": "Message", |
| 429 | "type": "string", |
| 430 | "full_width": true, |
| 431 | "options": [ |
| 432 | "full_width", |
| 433 | "wrap", |
| 434 | "visible", |
| 435 | "main_text", // Primary content |
| 436 | "fts", // Full-text searchable |
| 437 | "rich_text" // May contain formatting |
| 438 | ] |
| 439 | } |
| 440 | }, |
| 441 | |
| 442 | // Data with microsecond timestamps |
| 443 | "data": [ |
| 444 | [ |
| 445 | 1697644320000000, // Microseconds |
| 446 | {"severity": "error"}, // rowOptions |
| 447 | "ERROR", // level |
| 448 | "nginx", // source |
| 449 | "Connection failed" // message |
| 450 | ] |
| 451 | ], |
| 452 | |
| 453 | // Histogram configuration |
| 454 | "available_histograms": [ |
| 455 | {"id": "priority", "name": "Priority", "order": 1}, |
| 456 | {"id": "source", "name": "Source", "order": 2} |
| 457 | ], |
| 458 | "histogram": { |
| 459 | "id": "priority", |
| 460 | "name": "Priority", |
| 461 | "chart": { |
| 462 | "summary": {/* Netdata chart metadata */}, |
| 463 | "result": { |
| 464 | "labels": ["time", "ERROR", "WARN", "INFO"], |
| 465 | "data": [ |
| 466 | [1697644200, 5, 12, 234], |
| 467 | [1697644260, 3, 8, 198] |
| 468 | ] |
| 469 | } |
| 470 | } |
| 471 | }, |
| 472 | |
| 473 | // Pagination metadata |
| 474 | "items": { |
| 475 | "evaluated": 50000, // Total scanned |
| 476 | "matched": 2520, // Match filters |
| 477 | "unsampled": 100, // Skipped (sampling) |
| 478 | "estimated": 0, // Statistical estimate |
| 479 | "returned": 100, // In this response |
| 480 | "max_to_return": 100, |
| 481 | "before": 0, |
| 482 | "after": 2420 |
| 483 | }, |
| 484 | |
| 485 | // Navigation anchor |
| 486 | "anchor": { |
| 487 | "last_modified": 1697644320000000, |
| 488 | "direction": "backward" // or "forward" |
| 489 | }, |
| 490 | |
| 491 | // Request echo (optional) |
| 492 | "request": { |
| 493 | "query": "*error* *warning*", |
| 494 | "filters": ["priority:error,warning"], |
| 495 | "histogram": "priority" |
| 496 | }, |
| 497 | |
| 498 | // Additional metadata |
| 499 | "expires": 1697644920000, |
| 500 | "sampling": 10 // 1 in N sampling |
| 501 | } |
| 502 | ``` |
| 503 | |
| 504 | ### Key Differences from Simple Tables |
| 505 | |
| 506 | | Feature | Simple Table | Log Explorer | |
| 507 | |---------|--------------|--------------| |
| 508 | | **Facet counts** | Frontend computes from data | Backend computes in facets library | |
| 509 | | **Full-text search** | Frontend substring matching | Backend pattern matching | |
| 510 | | **Histograms** | Not supported | Optional (backend generated) | |
| 511 | | Filtering | Static multiselect | Dynamic facets with counts | |
| 512 | | Pagination | All data at once | Anchor-based infinite scroll | |
| 513 | | Time visualization | None | Histogram chart | |
| 514 | | Navigation | None | Bi-directional with timestamps | |
| 515 | | Performance | All data loaded | Sampling for large datasets | |
| 516 | |
| 517 | ### Log-Specific Column Options |
| 518 | |
| 519 | | Option | UI Effect | |
| 520 | |--------|-----------| |
| 521 | | `"facet"` | Field is filterable via facets | |
| 522 | | `"fts"` | Full-text searchable | |
| 523 | | `"main_text"` | Primary content field | |
| 524 | | `"rich_text"` | May contain formatting | |
| 525 | | `"pretty_xml"` | Format as XML | |
| 526 | | `"hidden"` | Hide by default | |
| 527 | |
| 528 | ## Facet Value Pills and Aggregated Counts |
| 529 | |
| 530 | ### Overview |
| 531 | |
| 532 | Facet values in the sidebar display pills with counts that change based on the function's aggregation mode. The UI uses a smart component that displays different formats: |
| 533 | |
| 534 | 1. **Simple Counts**: Just the number of matching rows |
| 535 | 2. **Aggregated Counts**: Shows both aggregated count and original count with a union symbol |
| 536 | |
| 537 | ### UI Implementation |
| 538 | |
| 539 | The pills are rendered using this logic: |
| 540 | |
| 541 | ```javascript |
| 542 | {!!actualCount && <TextSmall>{actualCount} ⊃ </TextSmall>} |
| 543 | <TextSmall>{(pill || count).toString()}</TextSmall> |
| 544 | ``` |
| 545 | |
| 546 | **Symbol**: `⊃` renders as `⊃` (superset symbol, looks like rotated 'u') |
| 547 | |
| 548 | **Display Formats**: |
| 549 | - Simple: `"42"` (just the count) |
| 550 | - Aggregated: `"15 ⊃ 42"` (15 aggregated items containing 42 total) |
| 551 | |
| 552 | ### Backend Configuration |
| 553 | |
| 554 | Functions enable aggregated counts by including an `aggregated_view` object in their response: |
| 555 | |
| 556 | ```json |
| 557 | { |
| 558 | "aggregated_view": { |
| 559 | "column": "Count", |
| 560 | "results_label": "unique combinations", |
| 561 | "aggregated_label": "sockets" |
| 562 | } |
| 563 | } |
| 564 | ``` |
| 565 | |
| 566 | **Example from network-connections function**: |
| 567 | ```c |
| 568 | // In network-viewer.c when aggregated=true |
| 569 | buffer_json_member_add_object(wb, "aggregated_view"); |
| 570 | { |
| 571 | buffer_json_member_add_string(wb, "column", "Count"); |
| 572 | buffer_json_member_add_string(wb, "results_label", "unique combinations"); |
| 573 | buffer_json_member_add_string(wb, "aggregated_label", "sockets"); |
| 574 | } |
| 575 | buffer_json_object_close(wb); |
| 576 | ``` |
| 577 | |
| 578 | ## Charts Configuration |
| 579 | |
| 580 | Functions can provide both standard charts (computed by frontend) and custom visualizations. |
| 581 | |
| 582 | ### Standard Charts |
| 583 | |
| 584 | **Configuration:** |
| 585 | ```json |
| 586 | { |
| 587 | "charts": { |
| 588 | "cpu_usage": { |
| 589 | "name": "CPU Usage by Service", |
| 590 | "type": "stacked-bar", |
| 591 | "columns": ["user_cpu", "system_cpu"], |
| 592 | "groupBy": "column", |
| 593 | "aggregation": "sum" |
| 594 | } |
| 595 | }, |
| 596 | "default_charts": [ |
| 597 | ["cpu_usage", "service_type"] |
| 598 | ] |
| 599 | } |
| 600 | ``` |
| 601 | |
| 602 | **Supported Types:** |
| 603 | - `"bar"` - Basic bar chart |
| 604 | - `"stacked-bar"` - Multi-column stacked bars |
| 605 | - `"doughnut"` - Pie/doughnut chart |
| 606 | - `"value"` - Simple numeric display |
| 607 | |
| 608 | **GroupBy Options:** |
| 609 | - `"column"` (default) - Group by selected filter column |
| 610 | - `"all"` - Aggregate all data together |
| 611 | |
| 612 | ### Custom Charts |
| 613 | |
| 614 | For specialized visualizations beyond standard chart types, some functions may use predefined custom chart types: |
| 615 | |
| 616 | ```json |
| 617 | { |
| 618 | "customCharts": { |
| 619 | "network_topology": { |
| 620 | "type": "network-viewer", |
| 621 | "config": { |
| 622 | "layout": "force", |
| 623 | "showLabels": true |
| 624 | } |
| 625 | } |
| 626 | } |
| 627 | } |
| 628 | ``` |
| 629 | |
| 630 | **Available Custom Types:** |
| 631 | - `"network-viewer"` - Interactive network topology (for network-connections function) |
| 632 | |
| 633 | ### Frontend Processing |
| 634 | |
| 635 | Charts are computed from table data: |
| 636 | 1. Frontend groups data by selected column |
| 637 | 2. Applies aggregation function (sum, count, avg, etc.) |
| 638 | 3. Renders chart with grouped results |
| 639 | |
| 640 | ### Table Row Grouping |
| 641 | |
| 642 | Simple tables support grouping rows with backend-defined aggregation rules. |
| 643 | |
| 644 | **Backend Summary Types:** |
| 645 | ```c |
| 646 | typedef enum { |
| 647 | RRDF_FIELD_SUMMARY_COUNT, // Count rows in group |
| 648 | RRDF_FIELD_SUMMARY_UNIQUECOUNT, // Count unique values |
| 649 | RRDF_FIELD_SUMMARY_SUM, // Sum numeric values |
| 650 | RRDF_FIELD_SUMMARY_MIN, // Minimum value |
| 651 | RRDF_FIELD_SUMMARY_MAX, // Maximum value |
| 652 | RRDF_FIELD_SUMMARY_MEAN, // Average value |
| 653 | RRDF_FIELD_SUMMARY_MEDIAN, // Median value |
| 654 | } RRDF_FIELD_SUMMARY; |
| 655 | ``` |
| 656 | |
| 657 | **Column Summary Configuration:** |
| 658 | ```c |
| 659 | buffer_rrdf_table_add_field( |
| 660 | wb, field_id++, "cpu", "CPU Usage", |
| 661 | RRDF_FIELD_TYPE_INTEGER, |
| 662 | RRDF_FIELD_VISUAL_VALUE, |
| 663 | RRDF_FIELD_TRANSFORM_NUMBER, |
| 664 | 2, "%", NAN, RRDF_FIELD_SORT_DESCENDING, NULL, |
| 665 | RRDF_FIELD_SUMMARY_SUM, // How to aggregate when grouping |
| 666 | RRDF_FIELD_FILTER_RANGE, |
| 667 | RRDF_FIELD_OPTS_VISIBLE, NULL |
| 668 | ); |
| 669 | ``` |
| 670 | |
| 671 | **Group By Support:** |
| 672 | ```json |
| 673 | { |
| 674 | "group_by": { |
| 675 | "aggregated": [{ |
| 676 | "id": "by_status", |
| 677 | "name": "By Status", |
| 678 | "column": "status" |
| 679 | }] |
| 680 | } |
| 681 | } |
| 682 | ``` |
| 683 | |
| 684 | ### Frontend Processing |
| 685 | |
| 686 | The frontend processes table data to generate facet counts: |
| 687 | |
| 688 | ```javascript |
| 689 | const getFilterTableOptions = (data, { param, columns, aggregatedView } = {}) => |
| 690 | Object.entries( |
| 691 | data.reduce((h, fn) => { |
| 692 | h[fn[param]] = { |
| 693 | count: (h[fn[param]]?.count || 0) + (fn.hidden ? 0 : 1), |
| 694 | ...(aggregatedView && { |
| 695 | actualCount: (h[fn[param]]?.actualCount || 0) + |
| 696 | (fn.hidden ? 0 : fn[aggregatedView.column] || 1), |
| 697 | actualCountLabel: aggregatedView.aggregatedLabel, |
| 698 | countLabel: aggregatedView.resultsLabel, |
| 699 | }), |
| 700 | } |
| 701 | return h |
| 702 | }, {}) |
| 703 | ).map(([id, values]) => ({ id, ...values })) |
| 704 | ``` |
| 705 | |
| 706 | ### How It Works |
| 707 | |
| 708 | The aggregated count system uses an existing data column to track how many items were aggregated into each row: |
| 709 | |
| 710 | 1. **Backend declares** which column contains the aggregation count: |
| 711 | ```json |
| 712 | "aggregated_view": { |
| 713 | "column": "Count" // Use the "Count" column from data rows |
| 714 | } |
| 715 | ``` |
| 716 | |
| 717 | 2. **Frontend calculates** two values for each facet value: |
| 718 | - **`count`**: How many rows have this facet value |
| 719 | - **`actualCount`**: Sum of the aggregation column for all rows with this facet value |
| 720 | |
| 721 | 3. **Example**: Network connections aggregated by protocol |
| 722 | |
| 723 | **Data returned by backend:** |
| 724 | ``` |
| 725 | Direction | Protocol | LocalPort | RemotePort | Count |
| 726 | ---------|----------|-----------|------------|------- |
| 727 | Inbound | TCP | * | * | 5 |
| 728 | Inbound | UDP | * | * | 3 |
| 729 | Outbound | TCP | * | * | 7 |
| 730 | Outbound | UDP | * | * | 2 |
| 731 | ``` |
| 732 | |
| 733 | **Facet pills displayed in UI:** |
| 734 | - Direction facet: |
| 735 | - Inbound: `8 ⊃ 2` (8 connections aggregated into 2 rows) |
| 736 | - Outbound: `9 ⊃ 2` (9 connections aggregated into 2 rows) |
| 737 | - Protocol facet: |
| 738 | - TCP: `12 ⊃ 2` (12 connections aggregated into 2 rows) |
| 739 | - UDP: `5 ⊃ 2` (5 connections aggregated into 2 rows) |
| 740 | |
| 741 | **Reading the pills**: `12 ⊃ 2` means "12 original items shown in 2 table rows" |
| 742 | |
| 743 | ### Tooltip Content |
| 744 | |
| 745 | The tooltip provides human-readable context: |
| 746 | - Simple mode: `"42 results"` |
| 747 | - Aggregated mode: `"15 sockets aggregated in 42 unique combinations"` |
| 748 | |
| 749 | ### Current Implementations |
| 750 | |
| 751 | | Function | Aggregated Mode | Trigger | Count Meaning | |
| 752 | |----------|----------------|---------|---------------| |
| 753 | | `network-connections` | `sockets:aggregated` | Parameter | Sockets → unique combinations | |
| 754 | | Other functions | N/A | None currently | Single count only | |
| 755 | |
| 756 | ### Adding Aggregated Counts |
| 757 | |
| 758 | To add aggregated count support to a function: |
| 759 | |
| 760 | 1. **Backend**: Add `aggregated_view` object to response when in aggregated mode |
| 761 | 2. **Data**: Include aggregation column with numeric values |
| 762 | 3. **Frontend**: No changes needed - automatically processes based on `aggregated_view` presence |
| 763 | |
| 764 | ## Field Types and Enumerations |
| 765 | |
| 766 | ### Field Types (RRDF_FIELD_TYPE_*) |
| 767 | |
| 768 | #### Implemented in UI |
| 769 | |
| 770 | | Type | UI Component | Use Case | Notes | |
| 771 | |------|--------------|----------|-------| |
| 772 | | `string` | ValueCell | Text, names, categories | Left-aligned | |
| 773 | | `integer` | ValueCell | Numbers, counts, IDs | Right-aligned | |
| 774 | | `bar-with-integer` | BarCell | Percentages, metrics | Requires `max` | |
| 775 | | `duration` | BarCell | Time intervals | Auto-formats seconds | |
| 776 | | `timestamp` | DatetimeCell* | Date/time points | *UI maps to datetime | |
| 777 | | `feedTemplate` | FeedTemplateCell | Rich content | Auto full_width | |
| 778 | |
| 779 | #### Fallback Implementation |
| 780 | |
| 781 | | Type | Behavior | Notes | |
| 782 | |------|----------|-------| |
| 783 | | `boolean` | ValueCell | No special boolean UI | |
| 784 | | `float` | ValueCell | Go functions may emit floats; UI falls back to default | |
| 785 | | `detail-string` | ValueCell | No expandable functionality | |
| 786 | | `array` | ValueCell | Works with `pill` visualization | |
| 787 | | `none` | ValueCell | Avoid using | |
| 788 | |
| 789 | ### Visual Types (RRDF_FIELD_VISUAL_*) |
| 790 | |
| 791 | | Type | UI Component | Use Case | |
| 792 | |------|--------------|----------| |
| 793 | | `value` | ValueCell | Standard text display (default) | |
| 794 | | `bar` | BarCell | Progress bar without text | |
| 795 | | `pill` | PillCell | Badge/tag display | |
| 796 | | `richValue` | RichValueCell | Enhanced value display | |
| 797 | | `feedTemplate` | FeedTemplateCell | Full-width template | |
| 798 | | `rowOptions` | null | Special row configuration | |
| 799 | |
| 800 | **Fallback Behavior**: `gauge` is not a recognized visualization. If used, it is ignored, and the renderer falls back to using the field's `type` to select a component. |
| 801 | |
| 802 | ### Transform Types (RRDF_FIELD_TRANSFORM_*) |
| 803 | |
| 804 | **JSON path:** `columns[*].value_options.transform` (and `columns[*].value_options.decimal_points` for numeric formatting). |
| 805 | |
| 806 | | Type | Input | Output | Notes | |
| 807 | |------|-------|--------|-------| |
| 808 | | `none` | Any | Unchanged | Default | |
| 809 | | `number` | Number | Formatted with decimals | Uses `decimal_points` | |
| 810 | | `duration` | Seconds | "Xd Yh Zm" | Human-readable | |
| 811 | | `datetime` | Epoch ms | Localized date/time | | |
| 812 | | `datetime_usec` | Epoch μs | Localized date/time | For logs | |
| 813 | | `xml` | XML string | Formatted XML | No specialized UI | |
| 814 | | `text` | Any | Unchanged | UI falls back to default | |
| 815 | |
| 816 | ### Conditional Patterns and Dependencies |
| 817 | |
| 818 | #### Type and Transform Compatibility |
| 819 | |
| 820 | Not all `transform` values are compatible with all `type` values. The backend enforces the following compatibility rules: |
| 821 | |
| 822 | | Field Type (`type`) | Compatible Transforms (`transform`) | |
| 823 | |---|---| |
| 824 | | `timestamp` | `datetime_ms`, `datetime_usec` | |
| 825 | | `duration` | `duration_s` | |
| 826 | | `integer`, `bar-with-integer` | `number` | |
| 827 | | `string`, `boolean`, `array` | `none`, `xml` | |
| 828 | |
| 829 | Using an incompatible transform will result in unexpected behavior or errors. |
| 830 | |
| 831 | #### `rowOptions` Dummy Column |
| 832 | |
| 833 | To add `rowOptions` for row-level severity styling, a special "dummy" column must be added to the `columns` definition. This column is not displayed in the UI but provides the necessary metadata. It must be created with this specific combination of values: |
| 834 | |
| 835 | * **type**: `none` (`RRDF_FIELD_TYPE_NONE`) |
| 836 | * **visualization**: `rowOptions` (`RRDF_FIELD_VISUAL_ROW_OPTIONS`) |
| 837 | * **options flag**: `dummy` (`RRDF_FIELD_OPTS_DUMMY`) |
| 838 | |
| 839 | **Example C code:** |
| 840 | ```c |
| 841 | buffer_rrdf_table_add_field(wb, field_id++, "row_options", "Row Options", |
| 842 | RRDF_FIELD_TYPE_NONE, RRDF_FIELD_VISUAL_ROW_OPTIONS, RRDF_FIELD_TRANSFORM_NONE, |
| 843 | 0, NULL, NAN, RRDF_FIELD_SORT_ASCENDING, NULL, |
| 844 | RRDF_FIELD_SUMMARY_COUNT, RRDF_FIELD_FILTER_NONE, RRDF_FIELD_OPTS_DUMMY, NULL); |
| 845 | ``` |
| 846 | |
| 847 | ### Filter Types (RRDF_FIELD_FILTER_*) |
| 848 | |
| 849 | | Type | UI Component | Use Case | Location | |
| 850 | |------|--------------|----------|----------| |
| 851 | | `multiselect` | Checkboxes | Column filtering (default) | Dynamic filters | |
| 852 | | `range` | RangeFilter | Numeric min/max | Dynamic filters | |
| 853 | | `facet` | Facets component | With counts (logs) | Sidebar | |
| 854 | |
| 855 | ### Field Options (RRDF_FIELD_OPTS_*) |
| 856 | |
| 857 | | Option | Bit | UI Effect | |
| 858 | |--------|-----|-----------| |
| 859 | | `unique_key` | 0x01 | Row identifier (one required) | |
| 860 | | `visible` | 0x02 | Show by default | |
| 861 | | `sticky` | 0x04 | Pin column when scrolling | |
| 862 | | `full_width` | 0x08 | Expand to fill space | |
| 863 | | `wrap` | 0x10 | Enable text wrapping | |
| 864 | | `dummy` | 0x20 | Internal use only | |
| 865 | | `default_expanded_filter` | 0x40 | Expand filter by default | |
| 866 | |
| 867 | ### Sort Options (RRDF_FIELD_SORT_*) |
| 868 | |
| 869 | - `ascending` - Sort low to high |
| 870 | - `descending` - Sort high to low |
| 871 | - Fixed sort (0x80) - Prevent user sorting |
| 872 | |
| 873 | ### Summary Types (RRDF_FIELD_SUMMARY_*) |
| 874 | |
| 875 | Backend calculates but UI doesn't display directly: |
| 876 | - `count`, `sum`, `min`, `max`, `mean`, `median` |
| 877 | - `uniqueCount` - Number of unique values |
| 878 | - `extent` - Range [min, max] (UI supported) |
| 879 | - `unique` - List of unique values (UI supported) |
| 880 | |
| 881 | ## UI Implementation |
| 882 | |
| 883 | ### Column Width System |
| 884 | |
| 885 | The UI automatically sizes columns based on metadata: |
| 886 | |
| 887 | | Size | Pixels | Applied To | |
| 888 | |------|--------|------------| |
| 889 | | xxxs | 90px | unique_key fields, bar types | |
| 890 | | xxs | 110px | Default for most types | |
| 891 | | xs | 130px | Available but rarely used | |
| 892 | | sm | 160px | timestamp, datetime types | |
| 893 | | md-xl | 190-290px | Available but rarely used | |
| 894 | | xxl | 1000px | feedTemplate with full_width | |
| 895 | |
| 896 | **Algorithm**: |
| 897 | 1. If `full_width`: → xxl with expansion |
| 898 | 2. If `unique_key`: → xxxs |
| 899 | 3. By visualization: bar → xxxs |
| 900 | 4. By type: feedTemplate → xxl, timestamp → sm |
| 901 | 5. Default → xxs |
| 902 | |
| 903 | ### Component Mapping |
| 904 | |
| 905 | ```javascript |
| 906 | // Field type → Component |
| 907 | componentByType = { |
| 908 | "bar": BarCell, |
| 909 | "bar-with-integer": BarCell, |
| 910 | "duration": BarCell, |
| 911 | "pill": PillCell, |
| 912 | "feedTemplate": FeedTemplateCell, |
| 913 | "datetime": DatetimeCell, |
| 914 | // Others → ValueCell |
| 915 | } |
| 916 | |
| 917 | // Visualization → Component |
| 918 | componentByVisualization = { |
| 919 | "bar": BarCell, |
| 920 | "pill": PillCell, |
| 921 | "richValue": RichValueCell, |
| 922 | "feedTemplate": FeedTemplateCell, |
| 923 | "rowOptions": null, // Skip rendering |
| 924 | // Others → ValueCell |
| 925 | } |
| 926 | ``` |
| 927 | |
| 928 | ### UI Component Architecture |
| 929 | |
| 930 | The frontend uses a modular architecture with: |
| 931 | - **Value Components**: Handle different field type rendering |
| 932 | - **Table Normalizer**: Processes function responses into UI-ready format |
| 933 | - **Filter Components**: Implement multiselect, range, and facet filtering |
| 934 | - **Chart Components**: Standard chart rendering (bar, stacked-bar, doughnut) |
| 935 | - **Custom Visualizations**: Extensible system for specialized charts |
| 936 | |
| 937 | ## Backend Implementation |
| 938 | |
| 939 | ### Key Functions |
| 940 | |
| 941 | ```c |
| 942 | // Add a field to the table |
| 943 | buffer_rrdf_table_add_field( |
| 944 | BUFFER *wb, |
| 945 | size_t field_id, |
| 946 | const char *key, |
| 947 | const char *name, |
| 948 | RRDF_FIELD_TYPE type, |
| 949 | RRDF_FIELD_VISUAL visual, |
| 950 | RRDF_FIELD_TRANSFORM transform, |
| 951 | size_t decimal_points, |
| 952 | const char *units, |
| 953 | NETDATA_DOUBLE max, |
| 954 | RRDF_FIELD_SORT sort, |
| 955 | const char *pointer_to_dim_in_rrdr, |
| 956 | RRDF_FIELD_SUMMARY summary, |
| 957 | RRDF_FIELD_FILTER filter, |
| 958 | RRDF_FIELD_OPTS options, |
| 959 | const char *default_value |
| 960 | ); |
| 961 | ``` |
| 962 | |
| 963 | ### Real Examples |
| 964 | |
| 965 | ```c |
| 966 | // CPU usage with progress bar |
| 967 | buffer_rrdf_table_add_field( |
| 968 | wb, field_id++, "CPU", "CPU %", |
| 969 | RRDF_FIELD_TYPE_BAR_WITH_INTEGER, |
| 970 | RRDF_FIELD_VISUAL_BAR, |
| 971 | RRDF_FIELD_TRANSFORM_NUMBER, |
| 972 | 2, "%", 100.0, RRDF_FIELD_SORT_DESCENDING, NULL, |
| 973 | RRDF_FIELD_SUMMARY_SUM, |
| 974 | RRDF_FIELD_FILTER_RANGE, |
| 975 | RRDF_FIELD_OPTS_VISIBLE, NULL |
| 976 | ); |
| 977 | |
| 978 | // Process name (unique key) |
| 979 | buffer_rrdf_table_add_field( |
| 980 | wb, field_id++, "Name", "Name", |
| 981 | RRDF_FIELD_TYPE_STRING, |
| 982 | RRDF_FIELD_VISUAL_VALUE, |
| 983 | RRDF_FIELD_TRANSFORM_NONE, |
| 984 | 0, NULL, NAN, RRDF_FIELD_SORT_ASCENDING, NULL, |
| 985 | RRDF_FIELD_SUMMARY_COUNT, |
| 986 | RRDF_FIELD_FILTER_MULTISELECT, |
| 987 | RRDF_FIELD_OPTS_VISIBLE | RRDF_FIELD_OPTS_STICKY | RRDF_FIELD_OPTS_UNIQUE_KEY, |
| 988 | NULL |
| 989 | ); |
| 990 | |
| 991 | // Row options in data |
| 992 | buffer_json_add_array_item_object(wb); |
| 993 | buffer_json_member_add_object(wb, "rowOptions"); |
| 994 | buffer_json_member_add_string(wb, "severity", "error"); |
| 995 | buffer_json_object_close(wb); |
| 996 | buffer_json_object_close(wb); |
| 997 | ``` |
| 998 | |
| 999 | ### Key Backend Files |
| 1000 | |
| 1001 | - **Enums**: `netdata/src/libnetdata/buffer/functions_fields.h` |
| 1002 | - **Implementation**: `netdata/src/libnetdata/buffer/functions_fields.c` |
| 1003 | - **Facets Library**: `netdata/src/libnetdata/facets/` |
| 1004 | - **Example Functions**: |
| 1005 | - `apps.plugin/apps_functions.c` |
| 1006 | - `network-viewer.plugin/network-viewer.c` |
| 1007 | - `systemd-journal.plugin/systemd-journal.c` |
| 1008 | |
| 1009 | ## Best Practices |
| 1010 | |
| 1011 | ### Always Include |
| 1012 | - One field with `unique_key` option |
| 1013 | - Meaningful `name` for display |
| 1014 | - Appropriate `type` for the data |
| 1015 | - Info response with `"v": 3` |
| 1016 | |
| 1017 | ### For Numeric Data |
| 1018 | - Use `bar-with-integer` for percentages |
| 1019 | - Set appropriate `max` value |
| 1020 | - Include `units` for clarity |
| 1021 | - Use `range` filter |
| 1022 | |
| 1023 | ### For Time Data |
| 1024 | - `duration` type for intervals |
| 1025 | - `timestamp` type for points in time |
| 1026 | - Use appropriate transform |
| 1027 | |
| 1028 | ### For Status/Severity |
| 1029 | - Include `rowOptions` as last array element |
| 1030 | - Use standard severity levels: error, warning, notice, normal |
| 1031 | |
| 1032 | ### For Filtering |
| 1033 | - `multiselect` (default) for categories |
| 1034 | - `range` for numeric data |
| 1035 | - `facet` only for has_history=true |
| 1036 | |
| 1037 | ### Common Patterns |
| 1038 | - Numeric metrics → `bar-with-integer` + `range` filter |
| 1039 | - Categories → `string` + `multiselect` filter |
| 1040 | - Time intervals → `duration` + `duration` transform |
| 1041 | - Status → `string` + `pill` visualization |
| 1042 | - Row coloring → `rowOptions` with severity |
| 1043 | |
| 1044 | ## Known Functions |
| 1045 | |
| 1046 | ### Simple Table Functions (has_history=false) |
| 1047 | |
| 1048 | | Function | Plugin | Key Features | |
| 1049 | |----------|--------|--------------| |
| 1050 | | processes | apps.plugin | CPU/memory bars, grouping | |
| 1051 | | socket | ebpf.plugin | Complex filters, charts | |
| 1052 | | network-connections | network-viewer.plugin | Aggregated views, severity | |
| 1053 | | systemd-list-units | systemd-units.plugin | Unit status, severity | |
| 1054 | | ipmi-sensors | freeipmi.plugin | Hardware monitoring | |
| 1055 | | block-devices | proc.plugin | I/O statistics, charts | |
| 1056 | | network-interfaces | proc.plugin | Network stats, severity | |
| 1057 | | mount-points | diskspace.plugin | Filesystem usage | |
| 1058 | | cgroup-top | cgroups.plugin | Container metrics | |
| 1059 | | systemd-top | cgroups.plugin | Service metrics | |
| 1060 | | metrics-cardinality | web api | Dynamic columns | |
| 1061 | | streaming | web api | Replication status | |
| 1062 | | all-queries | web api | Monitors the progress of in-flight queries | |
| 1063 | |
| 1064 | ### Log Explorer Functions (has_history=true) |
| 1065 | |
| 1066 | | Function | Plugin | Key Features | |
| 1067 | |----------|--------|--------------| |
| 1068 | | systemd-journal | systemd-journal.plugin | System logs, faceted search | |
| 1069 | | windows-events | windows-events.plugin | Windows logs, faceted search | |
| 1070 | |
| 1071 | |
| 1072 | ## Development Checklist |
| 1073 | |
| 1074 | ### Creating a New Function |
| 1075 | |
| 1076 | - [ ] Choose function type (simple table or log explorer) |
| 1077 | - [ ] Implement info response with `"v": 3` |
| 1078 | - [ ] Define columns with appropriate types and options |
| 1079 | - [ ] Include one `unique_key` field |
| 1080 | - [ ] Add proper error handling |
| 1081 | - [ ] Test with POST requests |
| 1082 | - [ ] Document accepted/required parameters |
| 1083 | |
| 1084 | ### Format Validation |
| 1085 | |
| 1086 | - [ ] Verify JSON structure matches specification |
| 1087 | - [ ] Check all required fields are present |
| 1088 | - [ ] Test empty result sets |
| 1089 | - [ ] Test large datasets |
| 1090 | - [ ] Verify error responses |
| 1091 | - [ ] Test special characters in data |
| 1092 | - [ ] Check numeric precision |
| 1093 | - [ ] Verify date/time formatting |
| 1094 | - [ ] Test sorting and filtering |
| 1095 | |
| 1096 | ### UI Integration |
| 1097 | |
| 1098 | - [ ] Confirm field types map to UI components |
| 1099 | - [ ] Verify filters work correctly |
| 1100 | - [ ] Check column widths display properly |
| 1101 | - [ ] Test row coloring with severity |
| 1102 | - [ ] Verify transforms apply correctly |
| 1103 | - [ ] Check responsive behavior |
| 1104 | |
| 1105 | ### Performance |
| 1106 | |
| 1107 | - [ ] Handle large datasets efficiently |
| 1108 | - [ ] Implement sampling for logs if needed |
| 1109 | - [ ] Set appropriate `update_every` |
| 1110 | - [ ] Consider pagination/anchoring |
| 1111 | - [ ] Test with concurrent requests |
| 1112 | |
| 1113 | ## Corner Cases and Edge Handling |
| 1114 | |
| 1115 | ### Empty Result Sets |
| 1116 | |
| 1117 | The protocol handles empty results gracefully: |
| 1118 | |
| 1119 | ```json |
| 1120 | { |
| 1121 | "status": 200, |
| 1122 | "type": "table", |
| 1123 | "has_history": false, |
| 1124 | "columns": {...}, // Full column definitions |
| 1125 | "data": [] // Empty array is valid |
| 1126 | } |
| 1127 | ``` |
| 1128 | |
| 1129 | - Minimum valid response: `{"status": 200, "type": "table", "columns": {}, "data": []}` |
| 1130 | - UI displays "No data available" message |
| 1131 | - Column headers still render for context |
| 1132 | |
| 1133 | ### Null and Missing Values |
| 1134 | |
| 1135 | - `null` values in data arrays are rendered as empty cells |
| 1136 | - Missing array elements default to `null` |
| 1137 | - `NaN` for numeric fields: For fields with `transform: "number"`, `NaN` values will be displayed as the string "NaN". For `timestamp` fields with `datetime` or `datetime_usec` transforms, `NaN` epoch values will display as empty cells. |
| 1138 | - Empty strings render as empty cells |
| 1139 | - Backend uses `NAN` constant for missing numeric values |
| 1140 | |
| 1141 | ### Special Characters |
| 1142 | |
| 1143 | The protocol properly escapes: |
| 1144 | - JSON special characters (`"`, `\`, control chars) |
| 1145 | - HTML entities (escaped by React components): React automatically escapes HTML content to prevent XSS attacks. Any HTML tags or entities in the data will be displayed as literal text, not rendered as HTML. |
| 1146 | - Unicode characters (UTF-8 support throughout) |
| 1147 | - SQL injection protection in queries |
| 1148 | |
| 1149 | ### Numeric Precision |
| 1150 | |
| 1151 | - `decimal_points` field controls display precision, using JavaScript's `toFixed()` method. |
| 1152 | - Backend uses `NETDATA_DOUBLE` type for floating-point numbers. |
| 1153 | - Frontend's number formatting: |
| 1154 | - Some components use `Intl.NumberFormat` with a fixed locale (e.g., "en-US"). |
| 1155 | - Others use `toLocaleString()` which respects the browser's locale. |
| 1156 | - Therefore, locale-specific formatting is applied in some, but not all, cases. |
| 1157 | - Very large numbers: Due to the use of `toFixed()`, very large numbers will be displayed as a long string with the specified decimal places, not automatically in scientific notation. |
| 1158 | - Infinity/NaN handling: |
| 1159 | - `NaN` values in numeric fields will be displayed as the string "NaN". |
| 1160 | - `Infinity` and `-Infinity` values will be displayed as the strings "Infinity" and "-Infinity" respectively. |
| 1161 | - For `timestamp` fields with `datetime` or `datetime_usec` transforms, `NaN` epoch values will display as empty cells. |
| 1162 | |
| 1163 | ### Timestamp Format Requirements |
| 1164 | |
| 1165 | **Simple Tables**: Use milliseconds with `datetime` transform |
| 1166 | ```json |
| 1167 | { |
| 1168 | "type": "timestamp", |
| 1169 | "transform": "datetime", // Expects milliseconds |
| 1170 | "data": [1697644320000] // JavaScript Date format |
| 1171 | } |
| 1172 | ``` |
| 1173 | |
| 1174 | **Log Explorers**: Use microseconds with `datetime_usec` transform |
| 1175 | ```json |
| 1176 | { |
| 1177 | "type": "timestamp", |
| 1178 | "transform": "datetime_usec", // Expects microseconds |
| 1179 | "data": [1697644320000000] // Microsecond precision |
| 1180 | } |
| 1181 | ``` |
| 1182 | |
| 1183 | **Frontend Conversion:** |
| 1184 | ```javascript |
| 1185 | // datetime_usec automatically converts to milliseconds |
| 1186 | if (usec) { |
| 1187 | epoch = epoch ? Math.floor(epoch / 1000) : epoch |
| 1188 | } |
| 1189 | ``` |
| 1190 | |
| 1191 | **API Parameters**: `after` and `before` are automatically converted from milliseconds to seconds when sent to functions. |
| 1192 | |
| 1193 | **Display Format:** |
| 1194 | - Timezone: Determined by URL parameter (`utc`) or system default |
| 1195 | - Format: Uses `Intl.DateTimeFormat` with browser's locale |
| 1196 | - Both types render as localized date/time strings with seconds precision |
| 1197 | |
| 1198 | ### Sorting Capabilities |
| 1199 | |
| 1200 | - **Backend Control**: Each column defines default sort with `RRDF_FIELD_SORT_*` |
| 1201 | - **User Control**: `sortable: true` enables UI sorting (default) |
| 1202 | - **Fixed Sort**: Bit flag 0x80 prevents user sorting |
| 1203 | - **Initial Sort**: `default_sort_column` specifies startup sort |
| 1204 | - **Multi-Column**: UI supports sorting by any sortable column |
| 1205 | - **Performance**: Client-side sorting for simple tables |
| 1206 | |
| 1207 | ### Filtering Capabilities |
| 1208 | |
| 1209 | - **Multiselect**: Default filter type with checkboxes |
| 1210 | - Shows all unique values from the column |
| 1211 | - Multiple selections allowed |
| 1212 | - OR logic between selections |
| 1213 | - **Range**: Numeric filters with min/max sliders |
| 1214 | - Requires numeric field type |
| 1215 | - Auto-detects min/max from data |
| 1216 | - Inclusive filtering |
| 1217 | - **Facet**: Advanced filtering for log explorer |
| 1218 | - Shows counts next to each option |
| 1219 | - Dynamic updates with other filters |
| 1220 | - Indexed for performance |
| 1221 | |
| 1222 | ## Anchor-Based Pagination |
| 1223 | |
| 1224 | Log explorer functions use anchor-based pagination for efficient navigation through large datasets. |
| 1225 | |
| 1226 | ### Configuration |
| 1227 | |
| 1228 | ```json |
| 1229 | { |
| 1230 | "pagination": { |
| 1231 | "enabled": true, |
| 1232 | "column": "timestamp", |
| 1233 | "key": "anchor", |
| 1234 | "units": "timestamp_usec" |
| 1235 | } |
| 1236 | } |
| 1237 | ``` |
| 1238 | |
| 1239 | ### Frontend Implementation |
| 1240 | |
| 1241 | **Anchor Management:** |
| 1242 | ```javascript |
| 1243 | // Frontend calculates anchors from data boundaries |
| 1244 | anchorBefore: latestData[latestData.length - 1][pagination.column], |
| 1245 | anchorAfter: latestData[0][pagination.column], |
| 1246 | anchorUnits: pagination.units |
| 1247 | ``` |
| 1248 | |
| 1249 | **Infinite Scroll Navigation:** |
| 1250 | - **Backward**: Scroll down loads older data using `anchorBefore` |
| 1251 | - **Forward**: Scroll up loads newer data using `anchorAfter` |
| 1252 | - **State Tracking**: `hasNextPage`, `hasPrevPage` control load triggers |
| 1253 | |
| 1254 | **Required Parameters:** |
| 1255 | - `anchor: {VALUE}` - Pagination cursor value |
| 1256 | - `direction: "backward"|"forward"` - Navigation direction |
| 1257 | - `last: NUMBER` - Page size (default: 200) |
| 1258 | |
| 1259 | ### PLAY Mode Integration |
| 1260 | |
| 1261 | When in PLAY mode (`after < 0`), pagination automatically coordinates with real-time updates: |
| 1262 | |
| 1263 | ```javascript |
| 1264 | { |
| 1265 | direction: "forward", |
| 1266 | merge: true, |
| 1267 | tail: true, |
| 1268 | delta: true, |
| 1269 | anchor: anchorAfter |
| 1270 | } |
| 1271 | ``` |
| 1272 | |
| 1273 | ### Large Data Sets |
| 1274 | |
| 1275 | Simple tables load all data at once, but handle large sets efficiently: |
| 1276 | - Virtual scrolling for thousands of rows |
| 1277 | - Client-side filtering/sorting |
| 1278 | - No built-in pagination (all data in response) |
| 1279 | |
| 1280 | Log explorer uses anchor-based pagination: |
| 1281 | - Efficient navigation through millions of records |
| 1282 | - Configurable page size (`last` parameter) |
| 1283 | - Bi-directional infinite scrolling |
| 1284 | - Sampling for very large sets |
| 1285 | |
| 1286 | ## Incremental Updates (Delta Mode) |
| 1287 | |
| 1288 | Delta mode enables efficient real-time updates by sending only changes since the last request. |
| 1289 | |
| 1290 | ### When Delta is Enabled |
| 1291 | |
| 1292 | ```javascript |
| 1293 | { |
| 1294 | if_modified_since: 1697644320000000, // Previous modification timestamp |
| 1295 | direction: "forward", |
| 1296 | merge: true, |
| 1297 | tail: true, |
| 1298 | delta: true, |
| 1299 | data_only: true, |
| 1300 | anchor: anchorAfter |
| 1301 | } |
| 1302 | ``` |
| 1303 | |
| 1304 | ### Delta Response Types |
| 1305 | |
| 1306 | **Facets Delta:** |
| 1307 | ```json |
| 1308 | { |
| 1309 | "facetsDelta": [ |
| 1310 | { |
| 1311 | "id": "priority", |
| 1312 | "options": [ |
| 1313 | {"id": "ERROR", "count": 5}, // Incremental counts |
| 1314 | {"id": "WARN", "count": 12} |
| 1315 | ] |
| 1316 | } |
| 1317 | ] |
| 1318 | } |
| 1319 | ``` |
| 1320 | |
| 1321 | **Histogram Delta:** |
| 1322 | ```json |
| 1323 | { |
| 1324 | "histogramDelta": { |
| 1325 | "chart": { |
| 1326 | "result": { |
| 1327 | "labels": ["time", "ERROR", "WARN"], |
| 1328 | "data": [ |
| 1329 | [1697644320, 3, 8] // New data points only |
| 1330 | ] |
| 1331 | } |
| 1332 | } |
| 1333 | } |
| 1334 | } |
| 1335 | ``` |
| 1336 | |
| 1337 | ### Data Merging |
| 1338 | |
| 1339 | Frontend merges delta responses with existing data: |
| 1340 | - **Facet counts**: Accumulated using `count = (existing || 0) + (delta || 0)` |
| 1341 | - **Table data**: Appended/prepended based on `direction` |
| 1342 | - **Histogram data**: New data points added to existing chart |
| 1343 | |
| 1344 | ## Real-Time Updates (PLAY Mode) |
| 1345 | |
| 1346 | PLAY mode enables live data streaming with efficient polling and conditional updates. |
| 1347 | |
| 1348 | ### PLAY Mode Detection |
| 1349 | |
| 1350 | - **PLAY Mode**: `after < 0` (relative time from now) |
| 1351 | - **PAUSE Mode**: `after > 0` (absolute timestamp) |
| 1352 | |
| 1353 | ### Parameter Coordination |
| 1354 | |
| 1355 | When `if_modified_since` is present, the system automatically includes: |
| 1356 | |
| 1357 | ```json |
| 1358 | { |
| 1359 | "if_modified_since": 1697644320000000, |
| 1360 | "direction": "forward", |
| 1361 | "merge": true, |
| 1362 | "tail": true, |
| 1363 | "delta": true, |
| 1364 | "data_only": true, |
| 1365 | "anchor": "anchorAfter" |
| 1366 | } |
| 1367 | ``` |
| 1368 | |
| 1369 | ### Error Handling |
| 1370 | |
| 1371 | **304 Not Modified**: Indicates no new data available |
| 1372 | ```json |
| 1373 | { |
| 1374 | "status": 304 |
| 1375 | } |
| 1376 | ``` |
| 1377 | |
| 1378 | Frontend handles 304 responses gracefully without showing errors to users. |
| 1379 | |
| 1380 | ### Polling Behavior |
| 1381 | |
| 1382 | - **Polling Interval**: Based on function's `update_every` value |
| 1383 | - **Auto-Pause**: When window loses focus or user hovers over data |
| 1384 | - **Conditional Requests**: Uses `if_modified_since` to avoid unnecessary data transfers |
| 1385 | |
| 1386 | ### Required vs Optional Fields |
| 1387 | |
| 1388 | **Minimum Required Fields:** |
| 1389 | ```json |
| 1390 | { |
| 1391 | "status": 200, // Required |
| 1392 | "type": "table", // Required |
| 1393 | "columns": {}, // Required (can be empty) |
| 1394 | "data": [] // Required (can be empty) |
| 1395 | } |
| 1396 | ``` |
| 1397 | |
| 1398 | **Common Optional Fields:** |
| 1399 | - `has_history`: Default false |
| 1400 | - `help`: Documentation text |
| 1401 | - `update_every`: Default 1 |
| 1402 | - `expires`: Cache control |
| 1403 | - `default_sort_column`: Initial sort |
| 1404 | - `accepted_params`, `required_params`: Some backends include these in data responses (e.g., go.d functions) |
| 1405 | - All column options except `index`, `name`, `type` |
| 1406 | |
| 1407 | ### UI-required fields (cloud-frontend) |
| 1408 | - **Info response**: `v`, `type`, `has_history`, `accepted_params`, `required_params`, `help` |
| 1409 | - **Data response**: `type`, `columns`, `data` |
| 1410 | - **Error response**: `errorMessage` (camelCase) is used by the Functions UI |
| 1411 | |
| 1412 | **Note on casing:** cloud-frontend camelizes **successful** responses (info/data) before use, but **does not** camelize error payloads. |
| 1413 | |
| 1414 | ## Error Handling |
| 1415 | |
| 1416 | ### Error Response Format |
| 1417 | |
| 1418 | When a function encounters an error, the backend returns a JSON object. The primary error generator (`rrd_call_function_error`) produces the following minimal format: |
| 1419 | |
| 1420 | ```json |
| 1421 | { |
| 1422 | "status": 400, // The HTTP status code (e.g., 400, 404, 500) |
| 1423 | "errorMessage": "A descriptive error message" |
| 1424 | } |
| 1425 | ``` |
| 1426 | |
| 1427 | **Frontend Consumption and Interpretation:** |
| 1428 | The frontend is designed to handle a more comprehensive error structure, allowing for richer error display and localization. When an error occurs, the frontend will attempt to extract information from the received error object using the following hierarchy: |
| 1429 | |
| 1430 | * **`status`**: The HTTP status code, used for general error classification (e.g., 400 for bad request, 404 for not found). |
| 1431 | * **`errorMessage`**: Primary detailed message used by the cloud-frontend Functions UI. |
| 1432 | * **`error`**: A short, machine-readable error identifier (e.g., "MissingParameter"). While not consistently generated by `rrd_call_function_error`, other parts of the system or future backend implementations might provide this. |
| 1433 | * **`message`**: A user-friendly message. The frontend often maps `errorMessage` or an internal `errorMsgKey` to this for display. |
| 1434 | * **`help`**: Optional additional guidance for resolving the error. This field is not currently generated by `rrd_call_function_error`. |
| 1435 | |
| 1436 | **Compatibility note (cloud-frontend):** |
| 1437 | - Cloud-frontend Functions UI expects `errorMessage` (camelCase) and does **not** camelize error payloads. |
| 1438 | |
| 1439 | **Example of Frontend Interpretation (Conceptual):** |
| 1440 | The frontend might internally map specific `errorMessage` strings to predefined `errorMsgKey` values to provide localized or more context-specific messages to the user. For instance, a backend `errorMessage` like "The 'time_range' parameter is required" might be mapped to an `errorMsgKey` of "ErrMissingTimeRange" in the frontend, which then displays a user-friendly message like "Please specify a time range for this function." |
| 1441 | |
| 1442 | Therefore, while the backend currently provides `status` and `errorMessage`, developers should be aware that the frontend's error handling is capable of utilizing the more detailed fields (`error`, `message`, `help`) if they are provided by the backend in the future or by other API endpoints. |
| 1443 | |
| 1444 | **Common Error Codes:** |
| 1445 | |
| 1446 | |
| 1447 | ### Common Error Codes |
| 1448 | |
| 1449 | | Code | Use Case | Example | |
| 1450 | |------|----------|---------| |
| 1451 | | 400 | Bad input | Missing/invalid parameters | |
| 1452 | | 401 | Auth required | User not logged in | |
| 1453 | | 403 | Forbidden | Insufficient permissions | |
| 1454 | | 404 | Not found | No data matches query | |
| 1455 | | 500 | Server error | Internal failures | |
| 1456 | | 503 | Unavailable | Service overloaded | |
| 1457 | |
| 1458 | ## Protocol Integration |
| 1459 | |
| 1460 | ### PLUGINSD Commands |
| 1461 | |
| 1462 | Functions integrate with Netdata through the PLUGINSD protocol: |
| 1463 | |
| 1464 | **Registration:** |
| 1465 | ``` |
| 1466 | FUNCTION "function_name" timeout "help text" "tags" "http_access" priority version |
| 1467 | ``` |
| 1468 | |
| 1469 | **Execution Flow:** |
| 1470 | 1. **Collector → Agent**: `FUNCTION` registers the function |
| 1471 | 2. **Agent → Collector**: `FUNCTION_CALL` with transaction ID |
| 1472 | 3. **Collector → Agent**: `FUNCTION_RESULT_BEGIN` status format expires |
| 1473 | 4. **Collector → Agent**: Response payload |
| 1474 | 5. **Collector → Agent**: `FUNCTION_RESULT_END` |
| 1475 | |
| 1476 | **With Payload:** |
| 1477 | ``` |
| 1478 | FUNCTION_PAYLOAD_BEGIN transaction timeout function access source content_type |
| 1479 | <payload data> |
| 1480 | FUNCTION_PAYLOAD_END |
| 1481 | ``` |
| 1482 | |
| 1483 | **Cancellation:** |
| 1484 | ``` |
| 1485 | FUNCTION_CANCEL transaction_id |
| 1486 | ``` |
| 1487 | |
| 1488 | **Progress Updates:** |
| 1489 | ``` |
| 1490 | FUNCTION_PROGRESS transaction_id done total |
| 1491 | ``` |
| 1492 | |
| 1493 | ### Streaming Protocol |
| 1494 | |
| 1495 | Functions support streaming for real-time updates: |
| 1496 | |
| 1497 | ```c |
| 1498 | // Transaction management |
| 1499 | dictionary_set(parser->inflight.functions, transaction_str, &function_data); |
| 1500 | |
| 1501 | // Timeout handling |
| 1502 | if (*pf->stop_monotonic_ut + RRDFUNCTIONS_TIMEOUT_EXTENSION_UT < now_ut) { |
| 1503 | // Function timed out |
| 1504 | } |
| 1505 | |
| 1506 | // Progress callback |
| 1507 | if(stream_has_capability(s, STREAM_CAP_PROGRESS)) { |
| 1508 | // Enable progress updates |
| 1509 | } |
| 1510 | ``` |
| 1511 | |
| 1512 | ### Key Files |
| 1513 | - `pluginsd_functions.c`: Function execution and management |
| 1514 | - `stream-sender-execute.c`: Streaming function calls |
| 1515 | - `plugins.d/README.md`: Protocol documentation |
| 1516 | - `plugins.d/functions-table.md`: Table format specification |
| 1517 | |
| 1518 | ## Migration Guide |
| 1519 | |
| 1520 | ### Upgrading from GET to POST (v:3) |
| 1521 | |
| 1522 | 1. Update info response to include `"v": 3` |
| 1523 | 2. Change field IDs from hashes to plain names |
| 1524 | 3. Test with JSON POST payloads |
| 1525 | 4. Remove hash generation code |
| 1526 | 5. Update documentation |
| 1527 | |
| 1528 | ### Adding Log Explorer Features |
| 1529 | |
| 1530 | 1. Set `has_history: true` |
| 1531 | 2. Implement facets with counts |
| 1532 | 3. Add timestamp column with microseconds |
| 1533 | 4. Support anchor-based pagination |
| 1534 | 5. Add histogram data |
| 1535 | 6. Implement full-text search |
| 1536 | |
| 1537 | ## Status and Next Steps |
| 1538 | |
| 1539 | This document represents the complete v3 protocol specification. Key areas for future development: |
| 1540 | |
| 1541 | 1. **Protocol Extensions** |
| 1542 | - Streaming updates for real-time data |
| 1543 | - Aggregation pipelines |
| 1544 | - Custom visualization types |
| 1545 | |
| 1546 | 2. **UI Enhancements** |
| 1547 | - Additional field types (gauge, sparkline) |
| 1548 | - Custom column renderers |
| 1549 | - Advanced filtering options |
| 1550 | |
| 1551 | 3. **Performance Optimizations** |
| 1552 | - Server-side pagination for simple tables |
| 1553 | - Incremental updates |
| 1554 | - Result caching |
| 1555 | |
| 1556 | --- |
| 1557 | |
| 1558 | ## Appendix A: Validation Checklist and Progress |
| 1559 | |
| 1560 | *This section tracks the validation work completed and remaining tasks* |
| 1561 | |
| 1562 | ### Format Discovery |
| 1563 | - [x] Analyze `processes` function implementation in apps.plugin |
| 1564 | - [x] Analyze `network-connections` function in network-viewer.plugin |
| 1565 | - [x] Identify common patterns between implementations |
| 1566 | - [x] Extract all enum definitions used in responses |
| 1567 | - [x] Document each field type and possible values |
| 1568 | - [x] Identify optional vs required fields |
| 1569 | |
| 1570 | ### Enumeration Completeness |
| 1571 | - [x] Find all enum definitions in netdata C code |
| 1572 | - [x] Map enum values to their string representations |
| 1573 | - [x] Document the purpose of each enum value |
| 1574 | - [ ] Check for any conditional enum values |
| 1575 | |
| 1576 | ### Function Coverage |
| 1577 | - [x] Scan all collectors/plugins for function implementations |
| 1578 | - [x] List all simple table functions (has_history=false) |
| 1579 | - [x] Verify format consistency across all functions |
| 1580 | - [x] Document any function-specific extensions |
| 1581 | |
| 1582 | ### UI Mapping |
| 1583 | - [x] Analyze cloud-frontend code for function rendering |
| 1584 | - [x] Map each format field to UI component |
| 1585 | - [x] Document how enum values affect UI behavior |
| 1586 | - [x] Identify any frontend-specific transformations |
| 1587 | |
| 1588 | ### Corner Cases |
| 1589 | - [x] Empty result sets |
| 1590 | - [x] Large data sets (pagination?) |
| 1591 | - [x] Error responses |
| 1592 | - [x] Null/missing values |
| 1593 | - [x] Special characters in data |
| 1594 | - [x] Numeric precision/formatting |
| 1595 | - [x] Date/time formatting |
| 1596 | - [x] Sorting capabilities |
| 1597 | - [x] Filtering capabilities |
| 1598 | |
| 1599 | ### Protocol Validation |
| 1600 | - [x] Request format documentation |
| 1601 | - [x] Response format documentation |
| 1602 | - [x] Error handling patterns |
| 1603 | - [x] Streaming protocol integration |
| 1604 | |
| 1605 | ### Cross-Reference Checks |
| 1606 | - [x] Compare documented format with actual implementations |
| 1607 | - [x] Verify all functions conform to documented format |
| 1608 | - [x] Check for undocumented features in UI |
| 1609 | - [x] Validate against any existing documentation |
| 1610 | |
| 1611 | --- |
| 1612 | |
| 1613 | ## Appendix B: Investigation History |
| 1614 | |
| 1615 | ### Analysis Log |
| 1616 | |
| 1617 | *This section documents the investigation process to avoid repeating work* |
| 1618 | |
| 1619 | #### Session 1 - Initial Setup and Analysis |
| 1620 | - Created FUNCTIONS.md structure |
| 1621 | - Established checklist for comprehensive validation |
| 1622 | - Analyzed `processes` function in apps.plugin |
| 1623 | - Location: `/netdata/src/collectors/apps.plugin/apps_functions.c` |
| 1624 | - Registration: `apps_plugin.c:752` |
| 1625 | - Uses standard table format with array-based data rows |
| 1626 | - Analyzed `network-connections` function in network-viewer.plugin |
| 1627 | - Location: `/netdata/src/collectors/network-viewer.plugin/network-viewer.c` |
| 1628 | - Registration via PLUGINSD protocol |
| 1629 | - Supports aggregated and detailed views |
| 1630 | - Extracted all RRDF enum definitions from: |
| 1631 | - `/netdata/src/libnetdata/buffer/functions_fields.h` |
| 1632 | - `/netdata/src/libnetdata/buffer/functions_fields.c` |
| 1633 | - Documented complete enum value mappings for field types, visualizations, transforms, etc. |
| 1634 | |
| 1635 | #### Session 2 - Protocol Integration and Corner Cases |
| 1636 | - Investigated PLUGINSD protocol integration |
| 1637 | - Found in `pluginsd_functions.c` and `stream-sender-execute.c` |
| 1638 | - Commands: FUNCTION, FUNCTION_CALL, FUNCTION_RESULT_BEGIN/END |
| 1639 | - Support for payloads, cancellation, and progress updates |
| 1640 | - Analyzed corner case handling: |
| 1641 | - Empty results: `{"status": 200, "type": "table", "columns": {}, "data": []}` |
| 1642 | - Null values: Rendered as empty cells in UI |
| 1643 | - Special characters: Proper JSON escaping throughout |
| 1644 | - Numeric precision: Controlled by `decimal_points` field |
| 1645 | - Date/time: Millisecond epochs for tables, microsecond for logs |
| 1646 | - Identified required vs optional fields: |
| 1647 | - Required: status, type, columns, data |
| 1648 | - Optional: Everything else (has_history, help, etc.) |
| 1649 | - Documented sorting and filtering capabilities: |
| 1650 | - Sorting: Backend-controlled defaults, user-sortable columns |
| 1651 | - Filtering: Multiselect (default), range (numeric), facet (logs) |
| 1652 | - Large datasets: Virtual scrolling for tables, pagination for logs |
| 1653 | |
| 1654 | ### Key Findings |
| 1655 | 1. **Data Format**: Simple tables use array-of-arrays for data rows |
| 1656 | 2. **Column Definition**: Each column has extensive metadata controlling display and behavior |
| 1657 | 3. **Common Functions**: Both implement `buffer_rrdf_table_add_field()` for column definitions |
| 1658 | 4. **Response Builder**: Uses `buffer_json_*` functions to build JSON responses |
| 1659 | 5. **Field Options**: Bit flags allow combining multiple options per field |
| 1660 | 6. **Protocol Evolution**: GET with hashes → POST with plain field names (v:3) |
| 1661 | 7. **UI Mapping**: Comprehensive component mapping based on type/visualization |
| 1662 | 8. **Error Handling**: Standardized error response format with HTTP codes |
| 1663 | 9. **Performance**: Client-side operations for tables, server-side for logs |
| 1664 | 10. **Extensibility**: Optional fields allow function-specific features |
| 1665 | |
| 1666 | ### Format Compliance Summary |
| 1667 | - **All 12 simple table functions are fully compliant** with the documented format |
| 1668 | - **Common extensions** found: |
| 1669 | - `rowOptions` field for row severity/status (5/12 functions) |
| 1670 | - `charts` and `default_charts` definitions (most functions) |
| 1671 | - `group_by` aggregation options (most functions) |
| 1672 | - `accepted_params` and `required_params` (metrics-cardinality) |
| 1673 | - `default_sort_column` for initial sorting |
| 1674 | - **No breaking deviations** - all extensions are additive and optional |
| 1675 | |
| 1676 | --- |
| 1677 | |
| 1678 | ## Appendix C: Log Explorer UI Features Detail |
| 1679 | |
| 1680 | *Detailed UI behaviors when has_history=true* |
| 1681 | |
| 1682 | ### UI Features Enabled |
| 1683 | |
| 1684 | When properly formatted, the log explorer UI provides: |
| 1685 | |
| 1686 | 1. **Sidebar with Faceted Filters** |
| 1687 | - Shows facets with real-time counts |
| 1688 | - Multi-select filtering |
| 1689 | - Collapsible sections |
| 1690 | - Search within facets |
| 1691 | |
| 1692 | 2. **Time-based Histogram** |
| 1693 | - Visual log distribution over time |
| 1694 | - Click and drag to select time ranges |
| 1695 | - Switch between different histogram fields |
| 1696 | - Auto-updates with filters |
| 1697 | |
| 1698 | 3. **Advanced Table Features** |
| 1699 | - Infinite scroll using anchor navigation |
| 1700 | - No manual pagination controls |
| 1701 | - Automatic row coloring by severity |
| 1702 | - Full-width message display |
| 1703 | - Column pinning and resizing |
| 1704 | |
| 1705 | 4. **Search and Navigation** |
| 1706 | - Full-text search box |
| 1707 | - Bi-directional navigation (forward/backward) |
| 1708 | - Jump to specific time |
| 1709 | - Export filtered results |
| 1710 | |
| 1711 | 5. **Live Features** |
| 1712 | - Tail mode for real-time updates |
| 1713 | - Auto-refresh based on `update_every` |
| 1714 | - Delta updates for efficiency |
| 1715 | - Notification of new entries |
| 1716 | |
| 1717 | --- |
| 1718 | |
| 1719 | *Last Updated: Based on analysis of Netdata codebase and cloud-frontend implementation* |