@cryptotaxi247 / netdata-1 / commits / ce5f3d1ad

Replace legacy functions-table.md with comprehensive UI documentation (#20697)

- Remove outdated functions-table.md specification - Add FUNCTION_UI_REFERENCE.md: Complete technical reference for Netdata Functions v3 protocol - Add FUNCTION_UI_DEVELOPER_GUIDE.md: Practical developer guide for creating functions - Documents anonymized to remove internal codebase references - Covers both simple tables and log explorer functions - Includes complete field types, options, and UI behavior specifications

Costa Tsaousis committed Jul 18, 2025 at 10:28 UTC ce5f3d1ad29ee4be0cb2d1f5c61cc60868b2976c
3 files changed +2630 -418
src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md new
+953
@@ -0,0 +1,953 @@
1 +# Netdata Functions: Developer Guide
2 +
3 +> **Note**: This is the practical developer guide. For the complete technical specification, see [FUNCTIONS_REFERENCE.md](FUNCTIONS_REFERENCE.md).
4 +
5 +## Overview
6 +
7 +This guide teaches you how to create Netdata functions that provide interactive data through the web UI. You'll learn to build both simple tables and advanced log explorers.
8 +
9 +**What You'll Learn:**
10 +- How to create simple table functions for system monitoring
11 +- How to build log explorer functions with faceted search
12 +- All column types, options, and their UI effects
13 +- Query patterns, filtering, and aggregation
14 +- Real-world examples and best practices
15 +
16 +**Quick Navigation:**
17 +- [Part 1: Simple Table Functions](#part-1-simple-table-functions) - Basic monitoring data
18 +- [Part 2: Log Explorer Functions](#part-2-log-explorer-functions) - Historical data with search
19 +- [Part 3: Complete Options Reference](#part-3-complete-options-reference) - Every option explained
20 +
21 +---
22 +
23 +# Part 1: Simple Table Functions
24 +
25 +Simple table functions display current system state - processes, connections, services, etc. They're perfect for "top-like" views and system monitoring.
26 +
27 +**Implementation Architecture:**
28 +- **Frontend handles**: Filtering, search, facet counting, sorting
29 +- **Backend provides**: Raw data, column definitions, optional charts
30 +- **Performance**: Limited by browser processing (good for <10k rows)
31 +- **Query parameter**: Processed in frontend only (substring search)
32 +- **Histograms**: Not supported
33 +
34 +## Your First Simple Table Function
35 +
36 +Start with this minimal working example:
37 +
38 +```json
39 +{
40 + "status": 200,
41 + "type": "table",
42 + "has_history": false,
43 + "help": "Shows system services",
44 + "data": [
45 + ["nginx", 25, "Running"],
46 + ["mysql", 67, "Stopped"],
47 + ["redis", 91, "Running"]
48 + ],
49 + "columns": {
50 + "name": {
51 + "index": 0,
52 + "name": "Service",
53 + "type": "string",
54 + "unique_key": true
55 + },
56 + "cpu": {
57 + "index": 1,
58 + "name": "CPU %",
59 + "type": "bar-with-integer",
60 + "max": 100
61 + },
62 + "status": {
63 + "index": 2,
64 + "name": "Status",
65 + "type": "string"
66 + }
67 + }
68 +}
69 +```
70 +
71 +This creates a basic 3-column table showing service names, CPU usage bars, and status.
72 +
73 +## Essential Features for Simple Tables
74 +
75 +### 1. Required Fields
76 +
77 +Every simple table function needs:
78 +
79 +```json
80 +{
81 + "status": 200, // HTTP status (200 = success)
82 + "type": "table", // Always "table"
83 + "has_history": false, // Simple table (not log explorer)
84 + "columns": {...}, // Column definitions
85 + "data": [...] // Your actual data
86 +}
87 +```
88 +
89 +### 2. Column Definitions
90 +
91 +Each column must have:
92 +
93 +```json
94 +"column_id": {
95 + "index": 0, // Position in data arrays (0-based)
96 + "name": "Display Name", // Column header shown to users
97 + "type": "string" // How to render the data
98 +}
99 +```
100 +
101 +### 3. The Data Array
102 +
103 +Data is an array of arrays - each inner array is one row:
104 +
105 +```json
106 +"data": [
107 + ["nginx", 25, "Running"], // Row 1: index 0=name, 1=cpu, 2=status
108 + ["mysql", 67, "Stopped"], // Row 2
109 + ["redis", 91, "Running"] // Row 3
110 +]
111 +```
112 +
113 +**Important**: Values must be in the same order as column `index` fields.
114 +
115 +## Adding Visual Polish
116 +
117 +Make your table more useful with these enhancements:
118 +
119 +```json
120 +{
121 + "status": 200,
122 + "type": "table",
123 + "has_history": false,
124 + "help": "System services with CPU usage and status",
125 + "data": [
126 + ["nginx", 25, "Running", {"rowOptions": {"severity": "normal"}}],
127 + ["mysql", 67, "Stopped", {"rowOptions": {"severity": "error"}}],
128 + ["redis", 91, "Running", {"rowOptions": {"severity": "warning"}}]
129 + ],
130 + "columns": {
131 + "name": {
132 + "index": 0,
133 + "name": "Service Name",
134 + "type": "string",
135 + "unique_key": true,
136 + "sticky": true,
137 + "filter": "multiselect"
138 + },
139 + "cpu": {
140 + "index": 1,
141 + "name": "CPU Usage",
142 + "type": "bar-with-integer",
143 + "units": "%",
144 + "max": 100,
145 + "sort": "descending",
146 + "filter": "range"
147 + },
148 + "status": {
149 + "index": 2,
150 + "name": "Status",
151 + "type": "string",
152 + "visualization": "pill",
153 + "filter": "multiselect"
154 + }
155 + },
156 + "default_sort_column": "cpu"
157 +}
158 +```
159 +
160 +**New Features Added:**
161 +- **Row coloring**: `rowOptions` with severity levels
162 +- **Sticky columns**: Pin important columns when scrolling
163 +- **Filters**: Let users filter by categories or numeric ranges
164 +- **Progress bars**: Visual CPU usage display
165 +- **Pills**: Status badges with colors
166 +- **Default sorting**: Start sorted by CPU usage
167 +
168 +## Limitations of Simple Tables
169 +
170 +Simple tables have certain limitations due to their frontend-only processing architecture:
171 +
172 +- **No histograms**: Time-based visualization is not supported
173 +- **Performance limits**: Frontend processing is limited by browser memory (recommend <10k rows)
174 +- **No backend search**: Query parameter is not sent to backend - search happens client-side only
175 +- **Static facet counts**: Counts are computed by frontend from all received data
176 +
177 +For large datasets or advanced log analysis features, consider using log explorers (`has_history: true`).
178 +
179 +## Advanced Simple Table Features
180 +
181 +### Aggregated Views
182 +
183 +Some functions can show both detailed and aggregated data. When enabled, facet pills show both counts:
184 +
185 +```json
186 +{
187 + "aggregated_view": {
188 + "column": "Count",
189 + "results_label": "unique combinations",
190 + "aggregated_label": "connections"
191 + }
192 +}
193 +```
194 +
195 +This enables smart facet pills like `"15 ⊃ 42"` meaning "15 connections aggregated into 42 rows".
196 +
197 +### Charts Integration
198 +
199 +Simple tables support interactive charts computed from your table data. The backend defines available charts, and the frontend computes and renders them.
200 +
201 +**Chart Types Available:**
202 +- `"bar"` - Basic bar chart
203 +- `"stacked-bar"` - Multi-column stacked bars (most common)
204 +- `"doughnut"` - Pie/doughnut chart
205 +- `"value"` - Simple numeric display
206 +
207 +**Example Configuration:**
208 +```json
209 +{
210 + "charts": {
211 + "cpu_usage": {
212 + "name": "CPU Usage by Service",
213 + "type": "stacked-bar",
214 + "columns": ["user_cpu", "system_cpu", "guest_cpu"],
215 + "groupBy": "column",
216 + "aggregation": "sum"
217 + },
218 + "memory_breakdown": {
219 + "name": "Memory Types",
220 + "type": "doughnut",
221 + "columns": ["resident", "virtual", "shared"],
222 + "groupBy": "all",
223 + "aggregation": "sum"
224 + }
225 + },
226 + "default_charts": [
227 + ["cpu_usage", "status"],
228 + ["memory_breakdown", "status"]
229 + ]
230 +}
231 +```
232 +
233 +**Chart Options:**
234 +- **`groupBy`**:
235 + - `"column"` (default): Group by selected filter column
236 + - `"all"`: Aggregate all data together
237 +- **`aggregation`**: `sum`, `mean`, `max`, `min`, `count`
238 +
239 +**How It Works:**
240 +1. Frontend takes your table data
241 +2. Groups by selected column (e.g., "service type")
242 +3. Aggregates values using specified method (e.g., sum CPU values)
243 +4. Renders chart with one bar/slice per group
244 +
245 +### Table Row Grouping
246 +
247 +Simple tables support grouping rows with customizable aggregation. When users group by a column, rows with the same value are combined using the aggregation method you specify.
248 +
249 +**Enable Grouping Options:**
250 +```json
251 +{
252 + "group_by": {
253 + "aggregated": [{
254 + "id": "by_status",
255 + "name": "By Status",
256 + "column": "status"
257 + }, {
258 + "id": "by_user",
259 + "name": "By User",
260 + "column": "user"
261 + }]
262 + }
263 +}
264 +```
265 +
266 +**Define Column Aggregation:**
267 +Each column needs a summary type to control how values are aggregated when grouping:
268 +
269 +```c
270 +// In your C backend code
271 +buffer_rrdf_table_add_field(
272 + wb, field_id++, "cpu_percent", "CPU %",
273 + RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
274 + RRDF_FIELD_VISUAL_BAR,
275 + RRDF_FIELD_TRANSFORM_NUMBER,
276 + 2, "%", 100.0, RRDF_FIELD_SORT_DESCENDING, NULL,
277 + RRDF_FIELD_SUMMARY_SUM, // Sum CPU values when grouping
278 + RRDF_FIELD_FILTER_RANGE,
279 + RRDF_FIELD_OPTS_VISIBLE, NULL
280 +);
281 +
282 +buffer_rrdf_table_add_field(
283 + wb, field_id++, "process_count", "Processes",
284 + RRDF_FIELD_TYPE_INTEGER,
285 + RRDF_FIELD_VISUAL_VALUE,
286 + RRDF_FIELD_TRANSFORM_NUMBER,
287 + 0, "processes", NAN, RRDF_FIELD_SORT_DESCENDING, NULL,
288 + RRDF_FIELD_SUMMARY_COUNT, // Count processes when grouping
289 + RRDF_FIELD_FILTER_RANGE,
290 + RRDF_FIELD_OPTS_VISIBLE, NULL
291 +);
292 +```
293 +
294 +**Available Summary/Aggregation Types:**
295 +- `RRDF_FIELD_SUMMARY_COUNT` - Count rows in group
296 +- `RRDF_FIELD_SUMMARY_SUM` - Sum numeric values
297 +- `RRDF_FIELD_SUMMARY_MEAN` - Average values
298 +- `RRDF_FIELD_SUMMARY_MIN` - Minimum value
299 +- `RRDF_FIELD_SUMMARY_MAX` - Maximum value
300 +- `RRDF_FIELD_SUMMARY_UNIQUECOUNT` - Count unique values
301 +- `RRDF_FIELD_SUMMARY_MEDIAN` - Median value
302 +
303 +**Example Result:**
304 +When user groups by "service type", rows like:
305 +```
306 +nginx_worker1 25% Running
307 +nginx_worker2 30% Running
308 +mysql_main 45% Running
309 +```
310 +
311 +Become:
312 +```
313 +nginx 55% 2 processes (25% + 30% CPU, 2 processes counted)
314 +mysql 45% 1 process (45% CPU, 1 process counted)
315 +```
316 +
317 +---
318 +
319 +# Part 2: Log Explorer Functions
320 +
321 +Log explorer functions (`has_history: true`) provide advanced log analysis with full-text search, faceted filtering, time navigation, and histograms. Perfect for systemd journals, event logs, and audit trails.
322 +
323 +**Implementation Architecture:**
324 +- **Backend handles**: Query processing, pattern matching, facet counting, filtering
325 +- **Frontend provides**: UI for facets, histogram display, infinite scroll
326 +- **Performance**: Scales to millions of records with sampling and server-side filtering
327 +- **Query parameter**: Processed by backend using Netdata simple patterns
328 +- **Histograms**: Optional, generated by backend in Netdata chart format
329 +- **Uses facets library**: All processing leverages `libnetdata/facets/`
330 +
331 +### Complete Log Explorer Example
332 +
333 +```json
334 +{
335 + "status": 200,
336 + "type": "table",
337 + "has_history": true,
338 + "help": "System journal with real-time updates",
339 + "accepted_params": [
340 + "info", "after", "before", "direction", "last", "anchor",
341 + "query", "facets", "histogram", "if_modified_since",
342 + "data_only", "delta", "tail", "sampling"
343 + ],
344 + "table": {"id": "journal", "has_history": true},
345 + "columns": {
346 + "timestamp": {
347 + "index": 0,
348 + "name": "Time",
349 + "type": "timestamp",
350 + "transform": "datetime_usec",
351 + "sort": "descending|fixed",
352 + "sticky": true
353 + },
354 + "priority": {
355 + "index": 1,
356 + "name": "Level",
357 + "type": "string",
358 + "visualization": "pill",
359 + "filter": "facet",
360 + "options": ["facet", "visible", "sticky"]
361 + },
362 + "message": {
363 + "index": 2,
364 + "name": "Message",
365 + "type": "string",
366 + "full_width": true,
367 + "options": ["full_width", "wrap", "visible", "main_text", "fts"]
368 + }
369 + },
370 + "data": [
371 + [1697644320000000, {"rowOptions": {"severity": "error"}}, "ERROR", "Service failed"],
372 + [1697644319000000, {"rowOptions": {"severity": "normal"}}, "INFO", "Service started"]
373 + ],
374 + "facets": [
375 + {
376 + "id": "priority",
377 + "name": "Log Level",
378 + "order": 1,
379 + "defaultExpanded": true,
380 + "options": [
381 + {"id": "ERROR", "name": "ERROR", "count": 45, "order": 1},
382 + {"id": "INFO", "name": "INFO", "count": 234, "order": 3}
383 + ]
384 + }
385 + ],
386 + "items": {
387 + "evaluated": 50000,
388 + "matched": 2520,
389 + "returned": 100,
390 + "max_to_return": 100
391 + },
392 + "anchor": {
393 + "last_modified": 1697644320000000,
394 + "direction": "backward"
395 + }
396 +}
397 +```
398 +
399 +**Key Features Demonstrated:**
400 +- `has_history: true` - Enables log explorer UI with infinite scroll
401 +- `accepted_params` - Full parameter support for advanced features
402 +- **Faceted search** - Real-time counts computed by backend
403 +- **Anchor pagination** - Efficient navigation through large datasets
404 +- **Microsecond timestamps** - High precision for log entries
405 +- **Full-text search** - Backend pattern matching with `"fts"` fields
406 +- **Row coloring** - Severity-based visual indicators
407 +
408 +## Essential Log Explorer Features
409 +
410 +### 1. Faceted Search Sidebar
411 +
412 +Facets provide dynamic filtering with real-time counts computed by the backend:
413 +
414 +```json
415 +{
416 + "facets": [
417 + {
418 + "id": "level",
419 + "name": "Log Level",
420 + "order": 1,
421 + "defaultExpanded": true,
422 + "options": [
423 + {"id": "ERROR", "name": "ERROR", "count": 45, "order": 1},
424 + {"id": "WARN", "name": "WARN", "count": 123, "order": 2},
425 + {"id": "INFO", "name": "INFO", "count": 2341, "order": 3}
426 + ]
427 + }
428 + ]
429 +}
430 +```
431 +
432 +**Important**: These counts are calculated by the backend facets library during query execution, not by the frontend. The backend scans through matching records and maintains counters for each facet value.
433 +
434 +### 2. Full-Text Search
435 +
436 +Enable powerful query search across all fields (processed by backend):
437 +
438 +```json
439 +{
440 + "columns": {
441 + "message": {
442 + "options": ["fts"], // Make field full-text searchable
443 + // ... other options
444 + }
445 + }
446 +}
447 +```
448 +
449 +**Important**: Unlike simple tables, the query parameter is sent to the backend and processed server-side using the facets library. The backend performs pattern matching and only returns matching records.
450 +
451 +**Query Patterns** (using Netdata simple patterns):
452 +- `"error"` - Find "error" anywhere (substring match)
453 +- `"error|warning"` - Find "error" OR "warning" (pipe separator)
454 +- `"!debug"` - Exclude logs containing "debug"
455 +- `"!*debugging*|*debug*"` - Include "debug" but exclude "debugging"
456 +- `"connection failed"` - Find exact phrase (spaces are literal)
457 +
458 +**Pattern Evaluation Rules:**
459 +1. **Within field**: Left-to-right, first match wins
460 +2. **Across fields**: ALL fields evaluated, ANY negative match excludes row
461 +3. **Case-insensitive** matching throughout
462 +
463 +### 3. Time-Based Histograms
464 +
465 +Visualize log distribution over time (log explorers only):
466 +
467 +```json
468 +{
469 + "available_histograms": [
470 + {"id": "level", "name": "Log Level", "order": 1},
471 + {"id": "source", "name": "Source", "order": 2}
472 + ],
473 + "histogram": {
474 + "id": "level",
475 + "name": "Log Level",
476 + "chart": {
477 + "result": {
478 + "labels": ["time", "ERROR", "WARN", "INFO"],
479 + "data": [
480 + [1697644200, 5, 12, 234],
481 + [1697644260, 3, 8, 198]
482 + ]
483 + }
484 + }
485 + }
486 +}
487 +```
488 +
489 +**Important**: Histograms are NOT supported for simple tables (`has_history: false`). They are optional for log explorers and use the same format as Netdata's `/api/v3/data` endpoint. The backend generates histogram data using the facets library.
490 +
491 +### 4. Anchor-Based Navigation
492 +
493 +Handle large datasets with efficient pagination:
494 +
495 +```json
496 +{
497 + "items": {
498 + "evaluated": 50000, // Total scanned
499 + "matched": 2520, // Match filters
500 + "returned": 100, // In this response
501 + "max_to_return": 100
502 + },
503 + "anchor": {
504 + "last_modified": 1697644320000000,
505 + "direction": "backward"
506 + }
507 +}
508 +```
509 +
510 +### Parameter Integration Patterns
511 +
512 +**Basic Monitoring Function:**
513 +```json
514 +{
515 + "accepted_params": ["info", "after", "before"]
516 +}
517 +```
518 +
519 +**Advanced Log Function:**
520 +```json
521 +{
522 + "accepted_params": [
523 + "info", "after", "before", "direction", "last", "anchor",
524 + "query", "facets", "histogram", "if_modified_since",
525 + "data_only", "delta", "tail", "sampling", "slice"
526 + ]
527 +}
528 +```
529 +
530 +**UI Feature Enablement:**
531 +- `"slice"` → Enables "Full data queries" toggle
532 +- `"direction"` → Enables bidirectional pagination
533 +- `"tail"` → Enables streaming mode
534 +- `"delta"` → Enables incremental updates
535 +- `"query"` → Enables full-text search
536 +
537 +## Advanced Log Explorer Features
538 +
539 +### Column Options for Logs
540 +
541 +Log explorers support special column options:
542 +
543 +| Option | Effect |
544 +|--------|--------|
545 +| `"fts"` | Full-text searchable by query |
546 +| `"facet"` | Appears in sidebar filters with counts |
547 +| `"main_text"` | Primary content (usually message) |
548 +| `"rich_text"` | May contain formatting |
549 +| `"hidden"` | Hide by default |
550 +
551 +Example:
552 +```json
553 +{
554 + "message": {
555 + "type": "string",
556 + "full_width": true,
557 + "options": ["full_width", "wrap", "visible", "main_text", "fts", "rich_text"]
558 + }
559 +}
560 +```
561 +
562 +### Log-Specific UI Behavior
563 +
564 +When `has_history: true`:
565 +- **Sidebar**: Shows faceted filters instead of simple filters
566 +- **Search box**: Queries all `fts` fields using pattern matching
567 +- **Infinite scroll**: Loads more data as you scroll
568 +- **Time navigation**: Jump to specific time periods
569 +- **Sampling**: For very large datasets
570 +
571 +---
572 +
573 +# Part 3: Complete Options Reference
574 +
575 +This section documents every field type, option, and behavior for quick reference while developing.
576 +
577 +## Field Types and UI Rendering
578 +
579 +### Text and Categories
580 +
581 +```json
582 +{
583 + "type": "string",
584 + "visualization": "value" // Default: plain text, left-aligned
585 +}
586 +```
587 +
588 +```json
589 +{
590 + "type": "string",
591 + "visualization": "pill" // Colored badges for status/categories
592 +}
593 +```
594 +
595 +### Numbers and Metrics
596 +
597 +```json
598 +{
599 + "type": "integer", // Right-aligned numbers
600 + "transform": "number", // Respect decimal_points
601 + "decimal_points": 2
602 +}
603 +```
604 +
605 +```json
606 +{
607 + "type": "bar-with-integer", // Progress bars with values
608 + "max": 100, // Required for bars
609 + "units": "%"
610 +}
611 +```
612 +
613 +### Timestamp Format Requirements
614 +
615 +**Simple Tables**: Use milliseconds with `datetime` transform
616 +```json
617 +{
618 + "type": "timestamp",
619 + "transform": "datetime", // Expects milliseconds
620 + "data": [1697644320000] // JavaScript Date format
621 +}
622 +```
623 +
624 +**Log Explorers**: Use microseconds with `datetime_usec` transform
625 +```json
626 +{
627 + "type": "timestamp",
628 + "transform": "datetime_usec", // Expects microseconds
629 + "data": [1697644320000000] // Microsecond precision
630 +}
631 +```
632 +
633 +**Frontend Conversion:**
634 +```javascript
635 +// datetime_usec automatically converts to milliseconds
636 +if (usec) {
637 + epoch = epoch ? Math.floor(epoch / 1000) : epoch
638 +}
639 +```
640 +
641 +**API Parameters**: `after` and `before` are automatically converted from milliseconds to seconds when sent to functions.
642 +
643 +```json
644 +{
645 + "type": "duration",
646 + "transform": "duration_s" // Formats seconds as "1d 2h 3m"
647 +}
648 +```
649 +
650 +### Rich Content
651 +
652 +```json
653 +{
654 + "type": "feedTemplate", // Full-width rich content
655 + "full_width": true // Automatically applied
656 +}
657 +```
658 +
659 +## Essential Column Options
660 +
661 +### Layout Control
662 +
663 +```json
664 +{
665 + "unique_key": true, // Row identifier (exactly one required)
666 + "sticky": true, // Pin when scrolling horizontally
667 + "visible": true, // Show by default
668 + "full_width": true, // Expand to fill available space
669 + "wrap": true // Enable text wrapping
670 +}
671 +```
672 +
673 +### Filtering Options
674 +
675 +```json
676 +{
677 + "filter": "multiselect" // Checkboxes (default for simple tables)
678 +}
679 +```
680 +
681 +```json
682 +{
683 + "filter": "range" // Min/max sliders for numbers
684 +}
685 +```
686 +
687 +```json
688 +{
689 + "filter": "facet" // Sidebar with counts (log explorers only)
690 +}
691 +```
692 +
693 +### Sorting Options
694 +
695 +```json
696 +{
697 + "sort": "descending", // Default sort direction
698 + "sortable": true // Allow user sorting (default)
699 +}
700 +```
701 +
702 +```json
703 +{
704 + "sort": "descending|fixed", // Prevent user from changing sort
705 + "sortable": false
706 +}
707 +```
708 +
709 +## Row-Level Features
710 +
711 +### Row Coloring by Severity
712 +
713 +Add row coloring by including `rowOptions` as the last element:
714 +
715 +```json
716 +{
717 + "data": [
718 + ["normal data", "values", {"rowOptions": {"severity": "normal"}}],
719 + ["warning data", "values", {"rowOptions": {"severity": "warning"}}],
720 + ["error data", "values", {"rowOptions": {"severity": "error"}}],
721 + ["notice data", "values", {"rowOptions": {"severity": "notice"}}]
722 + ]
723 +}
724 +```
725 +
726 +**Severity Levels:**
727 +- `"normal"` - Default appearance
728 +- `"warning"` - Yellow background
729 +- `"error"` - Red background
730 +- `"notice"` - Blue background
731 +
732 +## Chart and Grouping Configuration
733 +
734 +### Chart Definition
735 +
736 +```json
737 +{
738 + "charts": {
739 + "resource_usage": {
740 + "name": "Resource Usage",
741 + "type": "stacked-bar",
742 + "columns": ["cpu", "memory", "disk"],
743 + "groupBy": "column",
744 + "aggregation": "sum"
745 + },
746 + "status_distribution": {
747 + "name": "Status Distribution",
748 + "type": "doughnut",
749 + "columns": ["count"],
750 + "groupBy": "all",
751 + "aggregation": "count"
752 + }
753 + },
754 + "default_charts": [
755 + ["resource_usage", "service_type"],
756 + ["status_distribution", "status"]
757 + ]
758 +}
759 +```
760 +
761 +### Grouping Configuration
762 +
763 +```json
764 +{
765 + "group_by": {
766 + "aggregated": [
767 + {
768 + "id": "by_service",
769 + "name": "By Service Type",
770 + "column": "service_type"
771 + },
772 + {
773 + "id": "by_status",
774 + "name": "By Status",
775 + "column": "status"
776 + }
777 + ]
778 + }
779 +}
780 +```
781 +
782 +## Backend Implementation Examples
783 +
784 +### C Code for Simple Tables
785 +
786 +```c
787 +// Add a progress bar column
788 +buffer_rrdf_table_add_field(
789 + wb, field_id++, "cpu", "CPU Usage",
790 + RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
791 + RRDF_FIELD_VISUAL_BAR,
792 + RRDF_FIELD_TRANSFORM_NUMBER,
793 + 2, "%", 100.0, RRDF_FIELD_SORT_DESCENDING, NULL,
794 + RRDF_FIELD_SUMMARY_SUM,
795 + RRDF_FIELD_FILTER_RANGE,
796 + RRDF_FIELD_OPTS_VISIBLE, NULL
797 +);
798 +
799 +// Add row coloring
800 +buffer_json_add_array_item_object(wb);
801 +buffer_json_member_add_object(wb, "rowOptions");
802 +buffer_json_member_add_string(wb, "severity", "error");
803 +buffer_json_object_close(wb);
804 +buffer_json_object_close(wb);
805 +```
806 +
807 +### Using the Facets Library (Log Explorers)
808 +
809 +```c
810 +// Initialize facets for log exploration
811 +FACETS *facets = facets_create(...);
812 +
813 +// Set up query search
814 +facets_set_query(facets, query_string);
815 +
816 +// Add a faceted field
817 +facets_register_facet_id(facets, "level",
818 + FACET_KEY_OPTION_FACET | FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_REORDER);
819 +
820 +// Generate the response
821 +facets_table_config(facets, wb);
822 +```
823 +
824 +## Error Handling
825 +
826 +Always handle the info request first:
827 +
828 +```json
829 +{
830 + "v": 3, // Enable POST requests
831 + "status": 200,
832 + "type": "table",
833 + "has_history": false, // or true for log explorers
834 + "help": "Function description",
835 + "accepted_params": [...],
836 + "required_params": [...]
837 +}
838 +```
839 +
840 +### Complete Info Response Example
841 +
842 +```json
843 +{
844 + "v": 3,
845 + "status": 200,
846 + "type": "table",
847 + "has_history": true,
848 + "help": "System log explorer with faceted search",
849 + "accepted_params": [
850 + "info", "after", "before", "direction", "last", "anchor",
851 + "query", "facets", "histogram", "if_modified_since",
852 + "data_only", "delta", "tail", "sampling"
853 + ],
854 + "required_params": [
855 + {
856 + "id": "unit",
857 + "name": "System Unit",
858 + "type": "select",
859 + "options": [
860 + {"id": "nginx.service", "name": "Nginx"},
861 + {"id": "mysql.service", "name": "MySQL"}
862 + ]
863 + }
864 + ]
865 +}
866 +```
867 +
868 +**Critical Fields:**
869 +- `"v": 3` enables POST requests with JSON payloads
870 +- `accepted_params` determines which parameters the function accepts
871 +- `required_params` generates filter UI and validates execution
872 +
873 +For errors, return:
874 +
875 +```json
876 +{
877 + "status": 400,
878 + "error_message": "Descriptive error message"
879 +}
880 +```
881 +
882 +### Performance Optimization
883 +
884 +**For Large Datasets:**
885 +
886 +1. **Enable Sampling**:
887 +```json
888 +{
889 + "accepted_params": ["sampling"],
890 + "sampling": 10 // 1 in 10 sampling
891 +}
892 +```
893 +
894 +2. **Use Delta Updates**:
895 +```json
896 +{
897 + "if_modified_since": 1697644320000000,
898 + "delta": true,
899 + "data_only": true
900 +}
901 +```
902 +
903 +3. **Implement Tail Limiting**:
904 +```c
905 +// Limit tail data to prevent memory issues
906 +if (tail_mode) {
907 + limit_results_to(500);
908 +}
909 +```
910 +
911 +4. **Support Anchor Pagination**:
912 +```json
913 +{
914 + "pagination": {
915 + "enabled": true,
916 + "column": "timestamp",
917 + "key": "anchor",
918 + "units": "timestamp_usec"
919 + }
920 +}
921 +```
922 +
923 +## Best Practices Summary
924 +
925 +### For Simple Tables
926 +1. Always include one `unique_key` column
927 +2. Use `bar-with-integer` for metrics with `max` values
928 +3. Add `filter: "range"` for numbers, `"multiselect"` for categories
929 +4. Use `rowOptions` for status indication
930 +5. Set meaningful `default_sort_column`
931 +6. Define column `summary` types for grouping (SUM for metrics, COUNT for processes)
932 +7. Add charts for key metrics with appropriate `groupBy` and `aggregation`
933 +8. Include `group_by` options for common analysis patterns
934 +
935 +### For Log Explorers
936 +1. Set `has_history: true`
937 +2. Use microsecond timestamps with `datetime_usec`
938 +3. Mark important fields with `"fts"` for search
939 +4. Use `filter: "facet"` instead of `"multiselect"`
940 +5. Include proper facets with counts
941 +6. Implement anchor-based pagination
942 +
943 +### General
944 +1. Test with `?info` requests first
945 +2. Include helpful `help` text
946 +3. Handle errors gracefully
947 +4. Use appropriate `units` for clarity
948 +5. Follow existing function patterns in your codebase
949 +
950 +
951 +---
952 +
953 +This guide covers everything you need to build both simple monitoring functions and advanced log explorers. For implementation details and edge cases, see [FUNCTIONS_REFERENCE.md](FUNCTIONS_REFERENCE.md).
\ No newline at end of file
src/plugins.d/FUNCTION_UI_REFERENCE.md new
+1677
@@ -0,0 +1,1677 @@
1 +# Netdata Functions v3 Protocol - Technical Reference
2 +
3 +> **Note**: This is the technical specification. For a practical guide to implementing functions, see [FUNCTIONS_DEVELOPER_GUIDE.md](FUNCTIONS_DEVELOPER_GUIDE.md).
4 +
5 +## Overview
6 +
7 +This document provides the complete technical reference for Netdata Functions protocol v3, combining all information about simple tables (has_history=false) and log explorers (has_history=true). This is the authoritative internal documentation for maintaining and extending Netdata functions.
8 +
9 +## Table of Contents
10 +
11 +1. [Protocol Overview](#protocol-overview)
12 +2. [Request Flow](#request-flow)
13 +3. [Simple Table Format](#simple-table-format)
14 +4. [Log Explorer Format](#log-explorer-format)
15 +5. [Field Types and Enumerations](#field-types-and-enumerations)
16 +6. [UI Implementation](#ui-implementation)
17 +7. [Backend Implementation](#backend-implementation)
18 +8. [Best Practices](#best-practices)
19 +9. [Known Functions](#known-functions)
20 +10. [Development Checklist](#development-checklist)
21 +
22 +## Protocol Overview
23 +
24 +Netdata Functions allow collectors/plugins to expose interactive data through a streaming protocol. The protocol has evolved from GET-based CLI parameters (legacy) to POST-based JSON payloads (modern v3).
25 +
26 +### Function Types
27 +
28 +1. **Simple Table View** (`has_history: false`)
29 + - Basic tabular data display with frontend-side filtering and search
30 + - Examples: `processes`, `network-connections`, `block-devices`
31 +
32 +2. **Log Explorer Format** (`has_history: true`)
33 + - Advanced table with backend-powered faceted search, histograms, and infinite scroll
34 + - Examples: `systemd-journal`, `windows-events`
35 +
36 +### Critical Implementation Differences
37 +
38 +| Aspect | Simple Tables | Log Explorers |
39 +|--------|---------------|---------------|
40 +| **Data Processing** | All data sent to frontend | Backend filters before sending |
41 +| **Facet Counts** | Frontend counts occurrences in received data | Backend computes counts during query execution |
42 +| **Full-Text Search** | Frontend substring search across visible data | Backend pattern matching with facets library |
43 +| **Histograms** | Not supported - no time-based visualization | Optional - backend generates Netdata chart format |
44 +| **Performance** | Limited by browser memory and processing | Scales to millions of records with sampling |
45 +| **Query Parameter** | Ignored by backend (frontend only) | Processed by backend using simple patterns |
46 +
47 +## Request Flow
48 +
49 +### Modern Flow (v:3)
50 +
51 +1. **Info Request** (Always GET)
52 + ```
53 + GET /api/v3/function?function=systemd-journal info after:1234567890 before:1234567890
54 + ```
55 +
56 +2. **Info Response**
57 + ```json
58 + {
59 + "v": 3, // Indicates POST should be used for data requests
60 + "accepted_params": [...],
61 + "required_params": [...],
62 + "status": 200,
63 + "type": "table",
64 + "has_history": false
65 + }
66 + ```
67 +
68 +3. **Data Request** (POST when v=3)
69 + ```json
70 + {
71 + "query": "*error* !*debug*",
72 + "selections": {
73 + "priority": ["error", "warning"],
74 + "unit": ["nginx.service"]
75 + },
76 + "after": 1234567890,
77 + "before": 1234567890,
78 + "last": 100
79 + }
80 + ```
81 +
82 +### Preflight Info Request Details
83 +
84 +**Request Format:**
85 +```
86 +GET /api/v3/function?function=systemd-journal info after:1234567890 before:1234567890
87 +```
88 +
89 +**Required Response Fields:**
90 +```json
91 +{
92 + "v": 3,
93 + "status": 200,
94 + "type": "table",
95 + "has_history": false,
96 + "accepted_params": ["info", "after", "before", "direction", "last"],
97 + "required_params": [
98 + {
99 + "id": "priority",
100 + "name": "Log Level",
101 + "type": "select",
102 + "options": [
103 + {"id": "error", "name": "Error", "defaultSelected": true},
104 + {"id": "warn", "name": "Warning"}
105 + ]
106 + }
107 + ],
108 + "help": "Function description"
109 +}
110 +```
111 +
112 +**Frontend Processing:**
113 +- `accepted_params`: Validates which parameters can be sent to function
114 +- `required_params`: Generates filter UI, prevents execution if missing
115 +- `v: 3`: Enables POST requests with JSON payloads
116 +- Missing required parameters show user-friendly error messages
117 +
118 +### Backend Implementation
119 +
120 +```c
121 +// Simplified from logs_query_status.h
122 +if(payload) {
123 + // POST request - parse JSON payload
124 + facets_use_hashes_for_ids(facets, false); // Use plain field names
125 + rq->fields_are_ids = false;
126 +} else {
127 + // GET request - parse CLI parameters (legacy)
128 + facets_use_hashes_for_ids(facets, true); // Use hash IDs
129 + rq->fields_are_ids = true;
130 +}
131 +```
132 +
133 +### Key Differences
134 +
135 +| Aspect | GET (Legacy) | POST (Modern v3) |
136 +|--------|--------------|------------------|
137 +| Version | v < 3 | v = 3 |
138 +| Field IDs | 11-char hashes | Plain names |
139 +| Parameters | URL encoded | JSON body |
140 +| Facet filters | `field_hash:value1,value2` | `{"field": ["value1", "value2"]}` |
141 +| Full-text search | `query:search terms` | `{"query": "search terms"}` |
142 +
143 +### Standard Accepted Parameters
144 +
145 +**Core Parameters (All Functions):**
146 +- `"info"` - Function info requests (always supported)
147 +- `"after"` - Time range start (seconds epoch)
148 +- `"before"` - Time range end (seconds epoch)
149 +
150 +**Log Explorer Parameters (has_history=true):**
151 +- `"direction"` - Query direction: `"backward"` | `"forward"`
152 +- `"last"` - Result limit (default: 200)
153 +- `"anchor"` - Pagination cursor (timestamp or row identifier)
154 +- `"query"` - Full-text search using Netdata simple patterns
155 +- `"facets"` - Facet selection: `"field1,field2"`
156 +- `"histogram"` - Histogram field selection
157 +- `"if_modified_since"` - Conditional updates (microseconds epoch)
158 +- `"data_only"` - Skip metadata in response (boolean)
159 +- `"delta"` - Incremental responses (boolean)
160 +- `"tail"` - Streaming mode (boolean)
161 +- `"sampling"` - Data sampling control
162 +
163 +**UI Feature Parameters:**
164 +- `"slice"` - Enables "Full data queries" toggle in UI
165 +
166 +**Frontend Behavior:**
167 +```javascript
168 +// Only accepted parameters are sent to functions
169 +const allowedFilterIds = [...selectedFacets, ...requiredParamIds, ...acceptedParams]
170 +filtersToSend = allowedFilterIds.reduce((acc, filterId) => {
171 + if (filterId in filters) acc[filterId] = filters[filterId]
172 + return acc
173 +}, {})
174 +```
175 +
176 +### Query Parameter (Netdata Simple Patterns)
177 +
178 +The `query` parameter uses Netdata's simple pattern matching for full-text search across all fields:
179 +
180 +**Pattern Syntax:**
181 +- `|` - Pattern separator (OR logic between patterns)
182 +- `*` - Wildcard matching any number of characters
183 +- `!` - Negates the pattern (exclude matches)
184 +- Spaces are matched literally (not pattern separators)
185 +- Case-insensitive matching
186 +- Default behavior is substring matching (no wildcards needed)
187 +
188 +**Matching Modes:**
189 +- `pattern` - Substring match (default) - finds "pattern" anywhere
190 +- `*pattern` - Suffix match - finds strings ending with "pattern"
191 +- `pattern*` - Prefix match - finds strings starting with "pattern"
192 +- `*pattern*` - Substring match (explicit) - same as default
193 +
194 +**Examples:**
195 +```json
196 +{
197 + "query": "error" // Finds "error" anywhere (substring)
198 + "query": "error|warning" // Finds "error" OR "warning"
199 + "query": "error|warning|critical" // Multiple OR patterns
200 + "query": "!debug" // Exclude ALL rows containing "debug"
201 + "query": "!*debugging*|*debug*" // Include "debug" but exclude "debugging"
202 + "query": "connection failed" // Find exact phrase (spaces included)
203 + "query": "*error" // Find strings ending with "error"
204 + "query": "nginx*" // Find strings starting with "nginx"
205 +}
206 +```
207 +
208 +**Pattern Evaluation Rules:**
209 +
210 +1. **Within a field**: Left-to-right, first match wins
211 + - `"!*debugging*|*debug*"` - If text contains "debugging", it's negative match. Otherwise, if contains "debug", it's positive match.
212 +
213 +2. **Across all fields**: ALL fields are evaluated (no short-circuit)
214 + - Every field with FTS enabled is checked against the pattern
215 + - Positive and negative matches are counted separately
216 + - Field evaluation order doesn't affect the outcome
217 +
218 +3. **Row decision (after all fields evaluated)**:
219 + - **Excluded if**: ANY field has a negative match (regardless of positive matches)
220 + - **Included if**: At least one positive match AND zero negative matches
221 + - **Excluded if**: No positive matches found
222 +
223 +**Example**: Query `"!*debugging*|*debug*"`
224 +```
225 +Row 1: message="debug info", category="debugging tips"
226 + → message: positive match (debug)
227 + → category: negative match (debugging)
228 + → Result: EXCLUDED (has negative match)
229 +
230 +Row 2: message="debug info", category="testing"
231 + → message: positive match (debug)
232 + → category: no match
233 + → Result: INCLUDED (positive match, no negative)
234 +
235 +Row 3: message="error info", category="testing"
236 + → message: no match
237 + → category: no match
238 + → Result: EXCLUDED (no positive matches)
239 +```
240 +
241 +**Key Point**: The order fields are evaluated doesn't matter - the same counters are updated and the same decision is made regardless of whether positive or negative matches are found first.
242 +
243 +**Common Use Cases:**
244 +- `"timeout|failed|refused"` - Find various connection issues
245 +- `"!trace|!debug|nginx|apache"` - Find web server logs, but exclude debug/trace
246 +- `"error code 500"` - Find exact phrase with spaces
247 +- `"critical|fatal|emergency"` - Find severe log levels
248 +- `"!*test*|!*debug*|*"` - Include everything except test and debug content
249 +
250 +**Implementation Details:**
251 +- Uses `simple_pattern_create(query, "|", SIMPLE_PATTERN_SUBSTRING, false)`
252 +- Searches all fields marked with `FACET_KEY_OPTION_FTS` or when `FACETS_OPTION_ALL_KEYS_FTS` is set
253 +- No support for `?` single-character wildcards
254 +- Escaping with `\` is supported for literal matches
255 +
256 +**Important**: Hash IDs (like `priority_hash`) are obsolete and only exist for backward compatibility with GET requests. All new functions should use v:3 with plain field names.
257 +
258 +## Simple Table Format
259 +
260 +### Complete Structure
261 +
262 +```json
263 +{
264 + // Required fields
265 + "status": 200,
266 + "type": "table",
267 + "has_history": false,
268 + "data": [
269 + [value1, value2, value3, ...], // Row 1
270 + [value1, value2, value3, ...], // Row 2
271 + // Special rowOptions as last element
272 + [..., {"rowOptions": {"severity": "warning|error|notice|normal"}}]
273 + ],
274 + "columns": {
275 + "column_name": {
276 + // Required
277 + "index": 0, // Position in data array
278 + "name": "Display Name", // Column header
279 + "type": "string", // Field type
280 +
281 + // Optional
282 + "unique_key": false, // Row identifier (exactly one required)
283 + "visible": true, // Default visibility
284 + "sticky": false, // Pin when scrolling
285 + "visualization": "value", // How to render
286 + "transform": "none", // Value transformation
287 + "decimal_points": 2, // For numbers
288 + "units": "bytes", // Display units
289 + "max": 100, // For bar types
290 + "sort": "descending", // Default sort
291 + "sortable": true, // User can sort
292 + "filter": "multiselect", // Filter type
293 + "full_width": false, // Expand to fill
294 + "wrap": false, // Text wrapping
295 + "summary": "sum" // Aggregation (backend only)
296 + }
297 + },
298 +
299 + // Optional extensions
300 + "help": "Function description",
301 + "update_every": 1,
302 + "expires": 1234567890,
303 + "default_sort_column": "column_name",
304 + "group_by": {
305 + "aggregated": [{
306 + "id": "group_id",
307 + "name": "Group Name",
308 + "column": "column_to_group_by"
309 + }]
310 + },
311 + "charts": {
312 + "chart_id": {
313 + "name": "Chart Name",
314 + "type": "stacked|bar",
315 + "columns": ["col1", "col2"]
316 + }
317 + },
318 + "accepted_params": [{
319 + "id": "param_id",
320 + "name": "Parameter Name",
321 + "type": "string|integer|boolean",
322 + "default": "default_value"
323 + }],
324 + "required_params": ["param1", "param2"]
325 +}
326 +```
327 +
328 +### Row Options
329 +
330 +Special last element in data array for row styling:
331 +
332 +```json
333 +[..., {"rowOptions": {"severity": "error"}}] // Red background
334 +[..., {"rowOptions": {"severity": "warning"}}] // Yellow background
335 +[..., {"rowOptions": {"severity": "notice"}}] // Blue background
336 +[..., {"rowOptions": {"severity": "normal"}}] // Default appearance
337 +```
338 +
339 +## Log Explorer Format
340 +
341 +### Complete Structure
342 +
343 +```json
344 +{
345 + // Basic fields (same as simple table)
346 + "status": 200,
347 + "type": "table",
348 + "has_history": true, // REQUIRED: Enables log explorer UI
349 + "help": "System log explorer",
350 + "update_every": 1,
351 +
352 + // Table metadata
353 + "table": {
354 + "id": "logs",
355 + "has_history": true,
356 + "pin_alert": false
357 + },
358 +
359 + // Faceted filters (dynamic with counts)
360 + "facets": [
361 + {
362 + "id": "priority", // Plain field name (not hash)
363 + "name": "Priority",
364 + "order": 1,
365 + "defaultExpanded": true,
366 + "options": [
367 + {
368 + "id": "ERROR",
369 + "name": "ERROR",
370 + "count": 45, // Real-time count
371 + "order": 1
372 + }
373 + ]
374 + }
375 + ],
376 +
377 + // Enhanced columns
378 + "columns": {
379 + "timestamp": {
380 + "index": 0,
381 + "id": "timestamp",
382 + "name": "Time",
383 + "type": "timestamp",
384 + "transform": "datetime_usec",
385 + "sort": "descending|fixed",
386 + "sortable": false,
387 + "sticky": true
388 + },
389 + "level": {
390 + "index": 1,
391 + "id": "priority", // Links to facet
392 + "name": "Level",
393 + "type": "string",
394 + "visualization": "pill",
395 + "filter": "facet", // Not multiselect!
396 + "options": ["facet", "visible", "sticky"]
397 + },
398 + "message": {
399 + "index": 3,
400 + "id": "message",
401 + "name": "Message",
402 + "type": "string",
403 + "full_width": true,
404 + "options": [
405 + "full_width",
406 + "wrap",
407 + "visible",
408 + "main_text", // Primary content
409 + "fts", // Full-text searchable
410 + "rich_text" // May contain formatting
411 + ]
412 + }
413 + },
414 +
415 + // Data with microsecond timestamps
416 + "data": [
417 + [
418 + 1697644320000000, // Microseconds
419 + {"severity": "error"}, // rowOptions
420 + "ERROR", // level
421 + "nginx", // source
422 + "Connection failed" // message
423 + ]
424 + ],
425 +
426 + // Histogram configuration
427 + "available_histograms": [
428 + {"id": "priority", "name": "Priority", "order": 1},
429 + {"id": "source", "name": "Source", "order": 2}
430 + ],
431 + "histogram": {
432 + "id": "priority",
433 + "name": "Priority",
434 + "chart": {
435 + "summary": {/* Netdata chart metadata */},
436 + "result": {
437 + "labels": ["time", "ERROR", "WARN", "INFO"],
438 + "data": [
439 + [1697644200, 5, 12, 234],
440 + [1697644260, 3, 8, 198]
441 + ]
442 + }
443 + }
444 + },
445 +
446 + // Pagination metadata
447 + "items": {
448 + "evaluated": 50000, // Total scanned
449 + "matched": 2520, // Match filters
450 + "unsampled": 100, // Skipped (sampling)
451 + "estimated": 0, // Statistical estimate
452 + "returned": 100, // In this response
453 + "max_to_return": 100,
454 + "before": 0,
455 + "after": 2420
456 + },
457 +
458 + // Navigation anchor
459 + "anchor": {
460 + "last_modified": 1697644320000000,
461 + "direction": "backward" // or "forward"
462 + },
463 +
464 + // Request echo (optional)
465 + "request": {
466 + "query": "*error* *warning*",
467 + "filters": ["priority:error,warning"],
468 + "histogram": "priority"
469 + },
470 +
471 + // Additional metadata
472 + "expires": 1697644920000,
473 + "sampling": 10 // 1 in N sampling
474 +}
475 +```
476 +
477 +### Key Differences from Simple Tables
478 +
479 +| Feature | Simple Table | Log Explorer |
480 +|---------|--------------|--------------|
481 +| **Facet counts** | Frontend computes from data | Backend computes in facets library |
482 +| **Full-text search** | Frontend substring matching | Backend pattern matching |
483 +| **Histograms** | Not supported | Optional (backend generated) |
484 +| Filtering | Static multiselect | Dynamic facets with counts |
485 +| Pagination | All data at once | Anchor-based infinite scroll |
486 +| Time visualization | None | Histogram chart |
487 +| Navigation | None | Bi-directional with timestamps |
488 +| Performance | All data loaded | Sampling for large datasets |
489 +
490 +### Log-Specific Column Options
491 +
492 +| Option | UI Effect |
493 +|--------|-----------|
494 +| `"facet"` | Field is filterable via facets |
495 +| `"fts"` | Full-text searchable |
496 +| `"main_text"` | Primary content field |
497 +| `"rich_text"` | May contain formatting |
498 +| `"pretty_xml"` | Format as XML |
499 +| `"hidden"` | Hide by default |
500 +
501 +## Facet Value Pills and Aggregated Counts
502 +
503 +### Overview
504 +
505 +Facet values in the sidebar display pills with counts that change based on the function's aggregation mode. The UI uses a smart component that displays different formats:
506 +
507 +1. **Simple Counts**: Just the number of matching rows
508 +2. **Aggregated Counts**: Shows both aggregated count and original count with a union symbol
509 +
510 +### UI Implementation
511 +
512 +The pills are rendered using this logic:
513 +
514 +```javascript
515 +{!!actualCount && <TextSmall>{actualCount} &#8835;&nbsp;</TextSmall>}
516 +<TextSmall>{(pill || count).toString()}</TextSmall>
517 +```
518 +
519 +**Symbol**: `&#8835;` renders as `⊃` (superset symbol, looks like rotated 'u')
520 +
521 +**Display Formats**:
522 +- Simple: `"42"` (just the count)
523 +- Aggregated: `"15 ⊃ 42"` (15 aggregated items containing 42 total)
524 +
525 +### Backend Configuration
526 +
527 +Functions enable aggregated counts by including an `aggregated_view` object in their response:
528 +
529 +```json
530 +{
531 + "aggregated_view": {
532 + "column": "Count",
533 + "results_label": "unique combinations",
534 + "aggregated_label": "sockets"
535 + }
536 +}
537 +```
538 +
539 +**Example from network-connections function**:
540 +```c
541 +// In network-viewer.c when aggregated=true
542 +buffer_json_member_add_object(wb, "aggregated_view");
543 +{
544 + buffer_json_member_add_string(wb, "column", "Count");
545 + buffer_json_member_add_string(wb, "results_label", "unique combinations");
546 + buffer_json_member_add_string(wb, "aggregated_label", "sockets");
547 +}
548 +buffer_json_object_close(wb);
549 +```
550 +
551 +## Charts Configuration
552 +
553 +Functions can provide both standard charts (computed by frontend) and custom visualizations.
554 +
555 +### Standard Charts
556 +
557 +**Configuration:**
558 +```json
559 +{
560 + "charts": {
561 + "cpu_usage": {
562 + "name": "CPU Usage by Service",
563 + "type": "stacked-bar",
564 + "columns": ["user_cpu", "system_cpu"],
565 + "groupBy": "column",
566 + "aggregation": "sum"
567 + }
568 + },
569 + "default_charts": [
570 + ["cpu_usage", "service_type"]
571 + ]
572 +}
573 +```
574 +
575 +**Supported Types:**
576 +- `"bar"` - Basic bar chart
577 +- `"stacked-bar"` - Multi-column stacked bars
578 +- `"doughnut"` - Pie/doughnut chart
579 +- `"value"` - Simple numeric display
580 +
581 +**GroupBy Options:**
582 +- `"column"` (default) - Group by selected filter column
583 +- `"all"` - Aggregate all data together
584 +
585 +### Custom Charts
586 +
587 +For specialized visualizations beyond standard chart types, some functions may use predefined custom chart types:
588 +
589 +```json
590 +{
591 + "customCharts": {
592 + "network_topology": {
593 + "type": "network-viewer",
594 + "config": {
595 + "layout": "force",
596 + "showLabels": true
597 + }
598 + }
599 + }
600 +}
601 +```
602 +
603 +**Available Custom Types:**
604 +- `"network-viewer"` - Interactive network topology (for network-connections function)
605 +
606 +### Frontend Processing
607 +
608 +Charts are computed from table data:
609 +1. Frontend groups data by selected column
610 +2. Applies aggregation function (sum, count, avg, etc.)
611 +3. Renders chart with grouped results
612 +
613 +### Table Row Grouping
614 +
615 +Simple tables support grouping rows with backend-defined aggregation rules.
616 +
617 +**Backend Summary Types:**
618 +```c
619 +typedef enum {
620 + RRDF_FIELD_SUMMARY_COUNT, // Count rows in group
621 + RRDF_FIELD_SUMMARY_UNIQUECOUNT, // Count unique values
622 + RRDF_FIELD_SUMMARY_SUM, // Sum numeric values
623 + RRDF_FIELD_SUMMARY_MIN, // Minimum value
624 + RRDF_FIELD_SUMMARY_MAX, // Maximum value
625 + RRDF_FIELD_SUMMARY_MEAN, // Average value
626 + RRDF_FIELD_SUMMARY_MEDIAN, // Median value
627 +} RRDF_FIELD_SUMMARY;
628 +```
629 +
630 +**Column Summary Configuration:**
631 +```c
632 +buffer_rrdf_table_add_field(
633 + wb, field_id++, "cpu", "CPU Usage",
634 + RRDF_FIELD_TYPE_INTEGER,
635 + RRDF_FIELD_VISUAL_VALUE,
636 + RRDF_FIELD_TRANSFORM_NUMBER,
637 + 2, "%", NAN, RRDF_FIELD_SORT_DESCENDING, NULL,
638 + RRDF_FIELD_SUMMARY_SUM, // How to aggregate when grouping
639 + RRDF_FIELD_FILTER_RANGE,
640 + RRDF_FIELD_OPTS_VISIBLE, NULL
641 +);
642 +```
643 +
644 +**Group By Support:**
645 +```json
646 +{
647 + "group_by": {
648 + "aggregated": [{
649 + "id": "by_status",
650 + "name": "By Status",
651 + "column": "status"
652 + }]
653 + }
654 +}
655 +```
656 +
657 +### Frontend Processing
658 +
659 +The frontend processes table data to generate facet counts:
660 +
661 +```javascript
662 +const getFilterTableOptions = (data, { param, columns, aggregatedView } = {}) =>
663 + Object.entries(
664 + data.reduce((h, fn) => {
665 + h[fn[param]] = {
666 + count: (h[fn[param]]?.count || 0) + (fn.hidden ? 0 : 1),
667 + ...(aggregatedView && {
668 + actualCount: (h[fn[param]]?.actualCount || 0) +
669 + (fn.hidden ? 0 : fn[aggregatedView.column] || 1),
670 + actualCountLabel: aggregatedView.aggregatedLabel,
671 + countLabel: aggregatedView.resultsLabel,
672 + }),
673 + }
674 + return h
675 + }, {})
676 + ).map(([id, values]) => ({ id, ...values }))
677 +```
678 +
679 +### How It Works
680 +
681 +The aggregated count system uses an existing data column to track how many items were aggregated into each row:
682 +
683 +1. **Backend declares** which column contains the aggregation count:
684 + ```json
685 + "aggregated_view": {
686 + "column": "Count" // Use the "Count" column from data rows
687 + }
688 + ```
689 +
690 +2. **Frontend calculates** two values for each facet value:
691 + - **`count`**: How many rows have this facet value
692 + - **`actualCount`**: Sum of the aggregation column for all rows with this facet value
693 +
694 +3. **Example**: Network connections aggregated by protocol
695 +
696 + **Data returned by backend:**
697 + ```
698 + Direction | Protocol | LocalPort | RemotePort | Count
699 + ---------|----------|-----------|------------|-------
700 + Inbound | TCP | * | * | 5
701 + Inbound | UDP | * | * | 3
702 + Outbound | TCP | * | * | 7
703 + Outbound | UDP | * | * | 2
704 + ```
705 +
706 + **Facet pills displayed in UI:**
707 + - Direction facet:
708 + - Inbound: `8 ⊃ 2` (8 connections aggregated into 2 rows)
709 + - Outbound: `9 ⊃ 2` (9 connections aggregated into 2 rows)
710 + - Protocol facet:
711 + - TCP: `12 ⊃ 2` (12 connections aggregated into 2 rows)
712 + - UDP: `5 ⊃ 2` (5 connections aggregated into 2 rows)
713 +
714 + **Reading the pills**: `12 ⊃ 2` means "12 original items shown in 2 table rows"
715 +
716 +### Tooltip Content
717 +
718 +The tooltip provides human-readable context:
719 +- Simple mode: `"42 results"`
720 +- Aggregated mode: `"15 sockets aggregated in 42 unique combinations"`
721 +
722 +### Current Implementations
723 +
724 +| Function | Aggregated Mode | Trigger | Count Meaning |
725 +|----------|----------------|---------|---------------|
726 +| `network-connections` | `sockets:aggregated` | Parameter | Sockets → unique combinations |
727 +| Other functions | N/A | None currently | Single count only |
728 +
729 +### Adding Aggregated Counts
730 +
731 +To add aggregated count support to a function:
732 +
733 +1. **Backend**: Add `aggregated_view` object to response when in aggregated mode
734 +2. **Data**: Include aggregation column with numeric values
735 +3. **Frontend**: No changes needed - automatically processes based on `aggregated_view` presence
736 +
737 +## Field Types and Enumerations
738 +
739 +### Field Types (RRDF_FIELD_TYPE_*)
740 +
741 +#### Implemented in UI
742 +
743 +| Type | UI Component | Use Case | Notes |
744 +|------|--------------|----------|-------|
745 +| `string` | ValueCell | Text, names, categories | Left-aligned |
746 +| `integer` | ValueCell | Numbers, counts, IDs | Right-aligned |
747 +| `bar-with-integer` | BarCell | Percentages, metrics | Requires `max` |
748 +| `duration` | BarCell | Time intervals | Auto-formats seconds |
749 +| `timestamp` | DatetimeCell* | Date/time points | *UI maps to datetime |
750 +| `feedTemplate` | FeedTemplateCell | Rich content | Auto full_width |
751 +
752 +#### Fallback Implementation
753 +
754 +| Type | Behavior | Notes |
755 +|------|----------|-------|
756 +| `boolean` | ValueCell | No special boolean UI |
757 +| `detail-string` | ValueCell | No expandable functionality |
758 +| `array` | ValueCell | Works with `pill` visualization |
759 +| `none` | ValueCell | Avoid using |
760 +
761 +### Visual Types (RRDF_FIELD_VISUAL_*)
762 +
763 +| Type | UI Component | Use Case |
764 +|------|--------------|----------|
765 +| `value` | ValueCell | Standard text display (default) |
766 +| `bar` | BarCell | Progress bar without text |
767 +| `pill` | PillCell | Badge/tag display |
768 +| `richValue` | RichValueCell | Enhanced value display |
769 +| `feedTemplate` | FeedTemplateCell | Full-width template |
770 +| `rowOptions` | null | Special row configuration |
771 +
772 +**Fallback Behavior**: `gauge` is not a recognized visualization. If used, it is ignored, and the renderer falls back to using the field's `type` to select a component.
773 +
774 +### Transform Types (RRDF_FIELD_TRANSFORM_*)
775 +
776 +| Type | Input | Output | Notes |
777 +|------|-------|--------|-------|
778 +| `none` | Any | Unchanged | Default |
779 +| `number` | Number | Formatted with decimals | Uses `decimal_points` |
780 +| `duration` | Seconds | "Xd Yh Zm" | Human-readable |
781 +| `datetime` | Epoch ms | Localized date/time | |
782 +| `datetime_usec` | Epoch μs | Localized date/time | For logs |
783 +| `xml` | XML string | Formatted XML | No specialized UI |
784 +
785 +### Conditional Patterns and Dependencies
786 +
787 +#### Type and Transform Compatibility
788 +
789 +Not all `transform` values are compatible with all `type` values. The backend enforces the following compatibility rules:
790 +
791 +| Field Type (`type`) | Compatible Transforms (`transform`) |
792 +|---|---|
793 +| `timestamp` | `datetime_ms`, `datetime_usec` |
794 +| `duration` | `duration_s` |
795 +| `integer`, `bar-with-integer` | `number` |
796 +| `string`, `boolean`, `array` | `none`, `xml` |
797 +
798 +Using an incompatible transform will result in unexpected behavior or errors.
799 +
800 +#### `rowOptions` Dummy Column
801 +
802 +To add `rowOptions` for row-level severity styling, a special "dummy" column must be added to the `columns` definition. This column is not displayed in the UI but provides the necessary metadata. It must be created with this specific combination of values:
803 +
804 +* **type**: `none` (`RRDF_FIELD_TYPE_NONE`)
805 +* **visualization**: `rowOptions` (`RRDF_FIELD_VISUAL_ROW_OPTIONS`)
806 +* **options flag**: `dummy` (`RRDF_FIELD_OPTS_DUMMY`)
807 +
808 +**Example C code:**
809 +```c
810 +buffer_rrdf_table_add_field(wb, field_id++, "row_options", "Row Options",
811 + RRDF_FIELD_TYPE_NONE, RRDF_FIELD_VISUAL_ROW_OPTIONS, RRDF_FIELD_TRANSFORM_NONE,
812 + 0, NULL, NAN, RRDF_FIELD_SORT_ASCENDING, NULL,
813 + RRDF_FIELD_SUMMARY_COUNT, RRDF_FIELD_FILTER_NONE, RRDF_FIELD_OPTS_DUMMY, NULL);
814 +```
815 +
816 +### Filter Types (RRDF_FIELD_FILTER_*)
817 +
818 +| Type | UI Component | Use Case | Location |
819 +|------|--------------|----------|----------|
820 +| `multiselect` | Checkboxes | Column filtering (default) | Dynamic filters |
821 +| `range` | RangeFilter | Numeric min/max | Dynamic filters |
822 +| `facet` | Facets component | With counts (logs) | Sidebar |
823 +
824 +### Field Options (RRDF_FIELD_OPTS_*)
825 +
826 +| Option | Bit | UI Effect |
827 +|--------|-----|-----------|
828 +| `unique_key` | 0x01 | Row identifier (one required) |
829 +| `visible` | 0x02 | Show by default |
830 +| `sticky` | 0x04 | Pin column when scrolling |
831 +| `full_width` | 0x08 | Expand to fill space |
832 +| `wrap` | 0x10 | Enable text wrapping |
833 +| `dummy` | 0x20 | Internal use only |
834 +| `expanded_filter` | 0x40 | Expand filter by default |
835 +
836 +### Sort Options (RRDF_FIELD_SORT_*)
837 +
838 +- `ascending` - Sort low to high
839 +- `descending` - Sort high to low
840 +- Fixed sort (0x80) - Prevent user sorting
841 +
842 +### Summary Types (RRDF_FIELD_SUMMARY_*)
843 +
844 +Backend calculates but UI doesn't display directly:
845 +- `count`, `sum`, `min`, `max`, `mean`, `median`
846 +- `uniqueCount` - Number of unique values
847 +- `extent` - Range [min, max] (UI supported)
848 +- `unique` - List of unique values (UI supported)
849 +
850 +## UI Implementation
851 +
852 +### Column Width System
853 +
854 +The UI automatically sizes columns based on metadata:
855 +
856 +| Size | Pixels | Applied To |
857 +|------|--------|------------|
858 +| xxxs | 90px | unique_key fields, bar types |
859 +| xxs | 110px | Default for most types |
860 +| xs | 130px | Available but rarely used |
861 +| sm | 160px | timestamp, datetime types |
862 +| md-xl | 190-290px | Available but rarely used |
863 +| xxl | 1000px | feedTemplate with full_width |
864 +
865 +**Algorithm**:
866 +1. If `full_width`: → xxl with expansion
867 +2. If `unique_key`: → xxxs
868 +3. By visualization: bar → xxxs
869 +4. By type: feedTemplate → xxl, timestamp → sm
870 +5. Default → xxs
871 +
872 +### Component Mapping
873 +
874 +```javascript
875 +// Field type → Component
876 +componentByType = {
877 + "bar": BarCell,
878 + "bar-with-integer": BarCell,
879 + "duration": BarCell,
880 + "pill": PillCell,
881 + "feedTemplate": FeedTemplateCell,
882 + "datetime": DatetimeCell,
883 + // Others → ValueCell
884 +}
885 +
886 +// Visualization → Component
887 +componentByVisualization = {
888 + "bar": BarCell,
889 + "pill": PillCell,
890 + "richValue": RichValueCell,
891 + "feedTemplate": FeedTemplateCell,
892 + "rowOptions": null, // Skip rendering
893 + // Others → ValueCell
894 +}
895 +```
896 +
897 +### UI Component Architecture
898 +
899 +The frontend uses a modular architecture with:
900 +- **Value Components**: Handle different field type rendering
901 +- **Table Normalizer**: Processes function responses into UI-ready format
902 +- **Filter Components**: Implement multiselect, range, and facet filtering
903 +- **Chart Components**: Standard chart rendering (bar, stacked-bar, doughnut)
904 +- **Custom Visualizations**: Extensible system for specialized charts
905 +
906 +## Backend Implementation
907 +
908 +### Key Functions
909 +
910 +```c
911 +// Add a field to the table
912 +buffer_rrdf_table_add_field(
913 + BUFFER *wb,
914 + size_t field_id,
915 + const char *key,
916 + const char *name,
917 + RRDF_FIELD_TYPE type,
918 + RRDF_FIELD_VISUAL visual,
919 + RRDF_FIELD_TRANSFORM transform,
920 + size_t decimal_points,
921 + const char *units,
922 + NETDATA_DOUBLE max,
923 + RRDF_FIELD_SORT sort,
924 + const char *pointer_to_dim_in_rrdr,
925 + RRDF_FIELD_SUMMARY summary,
926 + RRDF_FIELD_FILTER filter,
927 + RRDF_FIELD_OPTS options,
928 + const char *default_value
929 +);
930 +```
931 +
932 +### Real Examples
933 +
934 +```c
935 +// CPU usage with progress bar
936 +buffer_rrdf_table_add_field(
937 + wb, field_id++, "CPU", "CPU %",
938 + RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
939 + RRDF_FIELD_VISUAL_BAR,
940 + RRDF_FIELD_TRANSFORM_NUMBER,
941 + 2, "%", 100.0, RRDF_FIELD_SORT_DESCENDING, NULL,
942 + RRDF_FIELD_SUMMARY_SUM,
943 + RRDF_FIELD_FILTER_RANGE,
944 + RRDF_FIELD_OPTS_VISIBLE, NULL
945 +);
946 +
947 +// Process name (unique key)
948 +buffer_rrdf_table_add_field(
949 + wb, field_id++, "Name", "Name",
950 + RRDF_FIELD_TYPE_STRING,
951 + RRDF_FIELD_VISUAL_VALUE,
952 + RRDF_FIELD_TRANSFORM_NONE,
953 + 0, NULL, NAN, RRDF_FIELD_SORT_ASCENDING, NULL,
954 + RRDF_FIELD_SUMMARY_COUNT,
955 + RRDF_FIELD_FILTER_MULTISELECT,
956 + RRDF_FIELD_OPTS_VISIBLE | RRDF_FIELD_OPTS_STICKY | RRDF_FIELD_OPTS_UNIQUE_KEY,
957 + NULL
958 +);
959 +
960 +// Row options in data
961 +buffer_json_add_array_item_object(wb);
962 +buffer_json_member_add_object(wb, "rowOptions");
963 +buffer_json_member_add_string(wb, "severity", "error");
964 +buffer_json_object_close(wb);
965 +buffer_json_object_close(wb);
966 +```
967 +
968 +### Key Backend Files
969 +
970 +- **Enums**: `netdata/src/libnetdata/buffer/functions_fields.h`
971 +- **Implementation**: `netdata/src/libnetdata/buffer/functions_fields.c`
972 +- **Facets Library**: `netdata/src/libnetdata/facets/`
973 +- **Example Functions**:
974 + - `apps.plugin/apps_functions.c`
975 + - `network-viewer.plugin/network-viewer.c`
976 + - `systemd-journal.plugin/systemd-journal.c`
977 +
978 +## Best Practices
979 +
980 +### Always Include
981 +- One field with `unique_key` option
982 +- Meaningful `name` for display
983 +- Appropriate `type` for the data
984 +- Info response with `"v": 3`
985 +
986 +### For Numeric Data
987 +- Use `bar-with-integer` for percentages
988 +- Set appropriate `max` value
989 +- Include `units` for clarity
990 +- Use `range` filter
991 +
992 +### For Time Data
993 +- `duration` type for intervals
994 +- `timestamp` type for points in time
995 +- Use appropriate transform
996 +
997 +### For Status/Severity
998 +- Include `rowOptions` as last array element
999 +- Use standard severity levels: error, warning, notice, normal
1000 +
1001 +### For Filtering
1002 +- `multiselect` (default) for categories
1003 +- `range` for numeric data
1004 +- `facet` only for has_history=true
1005 +
1006 +### Common Patterns
1007 +- Numeric metrics → `bar-with-integer` + `range` filter
1008 +- Categories → `string` + `multiselect` filter
1009 +- Time intervals → `duration` + `duration` transform
1010 +- Status → `string` + `pill` visualization
1011 +- Row coloring → `rowOptions` with severity
1012 +
1013 +## Known Functions
1014 +
1015 +### Simple Table Functions (has_history=false)
1016 +
1017 +| Function | Plugin | Key Features |
1018 +|----------|--------|--------------|
1019 +| processes | apps.plugin | CPU/memory bars, grouping |
1020 +| socket | ebpf.plugin | Complex filters, charts |
1021 +| network-connections | network-viewer.plugin | Aggregated views, severity |
1022 +| systemd-list-units | systemd-units.plugin | Unit status, severity |
1023 +| ipmi-sensors | freeipmi.plugin | Hardware monitoring |
1024 +| block-devices | proc.plugin | I/O statistics, charts |
1025 +| network-interfaces | proc.plugin | Network stats, severity |
1026 +| mount-points | diskspace.plugin | Filesystem usage |
1027 +| cgroup-top | cgroups.plugin | Container metrics |
1028 +| systemd-top | cgroups.plugin | Service metrics |
1029 +| metrics-cardinality | web api | Dynamic columns |
1030 +| streaming | web api | Replication status |
1031 +| all-queries | web api | Monitors the progress of in-flight queries |
1032 +
1033 +### Log Explorer Functions (has_history=true)
1034 +
1035 +| Function | Plugin | Key Features |
1036 +|----------|--------|--------------|
1037 +| systemd-journal | systemd-journal.plugin | System logs, faceted search |
1038 +| windows-events | windows-events.plugin | Windows logs, faceted search |
1039 +
1040 +
1041 +## Development Checklist
1042 +
1043 +### Creating a New Function
1044 +
1045 +- [ ] Choose function type (simple table or log explorer)
1046 +- [ ] Implement info response with `"v": 3`
1047 +- [ ] Define columns with appropriate types and options
1048 +- [ ] Include one `unique_key` field
1049 +- [ ] Add proper error handling
1050 +- [ ] Test with POST requests
1051 +- [ ] Document accepted/required parameters
1052 +
1053 +### Format Validation
1054 +
1055 +- [ ] Verify JSON structure matches specification
1056 +- [ ] Check all required fields are present
1057 +- [ ] Test empty result sets
1058 +- [ ] Test large datasets
1059 +- [ ] Verify error responses
1060 +- [ ] Test special characters in data
1061 +- [ ] Check numeric precision
1062 +- [ ] Verify date/time formatting
1063 +- [ ] Test sorting and filtering
1064 +
1065 +### UI Integration
1066 +
1067 +- [ ] Confirm field types map to UI components
1068 +- [ ] Verify filters work correctly
1069 +- [ ] Check column widths display properly
1070 +- [ ] Test row coloring with severity
1071 +- [ ] Verify transforms apply correctly
1072 +- [ ] Check responsive behavior
1073 +
1074 +### Performance
1075 +
1076 +- [ ] Handle large datasets efficiently
1077 +- [ ] Implement sampling for logs if needed
1078 +- [ ] Set appropriate `update_every`
1079 +- [ ] Consider pagination/anchoring
1080 +- [ ] Test with concurrent requests
1081 +
1082 +## Corner Cases and Edge Handling
1083 +
1084 +### Empty Result Sets
1085 +
1086 +The protocol handles empty results gracefully:
1087 +
1088 +```json
1089 +{
1090 + "status": 200,
1091 + "type": "table",
1092 + "has_history": false,
1093 + "columns": {...}, // Full column definitions
1094 + "data": [] // Empty array is valid
1095 +}
1096 +```
1097 +
1098 +- Minimum valid response: `{"status": 200, "type": "table", "columns": {}, "data": []}`
1099 +- UI displays "No data available" message
1100 +- Column headers still render for context
1101 +
1102 +### Null and Missing Values
1103 +
1104 +- `null` values in data arrays are rendered as empty cells
1105 +- Missing array elements default to `null`
1106 +- `NaN` for numeric fields: For fields with `transform: "number"`, `NaN` values will be displayed as the string "NaN". For `timestamp` fields with `datetime` or `datetime_usec` transforms, `NaN` epoch values will display as empty cells.
1107 +- Empty strings render as empty cells
1108 +- Backend uses `NAN` constant for missing numeric values
1109 +
1110 +### Special Characters
1111 +
1112 +The protocol properly escapes:
1113 +- JSON special characters (`"`, `\`, control chars)
1114 +- HTML entities (escaped by React components): React automatically escapes HTML content to prevent XSS attacks. Any HTML tags or entities in the data will be displayed as literal text, not rendered as HTML.
1115 +- Unicode characters (UTF-8 support throughout)
1116 +- SQL injection protection in queries
1117 +
1118 +### Numeric Precision
1119 +
1120 +- `decimal_points` field controls display precision, using JavaScript's `toFixed()` method.
1121 +- Backend uses `NETDATA_DOUBLE` type for floating-point numbers.
1122 +- Frontend's number formatting:
1123 + - Some components use `Intl.NumberFormat` with a fixed locale (e.g., "en-US").
1124 + - Others use `toLocaleString()` which respects the browser's locale.
1125 + - Therefore, locale-specific formatting is applied in some, but not all, cases.
1126 +- Very large numbers: Due to the use of `toFixed()`, very large numbers will be displayed as a long string with the specified decimal places, not automatically in scientific notation.
1127 +- Infinity/NaN handling:
1128 + - `NaN` values in numeric fields will be displayed as the string "NaN".
1129 + - `Infinity` and `-Infinity` values will be displayed as the strings "Infinity" and "-Infinity" respectively.
1130 + - For `timestamp` fields with `datetime` or `datetime_usec` transforms, `NaN` epoch values will display as empty cells.
1131 +
1132 +### Timestamp Format Requirements
1133 +
1134 +**Simple Tables**: Use milliseconds with `datetime` transform
1135 +```json
1136 +{
1137 + "type": "timestamp",
1138 + "transform": "datetime", // Expects milliseconds
1139 + "data": [1697644320000] // JavaScript Date format
1140 +}
1141 +```
1142 +
1143 +**Log Explorers**: Use microseconds with `datetime_usec` transform
1144 +```json
1145 +{
1146 + "type": "timestamp",
1147 + "transform": "datetime_usec", // Expects microseconds
1148 + "data": [1697644320000000] // Microsecond precision
1149 +}
1150 +```
1151 +
1152 +**Frontend Conversion:**
1153 +```javascript
1154 +// datetime_usec automatically converts to milliseconds
1155 +if (usec) {
1156 + epoch = epoch ? Math.floor(epoch / 1000) : epoch
1157 +}
1158 +```
1159 +
1160 +**API Parameters**: `after` and `before` are automatically converted from milliseconds to seconds when sent to functions.
1161 +
1162 +**Display Format:**
1163 +- Timezone: Determined by URL parameter (`utc`) or system default
1164 +- Format: Uses `Intl.DateTimeFormat` with browser's locale
1165 +- Both types render as localized date/time strings with seconds precision
1166 +
1167 +### Sorting Capabilities
1168 +
1169 +- **Backend Control**: Each column defines default sort with `RRDF_FIELD_SORT_*`
1170 +- **User Control**: `sortable: true` enables UI sorting (default)
1171 +- **Fixed Sort**: Bit flag 0x80 prevents user sorting
1172 +- **Initial Sort**: `default_sort_column` specifies startup sort
1173 +- **Multi-Column**: UI supports sorting by any sortable column
1174 +- **Performance**: Client-side sorting for simple tables
1175 +
1176 +### Filtering Capabilities
1177 +
1178 +- **Multiselect**: Default filter type with checkboxes
1179 + - Shows all unique values from the column
1180 + - Multiple selections allowed
1181 + - OR logic between selections
1182 +- **Range**: Numeric filters with min/max sliders
1183 + - Requires numeric field type
1184 + - Auto-detects min/max from data
1185 + - Inclusive filtering
1186 +- **Facet**: Advanced filtering for log explorer
1187 + - Shows counts next to each option
1188 + - Dynamic updates with other filters
1189 + - Indexed for performance
1190 +
1191 +## Anchor-Based Pagination
1192 +
1193 +Log explorer functions use anchor-based pagination for efficient navigation through large datasets.
1194 +
1195 +### Configuration
1196 +
1197 +```json
1198 +{
1199 + "pagination": {
1200 + "enabled": true,
1201 + "column": "timestamp",
1202 + "key": "anchor",
1203 + "units": "timestamp_usec"
1204 + }
1205 +}
1206 +```
1207 +
1208 +### Frontend Implementation
1209 +
1210 +**Anchor Management:**
1211 +```javascript
1212 +// Frontend calculates anchors from data boundaries
1213 +anchorBefore: latestData[latestData.length - 1][pagination.column],
1214 +anchorAfter: latestData[0][pagination.column],
1215 +anchorUnits: pagination.units
1216 +```
1217 +
1218 +**Infinite Scroll Navigation:**
1219 +- **Backward**: Scroll down loads older data using `anchorBefore`
1220 +- **Forward**: Scroll up loads newer data using `anchorAfter`
1221 +- **State Tracking**: `hasNextPage`, `hasPrevPage` control load triggers
1222 +
1223 +**Required Parameters:**
1224 +- `anchor: {VALUE}` - Pagination cursor value
1225 +- `direction: "backward"|"forward"` - Navigation direction
1226 +- `last: NUMBER` - Page size (default: 200)
1227 +
1228 +### PLAY Mode Integration
1229 +
1230 +When in PLAY mode (`after < 0`), pagination automatically coordinates with real-time updates:
1231 +
1232 +```javascript
1233 +{
1234 + direction: "forward",
1235 + merge: true,
1236 + tail: true,
1237 + delta: true,
1238 + anchor: anchorAfter
1239 +}
1240 +```
1241 +
1242 +### Large Data Sets
1243 +
1244 +Simple tables load all data at once, but handle large sets efficiently:
1245 +- Virtual scrolling for thousands of rows
1246 +- Client-side filtering/sorting
1247 +- No built-in pagination (all data in response)
1248 +
1249 +Log explorer uses anchor-based pagination:
1250 +- Efficient navigation through millions of records
1251 +- Configurable page size (`last` parameter)
1252 +- Bi-directional infinite scrolling
1253 +- Sampling for very large sets
1254 +
1255 +## Incremental Updates (Delta Mode)
1256 +
1257 +Delta mode enables efficient real-time updates by sending only changes since the last request.
1258 +
1259 +### When Delta is Enabled
1260 +
1261 +```javascript
1262 +{
1263 + if_modified_since: 1697644320000000, // Previous modification timestamp
1264 + direction: "forward",
1265 + merge: true,
1266 + tail: true,
1267 + delta: true,
1268 + data_only: true,
1269 + anchor: anchorAfter
1270 +}
1271 +```
1272 +
1273 +### Delta Response Types
1274 +
1275 +**Facets Delta:**
1276 +```json
1277 +{
1278 + "facetsDelta": [
1279 + {
1280 + "id": "priority",
1281 + "options": [
1282 + {"id": "ERROR", "count": 5}, // Incremental counts
1283 + {"id": "WARN", "count": 12}
1284 + ]
1285 + }
1286 + ]
1287 +}
1288 +```
1289 +
1290 +**Histogram Delta:**
1291 +```json
1292 +{
1293 + "histogramDelta": {
1294 + "chart": {
1295 + "result": {
1296 + "labels": ["time", "ERROR", "WARN"],
1297 + "data": [
1298 + [1697644320, 3, 8] // New data points only
1299 + ]
1300 + }
1301 + }
1302 + }
1303 +}
1304 +```
1305 +
1306 +### Data Merging
1307 +
1308 +Frontend merges delta responses with existing data:
1309 +- **Facet counts**: Accumulated using `count = (existing || 0) + (delta || 0)`
1310 +- **Table data**: Appended/prepended based on `direction`
1311 +- **Histogram data**: New data points added to existing chart
1312 +
1313 +## Real-Time Updates (PLAY Mode)
1314 +
1315 +PLAY mode enables live data streaming with efficient polling and conditional updates.
1316 +
1317 +### PLAY Mode Detection
1318 +
1319 +- **PLAY Mode**: `after < 0` (relative time from now)
1320 +- **PAUSE Mode**: `after > 0` (absolute timestamp)
1321 +
1322 +### Parameter Coordination
1323 +
1324 +When `if_modified_since` is present, the system automatically includes:
1325 +
1326 +```json
1327 +{
1328 + "if_modified_since": 1697644320000000,
1329 + "direction": "forward",
1330 + "merge": true,
1331 + "tail": true,
1332 + "delta": true,
1333 + "data_only": true,
1334 + "anchor": "anchorAfter"
1335 +}
1336 +```
1337 +
1338 +### Error Handling
1339 +
1340 +**304 Not Modified**: Indicates no new data available
1341 +```json
1342 +{
1343 + "status": 304
1344 +}
1345 +```
1346 +
1347 +Frontend handles 304 responses gracefully without showing errors to users.
1348 +
1349 +### Polling Behavior
1350 +
1351 +- **Polling Interval**: Based on function's `update_every` value
1352 +- **Auto-Pause**: When window loses focus or user hovers over data
1353 +- **Conditional Requests**: Uses `if_modified_since` to avoid unnecessary data transfers
1354 +
1355 +### Required vs Optional Fields
1356 +
1357 +**Minimum Required Fields:**
1358 +```json
1359 +{
1360 + "status": 200, // Required
1361 + "type": "table", // Required
1362 + "columns": {}, // Required (can be empty)
1363 + "data": [] // Required (can be empty)
1364 +}
1365 +```
1366 +
1367 +**Common Optional Fields:**
1368 +- `has_history`: Default false
1369 +- `help`: Documentation text
1370 +- `update_every`: Default 1
1371 +- `expires`: Cache control
1372 +- `default_sort_column`: Initial sort
1373 +- All column options except `index`, `name`, `type`
1374 +
1375 +## Error Handling
1376 +
1377 +### Error Response Format
1378 +
1379 +When a function encounters an error, the backend returns a JSON object. The primary error generation function (`rrd_call_function_error`) produces the following minimal format:
1380 +
1381 +```json
1382 +{
1383 + "status": 400, // The HTTP status code (e.g., 400, 404, 500)
1384 + "error_message": "A descriptive error message" // A human-readable message explaining the error
1385 +}
1386 +```
1387 +
1388 +**Frontend Consumption and Interpretation:**
1389 +The frontend is designed to handle a more comprehensive error structure, allowing for richer error display and localization. When an error occurs, the frontend will attempt to extract information from the received error object using the following hierarchy:
1390 +
1391 +* **`status`**: The HTTP status code, used for general error classification (e.g., 400 for bad request, 404 for not found).
1392 +* **`error_message`**: The primary detailed message from the backend. This is the most consistently populated field by current backend functions.
1393 +* **`error`**: A short, machine-readable error identifier (e.g., "MissingParameter"). While not consistently generated by `rrd_call_function_error`, other parts of the system or future backend implementations might provide this.
1394 +* **`message`**: A user-friendly message. The frontend often maps `error_message` or an internal `errorMsgKey` to this for display.
1395 +* **`help`**: Optional additional guidance for resolving the error. This field is not currently generated by `rrd_call_function_error`.
1396 +
1397 +**Example of Frontend Interpretation (Conceptual):**
1398 +The frontend might internally map specific `error_message` strings to predefined `errorMsgKey` values to provide localized or more context-specific messages to the user. For instance, a backend `error_message` like "The 'time_range' parameter is required" might be mapped to an `errorMsgKey` of "ErrMissingTimeRange" in the frontend, which then displays a user-friendly message like "Please specify a time range for this function."
1399 +
1400 +Therefore, while the backend currently provides `status` and `error_message`, developers should be aware that the frontend's error handling is capable of utilizing the more detailed fields (`error`, `message`, `help`) if they are provided by the backend in the future or by other API endpoints.
1401 +
1402 +**Common Error Codes:**
1403 +
1404 +
1405 +### Common Error Codes
1406 +
1407 +| Code | Use Case | Example |
1408 +|------|----------|---------|
1409 +| 400 | Bad input | Missing/invalid parameters |
1410 +| 401 | Auth required | User not logged in |
1411 +| 403 | Forbidden | Insufficient permissions |
1412 +| 404 | Not found | No data matches query |
1413 +| 500 | Server error | Internal failures |
1414 +| 503 | Unavailable | Service overloaded |
1415 +
1416 +## Protocol Integration
1417 +
1418 +### PLUGINSD Commands
1419 +
1420 +Functions integrate with Netdata through the PLUGINSD protocol:
1421 +
1422 +**Registration:**
1423 +```
1424 +FUNCTION "function_name" timeout "help text" "tags" "http_access" priority version
1425 +```
1426 +
1427 +**Execution Flow:**
1428 +1. **Collector → Agent**: `FUNCTION` registers the function
1429 +2. **Agent → Collector**: `FUNCTION_CALL` with transaction ID
1430 +3. **Collector → Agent**: `FUNCTION_RESULT_BEGIN` status format expires
1431 +4. **Collector → Agent**: Response payload
1432 +5. **Collector → Agent**: `FUNCTION_RESULT_END`
1433 +
1434 +**With Payload:**
1435 +```
1436 +FUNCTION_PAYLOAD_BEGIN transaction timeout function access source content_type
1437 +<payload data>
1438 +FUNCTION_PAYLOAD_END
1439 +```
1440 +
1441 +**Cancellation:**
1442 +```
1443 +FUNCTION_CANCEL transaction_id
1444 +```
1445 +
1446 +**Progress Updates:**
1447 +```
1448 +FUNCTION_PROGRESS transaction_id done total
1449 +```
1450 +
1451 +### Streaming Protocol
1452 +
1453 +Functions support streaming for real-time updates:
1454 +
1455 +```c
1456 +// Transaction management
1457 +dictionary_set(parser->inflight.functions, transaction_str, &function_data);
1458 +
1459 +// Timeout handling
1460 +if (*pf->stop_monotonic_ut + RRDFUNCTIONS_TIMEOUT_EXTENSION_UT < now_ut) {
1461 + // Function timed out
1462 +}
1463 +
1464 +// Progress callback
1465 +if(stream_has_capability(s, STREAM_CAP_PROGRESS)) {
1466 + // Enable progress updates
1467 +}
1468 +```
1469 +
1470 +### Key Files
1471 +- `pluginsd_functions.c`: Function execution and management
1472 +- `stream-sender-execute.c`: Streaming function calls
1473 +- `plugins.d/README.md`: Protocol documentation
1474 +- `plugins.d/functions-table.md`: Table format specification
1475 +
1476 +## Migration Guide
1477 +
1478 +### Upgrading from GET to POST (v:3)
1479 +
1480 +1. Update info response to include `"v": 3`
1481 +2. Change field IDs from hashes to plain names
1482 +3. Test with JSON POST payloads
1483 +4. Remove hash generation code
1484 +5. Update documentation
1485 +
1486 +### Adding Log Explorer Features
1487 +
1488 +1. Set `has_history: true`
1489 +2. Implement facets with counts
1490 +3. Add timestamp column with microseconds
1491 +4. Support anchor-based pagination
1492 +5. Add histogram data
1493 +6. Implement full-text search
1494 +
1495 +## Status and Next Steps
1496 +
1497 +This document represents the complete v3 protocol specification. Key areas for future development:
1498 +
1499 +1. **Protocol Extensions**
1500 + - Streaming updates for real-time data
1501 + - Aggregation pipelines
1502 + - Custom visualization types
1503 +
1504 +2. **UI Enhancements**
1505 + - Additional field types (gauge, sparkline)
1506 + - Custom column renderers
1507 + - Advanced filtering options
1508 +
1509 +3. **Performance Optimizations**
1510 + - Server-side pagination for simple tables
1511 + - Incremental updates
1512 + - Result caching
1513 +
1514 +---
1515 +
1516 +## Appendix A: Validation Checklist and Progress
1517 +
1518 +*This section tracks the validation work completed and remaining tasks*
1519 +
1520 +### Format Discovery
1521 +- [x] Analyze `processes` function implementation in apps.plugin
1522 +- [x] Analyze `network-connections` function in network-viewer.plugin
1523 +- [x] Identify common patterns between implementations
1524 +- [x] Extract all enum definitions used in responses
1525 +- [x] Document each field type and possible values
1526 +- [x] Identify optional vs required fields
1527 +
1528 +### Enumeration Completeness
1529 +- [x] Find all enum definitions in netdata C code
1530 +- [x] Map enum values to their string representations
1531 +- [x] Document the purpose of each enum value
1532 +- [ ] Check for any conditional enum values
1533 +
1534 +### Function Coverage
1535 +- [x] Scan all collectors/plugins for function implementations
1536 +- [x] List all simple table functions (has_history=false)
1537 +- [x] Verify format consistency across all functions
1538 +- [x] Document any function-specific extensions
1539 +
1540 +### UI Mapping
1541 +- [x] Analyze cloud-frontend code for function rendering
1542 +- [x] Map each format field to UI component
1543 +- [x] Document how enum values affect UI behavior
1544 +- [x] Identify any frontend-specific transformations
1545 +
1546 +### Corner Cases
1547 +- [x] Empty result sets
1548 +- [x] Large data sets (pagination?)
1549 +- [x] Error responses
1550 +- [x] Null/missing values
1551 +- [x] Special characters in data
1552 +- [x] Numeric precision/formatting
1553 +- [x] Date/time formatting
1554 +- [x] Sorting capabilities
1555 +- [x] Filtering capabilities
1556 +
1557 +### Protocol Validation
1558 +- [x] Request format documentation
1559 +- [x] Response format documentation
1560 +- [x] Error handling patterns
1561 +- [x] Streaming protocol integration
1562 +
1563 +### Cross-Reference Checks
1564 +- [x] Compare documented format with actual implementations
1565 +- [x] Verify all functions conform to documented format
1566 +- [x] Check for undocumented features in UI
1567 +- [x] Validate against any existing documentation
1568 +
1569 +---
1570 +
1571 +## Appendix B: Investigation History
1572 +
1573 +### Analysis Log
1574 +
1575 +*This section documents the investigation process to avoid repeating work*
1576 +
1577 +#### Session 1 - Initial Setup and Analysis
1578 +- Created FUNCTIONS.md structure
1579 +- Established checklist for comprehensive validation
1580 +- Analyzed `processes` function in apps.plugin
1581 + - Location: `/netdata/src/collectors/apps.plugin/apps_functions.c`
1582 + - Registration: `apps_plugin.c:752`
1583 + - Uses standard table format with array-based data rows
1584 +- Analyzed `network-connections` function in network-viewer.plugin
1585 + - Location: `/netdata/src/collectors/network-viewer.plugin/network-viewer.c`
1586 + - Registration via PLUGINSD protocol
1587 + - Supports aggregated and detailed views
1588 +- Extracted all RRDF enum definitions from:
1589 + - `/netdata/src/libnetdata/buffer/functions_fields.h`
1590 + - `/netdata/src/libnetdata/buffer/functions_fields.c`
1591 +- Documented complete enum value mappings for field types, visualizations, transforms, etc.
1592 +
1593 +#### Session 2 - Protocol Integration and Corner Cases
1594 +- Investigated PLUGINSD protocol integration
1595 + - Found in `pluginsd_functions.c` and `stream-sender-execute.c`
1596 + - Commands: FUNCTION, FUNCTION_CALL, FUNCTION_RESULT_BEGIN/END
1597 + - Support for payloads, cancellation, and progress updates
1598 +- Analyzed corner case handling:
1599 + - Empty results: `{"status": 200, "type": "table", "columns": {}, "data": []}`
1600 + - Null values: Rendered as empty cells in UI
1601 + - Special characters: Proper JSON escaping throughout
1602 + - Numeric precision: Controlled by `decimal_points` field
1603 + - Date/time: Millisecond epochs for tables, microsecond for logs
1604 +- Identified required vs optional fields:
1605 + - Required: status, type, columns, data
1606 + - Optional: Everything else (has_history, help, etc.)
1607 +- Documented sorting and filtering capabilities:
1608 + - Sorting: Backend-controlled defaults, user-sortable columns
1609 + - Filtering: Multiselect (default), range (numeric), facet (logs)
1610 + - Large datasets: Virtual scrolling for tables, pagination for logs
1611 +
1612 +### Key Findings
1613 +1. **Data Format**: Simple tables use array-of-arrays for data rows
1614 +2. **Column Definition**: Each column has extensive metadata controlling display and behavior
1615 +3. **Common Functions**: Both implement `buffer_rrdf_table_add_field()` for column definitions
1616 +4. **Response Builder**: Uses `buffer_json_*` functions to build JSON responses
1617 +5. **Field Options**: Bit flags allow combining multiple options per field
1618 +6. **Protocol Evolution**: GET with hashes → POST with plain field names (v:3)
1619 +7. **UI Mapping**: Comprehensive component mapping based on type/visualization
1620 +8. **Error Handling**: Standardized error response format with HTTP codes
1621 +9. **Performance**: Client-side operations for tables, server-side for logs
1622 +10. **Extensibility**: Optional fields allow function-specific features
1623 +
1624 +### Format Compliance Summary
1625 +- **All 12 simple table functions are fully compliant** with the documented format
1626 +- **Common extensions** found:
1627 + - `rowOptions` field for row severity/status (5/12 functions)
1628 + - `charts` and `default_charts` definitions (most functions)
1629 + - `group_by` aggregation options (most functions)
1630 + - `accepted_params` and `required_params` (metrics-cardinality)
1631 + - `default_sort_column` for initial sorting
1632 +- **No breaking deviations** - all extensions are additive and optional
1633 +
1634 +---
1635 +
1636 +## Appendix C: Log Explorer UI Features Detail
1637 +
1638 +*Detailed UI behaviors when has_history=true*
1639 +
1640 +### UI Features Enabled
1641 +
1642 +When properly formatted, the log explorer UI provides:
1643 +
1644 +1. **Sidebar with Faceted Filters**
1645 + - Shows facets with real-time counts
1646 + - Multi-select filtering
1647 + - Collapsible sections
1648 + - Search within facets
1649 +
1650 +2. **Time-based Histogram**
1651 + - Visual log distribution over time
1652 + - Click and drag to select time ranges
1653 + - Switch between different histogram fields
1654 + - Auto-updates with filters
1655 +
1656 +3. **Advanced Table Features**
1657 + - Infinite scroll using anchor navigation
1658 + - No manual pagination controls
1659 + - Automatic row coloring by severity
1660 + - Full-width message display
1661 + - Column pinning and resizing
1662 +
1663 +4. **Search and Navigation**
1664 + - Full-text search box
1665 + - Bi-directional navigation (forward/backward)
1666 + - Jump to specific time
1667 + - Export filtered results
1668 +
1669 +5. **Live Features**
1670 + - Tail mode for real-time updates
1671 + - Auto-refresh based on `update_every`
1672 + - Delta updates for efficiency
1673 + - Notification of new entries
1674 +
1675 +---
1676 +
1677 +*Last Updated: Based on analysis of Netdata codebase and cloud-frontend implementation*
\ No newline at end of file
src/plugins.d/functions-table.md deleted
-418
@@ -1,418 +0,0 @@
1 -
2 -> This document is a work in progress.
3 -
4 -Plugin functions can support any kind of responses. However, the UI of Netdata has defined some structures as responses it can parse, understand and visualize.
5 -
6 -One of these responses is the `table`. This is used in almost all functions implemented today.
7 -
8 -# Functions Tables
9 -
10 -Tables are defined when `"type": "table"` is set. The following is the standard header that should be available on all `table` responses:
11 -
12 -```json
13 -{
14 - "type": "table",
15 - "status": 200,
16 - "update_every": 1,
17 - "help": "help text",
18 - "hostname": "the hostname of the server sending this response, to appear at the title of the UI.",
19 - "expires": "UNIX epoch timestamp that the response expires",
20 - "has_history": "boolean: true when the datetime picker plays a role in the result set",
21 - // rest of the response
22 -}
23 -```
24 -
25 -## Preflight `info` request
26 -
27 -The UI, before making the first call to a function, it does a preflight request to understand what the function supports. The plugin receives this request as a FUNCTION call specifying the `info` parameter (possible among others).
28 -
29 -The response from the plugin is expected to have the following:
30 -
31 -```json
32 -{
33 - // standard table header - as above
34 - "accepted_params": [ "a", "b", "c", ...],
35 - "required_params": [
36 - {
37 - "id": "the keyword to use when sending / receiving this parameter",
38 - "name": "the name to present to users for this parameter",
39 - "help": "a help string to help users understand this parameter",
40 - "type": "the type of the parameter, either: 'select' or 'multiselect'",
41 - "options": [
42 - {
43 - "id": "the keyword to use when sending / receiving this option",
44 - "name": "the name to present to users for this option",
45 - "pill": "a short text to show next to this option as a pill",
46 - "info": "a longer text to show on a tooltip when the user is hovering this option"
47 - },
48 - // more options for this required parameter
49 - ]
50 - },
51 - // more required parameters
52 - ]
53 -}
54 -```
55 -
56 -If there are no required parameters, `required_params` can be omitted.
57 -If there are no accepted parameters, `accepted_params` can be omitted. `accepted_param` can be sent during normal responses to update the UI with a new set of parameters available, between calls.
58 -
59 -For `logs`, the UI requires this set of `accepted_params`.
60 -
61 -Ref [Pagination](#pagination), [Deltas](#incremental-responses)
62 -```json
63 -[
64 - "info", // boolean: requests the preflight `info` request
65 - "after", // interval start timestamp
66 - "before", // interval end timestamp
67 - "direction", // sort direction [backward,forward]
68 - "last", // number of records to retrieve
69 - "anchor", // timestamp to divide records in pages
70 - "facets",
71 - "histogram", // selects facet to be used on the histogram
72 - "if_modified_since", // used in PLAY mode, to indicate that the UI wants data newer than the specified timestamp
73 - "data_only", // boolean: requests data (logs) only
74 - "delta", // boolean: requests incremental responses
75 - "tail",
76 - "sampling",
77 - "slice"
78 -]
79 -```
80 -
81 -If there are `required_params`, the UI by default selects the first option. [](VERIFY_WITH_UI)
82 -
83 -## Table data
84 -
85 -To define table data, the UI expects this:
86 -
87 -```json
88 -{
89 - // header
90 - "columns": {
91 - "id": {
92 - "index": "number: the sort order for the columns, lower numbers are first",
93 - "name": "string: the name of the column as it should be presented to users",
94 - "unique_key": "boolean: true when the column uniquely identifies the row",
95 - "visible": "boolean: true when the column should be visible by default",
96 - "type": "enum: see column types",
97 - "units": "string: the units of the value, if any - this item can be omitted if the column does not have units [](VERIFY_WITH_UI)",
98 - "visualization": "enum: see visualization types",
99 - "value_options": {
100 - "units": "string: the units of the value [](VERIFY_WITH_UI)",
101 - "transform": "enum: see transformation types",
102 - "decimal_points": "number: the number of fractional digits for the number",
103 - "default_value": "whatever the value is: when the value is null, show this instead"
104 - },
105 - "max": "number: when the column is numeric, this is the max value the data have - this is used when range filtering is set and value bars",
106 - "pointer_to": "id of another field: this is used when detail-string is set, to point to the column this column is detail of",
107 - "sort": "enum: sorting order",
108 - "sortable": "boolean: whether the column is sortable by users",
109 - "sticky": "boolean: whether the column should always be visible in the UI",
110 - "summary": "string: ???",
111 - "filter": "enum: the filtering type for this column",
112 - "full_width": "boolean: the value is expected to get most of the available column space. When multiple columns are full_width, the available space is given to all of them.",
113 - "wrap": "boolean: true when the entire value should be shown, even when it occupies a big space.",
114 - "default_expanded_filter": "boolean: true when the filter of this column should be expanded by default.",
115 - "dummy": "boolean: when set to true, the column is not to be presented to users."
116 - },
117 - // more IDs
118 - },
119 - "data": [ // array of rows
120 - [ // array of columns
121 - // values for each column linked to their "index" in the columns
122 - ],
123 - // next row
124 - ],
125 - "default_sort_column": "id: the id of the column that should be sorted by default"
126 -}
127 -```
128 -
129 -**IMPORTANT**
130 -
131 -On Data values, `timestamp` column value must be in unix micro.
132 -
133 -
134 -### Sorting order
135 -
136 -- `ascending`
137 -- `descending`
138 -
139 -### Transformation types
140 -
141 -- `none`, just show the value, without any processing
142 -- `number`, just show a number with its units, respecting `decimal_points`
143 -- `duration`, makes the UI show a human readable duration, of the seconds given
144 -- `datetime`, makes the UI show a human readable datetime of the timestamp in UNIX epoch
145 -- `datetime_usec`, makes the UI show a human readable datetime of the timestamp in USEC UNIX epoch
146 -
147 -### Visualization types
148 -
149 -- `value`
150 -- `bar`
151 -- `pill`
152 -- `richValue`, this is not used yet, it is supposed to be a structure that will provide a value and options for it
153 -- `rowOptions`, defines options for the entire row - this column is hidden from the UI
154 -
155 -### rowOptions
156 -
157 -TBD
158 -
159 -### Column types
160 -
161 -- `none`
162 -- `integer`
163 -- `boolean`
164 -- `string`
165 -- `detail-string`
166 -- `bar-with-integer`
167 -- `duration`
168 -- `timestamp`
169 -- `array`
170 -
171 -### Filter types
172 -
173 -- `none`, this facet is not selectable by users
174 -- `multiselect`, the user can select any number of the available options
175 -- `facet`, similar to `multiselect`, but it also indicates that the column has been indexed and has values with counters. Columns set to `facet` must appear in the `facets` list.
176 -- `range`, the user can select a range of values (numeric)
177 -
178 -The plugin may send non visible columns with filter type `facet`. This means that the plugin can enable indexing on these columns, but it has not done it. Then the UI may send `facets:{ID1},{ID2},{ID3},...` to enable indexing of the columns specified.
179 -
180 -What is the default?
181 -
182 -#### Facets
183 -
184 -Facets are a special case of `multiselect` fields. They are used to provide additional information about each possible value, including their relative sort order and the number of times each value appears in the result set. Facets are filters handled by the plugin. So, the plugin will receive user selected filter like: `{KEY}:{VALUE1},{VALUE2},...`, where `{KEY}` is the id of the column and `{VALUEX}` is the id the facet option the user selected.
185 -
186 -```json
187 -{
188 - // header,
189 - "columns": ...,
190 - "data": ...,
191 - "facets": [
192 - {
193 - "id": "string: the unique id of the facet",
194 - "name": "string: the human readable name of the facet",
195 - "order": "integer: the sorting order of this facet - lower numbers move items above others"
196 - "options": [
197 - {
198 - "id": "string: the unique id of the facet value",
199 - "name": "string: the human readable version of the facet value",
200 - "count": "integer: the number of times this value appears in the result set",
201 - "order": "integer: the sorting order of this facet value - lower numbers move items above others"
202 - },
203 - // next option
204 - ],
205 - },
206 - // next facet
207 - ]
208 -}
209 -```
210 -
211 -## Charts
212 -
213 -```json
214 -{
215 - // header,
216 - "charts": {
217 -
218 - },
219 - "default_charts": [
220 -
221 - ]
222 -}
223 -```
224 -
225 -
226 -## Histogram
227 -
228 -```json
229 -{
230 - "available_histograms": [
231 - {
232 - "id": "string: the unique id of the histogram",
233 - "name": "string: the human readable name of the histogram",
234 - "order": "integer: the sorting order of available histograms - lower numbers move items above others"
235 - }
236 - ],
237 - "histogram": {
238 - "id": "string: the unique id of the histogram",
239 - "name": "string: the human readable name of the histogram",
240 - "chart": {
241 - "summary": {
242 - "nodes": [
243 - {
244 - "mg": "string",
245 - "nm": "string: node name",
246 - "ni": "integer: node index"
247 - }
248 - ],
249 - "contexts": [
250 - {
251 - "id": "string: context id"
252 - }
253 - ],
254 - "instances": [
255 - {
256 - "id": "string: instance id",
257 - "ni": "integer: instance index"
258 - }
259 - ],
260 - "dimensions": [
261 - {
262 - "id": "string: dimension id",
263 - "pri": "integer",
264 - "sts": {
265 - "min": "float: dimension min value",
266 - "max": "float: dimension max value",
267 - "avg": "float: dimension avarage value",
268 - "arp": "float",
269 - "con": "float"
270 - }
271 - }
272 - ]
273 - },
274 - "result": {
275 - "labels": [
276 - // histogram labels
277 - ],
278 - "point": {
279 - "value": "integer",
280 - "arp": "integer",
281 - "pa": "integer"
282 - },
283 - "data": [
284 - [
285 - "timestamp" // unix milli
286 - // one array per label
287 - [
288 - // values
289 - ],
290 - ]
291 - ]
292 - },
293 - "view": {
294 - "title": "string: histogram tittle",
295 - "update_every": "integer",
296 - "after": "timestamp: histogram window start",
297 - "before": "timestamp: histogram window end",
298 - "units": "string: histogram units",
299 - "chart_type": "string: histogram chart type",
300 - "min": "integer: histogram min value",
301 - "max": "integer: histogram max value",
302 - "dimensions": {
303 - "grouped_by": [
304 - // "string: histogram grouped by",
305 - ],
306 - "ids": [
307 - // "string: histogram label id",
308 - ],
309 - "names": [
310 - // "string: histogram human readable label name",
311 - ],
312 - "colors": [],
313 - "units": [
314 - // "string: histogram label unit",
315 - ],
316 - "sts": {
317 - "min": [
318 - // "float: label min value",
319 - ],
320 - "max": [
321 - // "float: label max value",
322 - ],
323 - "avg": [
324 - // "float: label avarage value",
325 - ],
326 - "arp": [
327 - // "float",
328 - ],
329 - "con": [
330 - // "float",
331 - ]
332 - }
333 - }
334 - },
335 - "totals": {
336 - "nodes": {
337 - "sl": "integer",
338 - "qr": "integer"
339 - },
340 - "contexts": {
341 - "sl": "integer",
342 - "qr": "integer"
343 - },
344 - "instances": {
345 - "sl": "integer",
346 - "qr": "integer"
347 - },
348 - "dimensions": {
349 - "sl": "integer",
350 - "qr": "integer"
351 - }
352 - },
353 - "db": {
354 - "update_every": "integer"
355 - }
356 - }
357 - }
358 -}
359 -```
360 -
361 -**IMPORTANT**
362 -
363 -On Result Data, `timestamps` must be in unix milli.
364 -
365 -## Grouping
366 -
367 -```json
368 -{
369 - // header,
370 - "group_by": {
371 -
372 - }
373 -}
374 -```
375 -
376 -## Datetime picker
377 -
378 -When `has_history: true`, the plugin must accept `after:TIMESTAMP_IN_SECONDS` and `before:TIMESTAMP_IN_SECONDS` parameters.
379 -The plugin can also turn pagination on, so that only a small set of the data are sent to the UI at a time.
380 -
381 -
382 -## Pagination
383 -
384 -The UI supports paginating results when `has_history: true`. So, when the result depends on the datetime picker and it is too big to be sent to the UI in one response, the plugin can enable datetime pagination like this:
385 -
386 -```json
387 -{
388 - // header,
389 - "columns": ...,
390 - "data": ...,
391 - "has_history": true,
392 - "pagination": {
393 - "enabled": "boolean: true to enable it",
394 - "column": "string: the column id that is used for pagination",
395 - "key": "string: the accepted_param that is used as the pagination anchor",
396 - "units": "enum: a transformation of the datetime picker to make it compatible with the anchor: timestamp, timestamp_usec"
397 - }
398 -}
399 -```
400 -
401 -Once pagination is enabled, the plugin must support the following parameters:
402 -
403 -- `{ANCHOR}:{VALUE}`, `{ANCHOR}` is the `pagination.key`, `{VALUE}` is the point the user wants to see entries at, formatted according to `pagination.units`.
404 -- `direction:backward` or `direction:forward` to specify if the data to be returned if before are after the anchor.
405 -- `last:NUMER`, the number of entries the plugin should return in the table data.
406 -- `query:STRING`, the full text search string the user wants to search for.
407 -- `if_modified_since:TIMESTAMP_USEC` and `tail:true`, used in PLAY mode, to indicate that the UI wants data newer than the specified timestamp. If there are no new data, the plugin must respond with 304 (Not Modified).
408 -
409 -### Incremental Responses
410 -
411 -- `delta:true` or `delta:false`, when the plugin supports incremental queries, it can accept the parameter `delta`. When set to true, the response of the plugin will be "added" to the previous response already available. This is used in combination with `if_modified_since` to optimize the amount of work the plugin has to do to respond.
412 -
413 -
414 -### Other
415 -
416 -- `slice:BOOLEAN` [](VERIFY_WITH_UI)
417 -- `sampling:NUMBER`
418 -