@cryptotaxi247 / netdata-1 / commits / 7fb529ebd

feat(go.d): add Top Queries Functions framework with PostgreSQL, MySQL, MSSQL, and MongoDB support (#21595)

Co-authored-by: ilyam8 <ilya@netdata.cloud>

Costa Tsaousis committed Jan 21, 2026 at 09:01 UTC 7fb529ebd20298fa659aba0910c97b2bc21bac22
67 files changed +8320 -4024
src/crates/netdata-plugin/docs/DYNCFG.md deleted
-468
@@ -1,468 +0,0 @@
1 -# Dynamic Configuration for External Plugins
2 -
3 -External plugins in Netdata can expose dynamic configuration capabilities through the DynCfg system. This document explains how to implement DynCfg in external plugins using the plugins.d protocol.
4 -
5 -## Overview
6 -
7 -The DynCfg system allows external plugins to:
8 -
9 -1. Register configurable entities (both single configurations and templates for creating jobs)
10 -2. Receive configuration commands from users
11 -3. Validate and apply configurations
12 -4. Persist configurations between Netdata agent restarts
13 -
14 -## Protocol Commands
15 -
16 -DynCfg for external plugins uses the following plugins.d protocol commands:
17 -
18 -1. `CONFIG`: Sent from the plugin to Netdata to register, update status, or delete configurations
19 -2. `FUNCTION`/`FUNCTION_PAYLOAD_BEGIN`: Received by the plugin to handle configuration commands
20 -3. `FUNCTION_RESULT_BEGIN`: Sent from the plugin to respond to commands
21 -
22 -## Implementing DynCfg in External Plugins
23 -
24 -### 1. Register a Configuration
25 -
26 -To register a configuration, the plugin sends the CONFIG command:
27 -
28 -```
29 -CONFIG <id> CREATE <status> <type> <path> <source_type> <source> <cmds> <view_access> <edit_access>
30 -```
31 -
32 -Where:
33 -
34 -- `id` is a unique identifier for the configurable entity (e.g., "go.d:nginx")
35 -- `status` can be:
36 - - `accepted`: Configuration is accepted but not running
37 - - `running`: Configuration is accepted and running
38 - - `failed`: Plugin fails to run the configuration
39 - - `incomplete`: Plugin needs additional settings
40 - - `disabled`: Configuration is disabled by a user
41 -- `type` can be:
42 - - `single`: A single configuration object (not addable or removable by users)
43 - - `template`: A template for creating multiple job configurations
44 - - `job`: A specific job configuration (derived from a template)
45 -- `path` is the UI organization path (usually "/collectors") that determines where in the configuration tree the item will appear in the UI. This is separate from the ID and controls the hierarchical navigation structure.
46 -- `source_type` can be:
47 - - `internal`: Based on internal code settings
48 - - `stock`: Default configurations
49 - - `user`: User configurations via a file
50 - - `dyncfg`: Configuration received via this mechanism
51 - - `discovered`: Dynamically discovered by the plugin
52 -- `source` provides more details about the exact source
53 -- `cmds` is a space or pipe (|) separated list of supported commands:
54 - - `schema`: Get JSON schema for the configuration
55 - - `get`: Get current configuration values
56 - - `update`: Receive configuration updates
57 - - `add`: Receive job creation commands (templates only)
58 - - `remove`: Remove a configuration (jobs only)
59 - - `enable`/`disable`: Enable or disable the configuration
60 - - `test`: Test a configuration without applying it
61 - - `restart`: Restart the configuration
62 - - `userconfig`: Get user-friendly configuration format
63 -- `view_access` and `edit_access` are permission bitmaps (use 0 for default permissions)
64 -
65 -Example:
66 -
67 -```
68 -CONFIG go.d:nginx CREATE accepted template /collectors internal internal schema|add|enable|disable 0 0
69 -CONFIG go.d:nginx:local_server CREATE running job /collectors dyncfg user schema|get|update|remove|enable|disable|restart 0 0
70 -```
71 -
72 -### 2. Respond to Configuration Commands
73 -
74 -The plugin receives configuration commands from Netdata as plugin functions. These come in two forms:
75 -
76 -#### Without Payload:
77 -
78 -```
79 -FUNCTION <transaction_id> <timeout_ms> "config <id> <command>" "<http_access>" "<source>"
80 -```
81 -
82 -Used for commands like: `schema`, `get`, `remove`, `enable`, `disable`, `restart`
83 -
84 -Example:
85 -
86 -```
87 -FUNCTION abcd1234 60 "config go.d:nginx:local_server get" "member" "netdata-cli"
88 -```
89 -
90 -#### With Payload:
91 -
92 -```
93 -FUNCTION_PAYLOAD_BEGIN <transaction_id> <timeout_ms> "config <id> <command>" "<http_access>" "<source>" "<content_type>"
94 -<payload_data>
95 -FUNCTION_PAYLOAD_END
96 -```
97 -
98 -Used for commands like: `update`, `add`, `test` that require additional data.
99 -
100 -Example:
101 -
102 -```
103 -FUNCTION_PAYLOAD_BEGIN abcd1234 60 "config go.d:nginx:local_server update" "member" "netdata-cli" "application/json"
104 -{
105 - "url": "http://localhost:80/stub_status",
106 - "timeout": 5,
107 - "update_every": 10
108 -}
109 -FUNCTION_PAYLOAD_END
110 -```
111 -
112 -### 3. Process Commands and Respond
113 -
114 -After receiving a command, the plugin should process it and respond with a function result:
115 -
116 -```
117 -FUNCTION_RESULT_BEGIN <transaction_id> <http_status_code> <content_type> <expiration>
118 -<result_data>
119 -FUNCTION_RESULT_END
120 -```
121 -
122 -Where:
123 -
124 -- `transaction_id` is the same ID received in the original command
125 -- `http_status_code` is the standard HTTP response code:
126 - - `200`: Success (DYNCFG_RESP_RUNNING) - Configuration accepted and running
127 - - `202`: Accepted (DYNCFG_RESP_ACCEPTED) - Configuration accepted but not running yet
128 - - `298`: Accepted but disabled (DYNCFG_RESP_ACCEPTED_DISABLED)
129 - - `299`: Accepted but restart required (DYNCFG_RESP_ACCEPTED_RESTART_REQUIRED)
130 - - `400`: Bad request - Invalid configuration
131 - - `404`: Not found - Configuration not found
132 - - `500`: Internal server error
133 -- `content_type` is typically "application/json"
134 -- `expiration` is the absolute timestamp (unix epoch) for result expiration
135 -
136 -The result data depends on the command:
137 -
138 -- `schema`: Return JSON Schema document
139 -- `get`: Return current configuration values
140 -- Other commands: Return a success or error message
141 -
142 -Success response example:
143 -
144 -```
145 -FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0
146 -{
147 - "status": 200,
148 - "message": "Configuration updated successfully"
149 -}
150 -FUNCTION_RESULT_END
151 -```
152 -
153 -Error response example:
154 -
155 -```
156 -FUNCTION_RESULT_BEGIN abcd1234 400 application/json 0
157 -{
158 - "status": 400,
159 - "error_message": "Invalid URL format"
160 -}
161 -FUNCTION_RESULT_END
162 -```
163 -
164 -### 4. Update Configuration Status
165 -
166 -To update the status of a configuration after it's been created:
167 -
168 -```
169 -CONFIG <id> STATUS <new_status>
170 -```
171 -
172 -Example:
173 -
174 -```
175 -CONFIG go.d:nginx:local_server STATUS running
176 -```
177 -
178 -This is useful when a configuration transitions from "accepted" to "running" or "failed" after being tested.
179 -
180 -### 5. Delete a Configuration
181 -
182 -When a configuration is no longer available (e.g., the monitored service is removed):
183 -
184 -```
185 -CONFIG <id> DELETE
186 -```
187 -
188 -Example:
189 -
190 -```
191 -CONFIG go.d:nginx:local_server DELETE
192 -```
193 -
194 -## JSON Schema for Configuration UI
195 -
196 -DynCfg uses JSON Schema to define the structure of configuration objects, which is used to generate the UI.
197 -
198 -### Static Schema Files (Optional)
199 -
200 -Before calling the plugin, Netdata will first attempt to find a static schema file. You can provide static schema files in:
201 -
202 -- `CONFIG_DIR/schema.d/` (user-provided schemas, typically `/etc/netdata/schema.d/`)
203 -- `LIBCONFIG_DIR/schema.d/` (stock schemas, typically `/usr/lib/netdata/conf.d/schema.d/`)
204 -
205 -Schema files should be named after the configuration ID with `.json` extension:
206 -
207 -```
208 -/etc/netdata/schema.d/go.d:nginx.json
209 -```
210 -
211 -This approach is useful for stable schemas that don't change frequently.
212 -
213 -### Dynamic Schema Generation
214 -
215 -If no static schema file is found, Netdata will send a `schema` command to the plugin. When handling a `schema` request, the plugin should return a JSON Schema document:
216 -
217 -```json
218 -{
219 - "type": "object",
220 - "properties": {
221 - "url": {
222 - "type": "string",
223 - "format": "uri",
224 - "title": "Server URL",
225 - "description": "The URL of the Nginx stub_status endpoint"
226 - },
227 - "timeout": {
228 - "type": "integer",
229 - "minimum": 1,
230 - "maximum": 60,
231 - "title": "Timeout",
232 - "description": "Connection timeout in seconds"
233 - },
234 - "update_every": {
235 - "type": "integer",
236 - "minimum": 1,
237 - "title": "Update Every",
238 - "description": "Data collection frequency in seconds"
239 - }
240 - },
241 - "required": [
242 - "url"
243 - ]
244 -}
245 -```
246 -
247 -For templates, the schema will be used when users add new jobs based on the template.
248 -
249 -## Action Behavior Reference
250 -
251 -When implementing DynCfg in your external plugin, be aware of how actions should behave based on the configuration type:
252 -
253 -| Action | TEMPLATE | JOB |
254 -|----------------|-----------------------------------------|-----------------------------------------|
255 -| **SCHEMA** | Return schema for creating new jobs | Use template's schema |
256 -| **GET** | Not applicable | Return current configuration |
257 -| **UPDATE** | Not applicable | Update configuration and apply if valid |
258 -| **ADD** | Create new job from template | Not applicable |
259 -| **REMOVE** | Not supported | Remove job (only for user-created jobs) |
260 -| **ENABLE** | Enable template and all its jobs | Enable specific job |
261 -| **DISABLE** | Disable template and all its jobs | Disable specific job |
262 -| **RESTART** | Restart all jobs based on template | Restart specific job |
263 -| **TEST** | Test a potential job configuration | Test configuration changes |
264 -| **USERCONFIG** | Return template in user-friendly format | Return job in user-friendly format |
265 -
266 -**Important Implementation Notes:**
267 -
268 -- When a template is disabled, send DISABLE commands to all jobs of that template
269 -- Reject ENABLE commands for jobs if their template is disabled
270 -- For job SCHEMA requests, return the same schema as the template
271 -- REMOVE should only work on dynamically added jobs, not ones from static configurations
272 -- Return appropriate response codes to indicate the status (running, accepted, disabled)
273 -
274 -## External Plugin Examples
275 -
276 -### C-based External Plugin (systemd-journal.plugin)
277 -
278 -The systemd-journal.plugin is a C-based external plugin that uses DynCfg to manage journal directory configurations. It implements a SINGLE configuration type to manage the list of journald directories to monitor:
279 -
280 -```c
281 -// Register the configuration
282 -functions_evloop_dyncfg_add(
283 - wg,
284 - "systemd-journal:monitored-directories", // ID
285 - "/logs/systemd-journal", // UI Path
286 - DYNCFG_STATUS_RUNNING, // Status
287 - DYNCFG_TYPE_SINGLE, // Type - single configuration
288 - DYNCFG_SOURCE_TYPE_INTERNAL, // Source type
289 - "internal", // Source
290 - DYNCFG_CMD_SCHEMA | DYNCFG_CMD_GET | DYNCFG_CMD_UPDATE, // Supported commands
291 - HTTP_ACCESS_NONE, // View permissions
292 - HTTP_ACCESS_NONE, // Edit permissions
293 - systemd_journal_directories_dyncfg_cb, // Callback function
294 - NULL // User data
295 -);
296 -```
297 -
298 -Key points about its implementation:
299 -
300 -- Uses a single, non-removable configuration object
301 -- Supports schema, get, and update commands
302 -- Validates directory paths for security
303 -- Updates the systemd-journal watcher when configuration changes
304 -
305 -### Go-based External Plugin (go.d.plugin)
306 -
307 -Here's a complete example showing how a Go-based external plugin might implement DynCfg for an Nginx module:
308 -
309 -### 1. Register the Template and Jobs on Startup
310 -
311 -```
312 -# Register the template for Nginx configurations
313 -CONFIG go.d:nginx CREATE accepted template /collectors internal internal schema|add|enable|disable 0 0
314 -
315 -# Register existing jobs
316 -CONFIG go.d:nginx:local_server CREATE running job /collectors user /etc/netdata/go.d/nginx.conf schema|get|update|remove|enable|disable|restart 0 0
317 -CONFIG go.d:nginx:production CREATE running job /collectors user /etc/netdata/go.d/nginx.conf schema|get|update|remove|enable|disable|restart 0 0
318 -```
319 -
320 -### 2. Handle Schema Command
321 -
322 -When receiving:
323 -
324 -```
325 -FUNCTION abcd1234 60 "config go.d:nginx schema" "member" "netdata-cli"
326 -```
327 -
328 -Respond with:
329 -
330 -```
331 -FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0
332 -{
333 - "type": "object",
334 - "properties": {
335 - "url": {
336 - "type": "string",
337 - "format": "uri",
338 - "title": "Server URL",
339 - "description": "The URL of the Nginx stub_status endpoint"
340 - },
341 - "timeout": {
342 - "type": "integer",
343 - "minimum": 1,
344 - "maximum": 60,
345 - "title": "Timeout",
346 - "description": "Connection timeout in seconds"
347 - },
348 - "update_every": {
349 - "type": "integer",
350 - "minimum": 1,
351 - "title": "Update Every",
352 - "description": "Data collection frequency in seconds"
353 - }
354 - },
355 - "required": ["url"]
356 -}
357 -FUNCTION_RESULT_END
358 -```
359 -
360 -### 3. Handle Get Command
361 -
362 -When receiving:
363 -
364 -```
365 -FUNCTION abcd1234 60 "config go.d:nginx:local_server get" "member" "netdata-cli"
366 -```
367 -
368 -Respond with:
369 -
370 -```
371 -FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0
372 -{
373 - "url": "http://localhost:80/stub_status",
374 - "timeout": 5,
375 - "update_every": 10
376 -}
377 -FUNCTION_RESULT_END
378 -```
379 -
380 -### 4. Handle Update Command
381 -
382 -When receiving:
383 -
384 -```
385 -FUNCTION_PAYLOAD_BEGIN abcd1234 60 "config go.d:nginx:local_server update" "member" "netdata-cli" "application/json"
386 -{
387 - "url": "http://localhost:8080/stub_status",
388 - "timeout": 3,
389 - "update_every": 5
390 -}
391 -FUNCTION_PAYLOAD_END
392 -```
393 -
394 -Process the update and respond:
395 -
396 -```
397 -FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0
398 -{
399 - "status": 200,
400 - "message": "Configuration updated successfully"
401 -}
402 -FUNCTION_RESULT_END
403 -```
404 -
405 -If a restart is required:
406 -
407 -```
408 -FUNCTION_RESULT_BEGIN abcd1234 299 application/json 0
409 -{
410 - "status": 299,
411 - "message": "Configuration updated, restart required to apply changes"
412 -}
413 -FUNCTION_RESULT_END
414 -```
415 -
416 -### 5. Handle Add Command (for templates)
417 -
418 -When receiving:
419 -
420 -```
421 -FUNCTION_PAYLOAD_BEGIN abcd1234 60 "config go.d:nginx add" "member" "netdata-cli" "application/json"
422 -{
423 - "name": "staging",
424 - "url": "http://staging:80/stub_status",
425 - "timeout": 5,
426 - "update_every": 10
427 -}
428 -FUNCTION_PAYLOAD_END
429 -```
430 -
431 -Process the new job and respond:
432 -
433 -```
434 -FUNCTION_RESULT_BEGIN abcd1234 200 application/json 0
435 -{
436 - "status": 200,
437 - "message": "Job 'staging' created successfully"
438 -}
439 -FUNCTION_RESULT_END
440 -```
441 -
442 -Then register the new job:
443 -
444 -```
445 -CONFIG go.d:nginx:staging CREATE running job /collectors dyncfg netdata-cli schema|get|update|remove|enable|disable|restart 0 0
446 -```
447 -
448 -## Best Practices
449 -
450 -1. **Use Consistent IDs**: Follow the pattern `component:template_name` for templates and `component:template_name:job_name` for jobs
451 -2. **Validate Thoroughly**: Always validate configuration changes before accepting them
452 -3. **Include Descriptive Messages**: Provide helpful error messages when rejections occur
453 -4. **Document Your Schema**: Include clear titles and descriptions for all properties in your JSON Schema
454 -5. **Handle Errors Gracefully**: Return appropriate HTTP status codes and error messages
455 -6. **Update Status Promptly**: When a configuration changes state (e.g., from "accepted" to "running"), update its status
456 -7. **Clean Up Configurations**: When a monitored resource is gone, delete its configuration with `CONFIG id DELETE`
457 -
458 -## Debugging Tips
459 -
460 -1. Set `NETDATA_DEBUG_DYNCFG=1` environment variable when running Netdata to see detailed logs
461 -2. If configurations aren't being registered, check for errors in the plugin output
462 -3. Verify configuration files are saved in `/var/lib/netdata/config/`
463 -4. Test configurations via the API: `/api/v3/config?id=<your-config-id>`
464 -
465 -## Related Documentation
466 -
467 -- [Main DynCfg Documentation](/src/daemon/dyncfg/README.md) - Core DynCfg system concepts and APIs
468 -- [Plugins.d Protocol](/src/plugins.d/README.md) - Complete documentation of the plugins.d protocol
src/crates/netdata-plugin/docs/FUNCTION_UI_DEVELOPER_GUIDE.md deleted
-953
@@ -1,953 +0,0 @@
1 -# Netdata Functions: Developer Guide
2 -
3 -> **Note**: This is the practical developer guide. For the complete technical specification, see [FUNCTIONS_REFERENCE.md](FUNCTIONS_REFERENCE.md).
4 -
5 -## Overview
6 -
7 -This guide teaches you how to create Netdata functions that provide interactive data through the web UI. You'll learn to build both simple tables and advanced log explorers.
8 -
9 -**What You'll Learn:**
10 -- How to create simple table functions for system monitoring
11 -- How to build log explorer functions with faceted search
12 -- All column types, options, and their UI effects
13 -- Query patterns, filtering, and aggregation
14 -- Real-world examples and best practices
15 -
16 -**Quick Navigation:**
17 -- [Part 1: Simple Table Functions](#part-1-simple-table-functions) - Basic monitoring data
18 -- [Part 2: Log Explorer Functions](#part-2-log-explorer-functions) - Historical data with search
19 -- [Part 3: Complete Options Reference](#part-3-complete-options-reference) - Every option explained
20 -
21 ----
22 -
23 -# Part 1: Simple Table Functions
24 -
25 -Simple table functions display current system state - processes, connections, services, etc. They're perfect for "top-like" views and system monitoring.
26 -
27 -**Implementation Architecture:**
28 -- **Frontend handles**: Filtering, search, facet counting, sorting
29 -- **Backend provides**: Raw data, column definitions, optional charts
30 -- **Performance**: Limited by browser processing (good for \<10k rows)
31 -- **Query parameter**: Processed in frontend only (substring search)
32 -- **Histograms**: Not supported
33 -
34 -## Your First Simple Table Function
35 -
36 -Start with this minimal working example:
37 -
38 -```json
39 -{
40 - "status": 200,
41 - "type": "table",
42 - "has_history": false,
43 - "help": "Shows system services",
44 - "data": [
45 - ["nginx", 25, "Running"],
46 - ["mysql", 67, "Stopped"],
47 - ["redis", 91, "Running"]
48 - ],
49 - "columns": {
50 - "name": {
51 - "index": 0,
52 - "name": "Service",
53 - "type": "string",
54 - "unique_key": true
55 - },
56 - "cpu": {
57 - "index": 1,
58 - "name": "CPU %",
59 - "type": "bar-with-integer",
60 - "max": 100
61 - },
62 - "status": {
63 - "index": 2,
64 - "name": "Status",
65 - "type": "string"
66 - }
67 - }
68 -}
69 -```
70 -
71 -This creates a basic 3-column table showing service names, CPU usage bars, and status.
72 -
73 -## Essential Features for Simple Tables
74 -
75 -### 1. Required Fields
76 -
77 -Every simple table function needs:
78 -
79 -```json
80 -{
81 - "status": 200, // HTTP status (200 = success)
82 - "type": "table", // Always "table"
83 - "has_history": false, // Simple table (not log explorer)
84 - "columns": {...}, // Column definitions
85 - "data": [...] // Your actual data
86 -}
87 -```
88 -
89 -### 2. Column Definitions
90 -
91 -Each column must have:
92 -
93 -```json
94 -"column_id": {
95 - "index": 0, // Position in data arrays (0-based)
96 - "name": "Display Name", // Column header shown to users
97 - "type": "string" // How to render the data
98 -}
99 -```
100 -
101 -### 3. The Data Array
102 -
103 -Data is an array of arrays - each inner array is one row:
104 -
105 -```json
106 -"data": [
107 - ["nginx", 25, "Running"], // Row 1: index 0=name, 1=cpu, 2=status
108 - ["mysql", 67, "Stopped"], // Row 2
109 - ["redis", 91, "Running"] // Row 3
110 -]
111 -```
112 -
113 -**Important**: Values must be in the same order as column `index` fields.
114 -
115 -## Adding Visual Polish
116 -
117 -Make your table more useful with these enhancements:
118 -
119 -```json
120 -{
121 - "status": 200,
122 - "type": "table",
123 - "has_history": false,
124 - "help": "System services with CPU usage and status",
125 - "data": [
126 - ["nginx", 25, "Running", {"rowOptions": {"severity": "normal"}}],
127 - ["mysql", 67, "Stopped", {"rowOptions": {"severity": "error"}}],
128 - ["redis", 91, "Running", {"rowOptions": {"severity": "warning"}}]
129 - ],
130 - "columns": {
131 - "name": {
132 - "index": 0,
133 - "name": "Service Name",
134 - "type": "string",
135 - "unique_key": true,
136 - "sticky": true,
137 - "filter": "multiselect"
138 - },
139 - "cpu": {
140 - "index": 1,
141 - "name": "CPU Usage",
142 - "type": "bar-with-integer",
143 - "units": "%",
144 - "max": 100,
145 - "sort": "descending",
146 - "filter": "range"
147 - },
148 - "status": {
149 - "index": 2,
150 - "name": "Status",
151 - "type": "string",
152 - "visualization": "pill",
153 - "filter": "multiselect"
154 - }
155 - },
156 - "default_sort_column": "cpu"
157 -}
158 -```
159 -
160 -**New Features Added:**
161 -- **Row coloring**: `rowOptions` with severity levels
162 -- **Sticky columns**: Pin important columns when scrolling
163 -- **Filters**: Let users filter by categories or numeric ranges
164 -- **Progress bars**: Visual CPU usage display
165 -- **Pills**: Status badges with colors
166 -- **Default sorting**: Start sorted by CPU usage
167 -
168 -## Limitations of Simple Tables
169 -
170 -Simple tables have certain limitations due to their frontend-only processing architecture:
171 -
172 -- **No histograms**: Time-based visualization is not supported
173 -- **Performance limits**: Frontend processing is limited by browser memory (recommend \<10k rows)
174 -- **No backend search**: Query parameter is not sent to backend - search happens client-side only
175 -- **Static facet counts**: Counts are computed by frontend from all received data
176 -
177 -For large datasets or advanced log analysis features, consider using log explorers (`has_history: true`).
178 -
179 -## Advanced Simple Table Features
180 -
181 -### Aggregated Views
182 -
183 -Some functions can show both detailed and aggregated data. When enabled, facet pills show both counts:
184 -
185 -```json
186 -{
187 - "aggregated_view": {
188 - "column": "Count",
189 - "results_label": "unique combinations",
190 - "aggregated_label": "connections"
191 - }
192 -}
193 -```
194 -
195 -This enables smart facet pills like `"15 ⊃ 42"` meaning "15 connections aggregated into 42 rows".
196 -
197 -### Charts Integration
198 -
199 -Simple tables support interactive charts computed from your table data. The backend defines available charts, and the frontend computes and renders them.
200 -
201 -**Chart Types Available:**
202 -- `"bar"` - Basic bar chart
203 -- `"stacked-bar"` - Multi-column stacked bars (most common)
204 -- `"doughnut"` - Pie/doughnut chart
205 -- `"value"` - Simple numeric display
206 -
207 -**Example Configuration:**
208 -```json
209 -{
210 - "charts": {
211 - "cpu_usage": {
212 - "name": "CPU Usage by Service",
213 - "type": "stacked-bar",
214 - "columns": ["user_cpu", "system_cpu", "guest_cpu"],
215 - "groupBy": "column",
216 - "aggregation": "sum"
217 - },
218 - "memory_breakdown": {
219 - "name": "Memory Types",
220 - "type": "doughnut",
221 - "columns": ["resident", "virtual", "shared"],
222 - "groupBy": "all",
223 - "aggregation": "sum"
224 - }
225 - },
226 - "default_charts": [
227 - ["cpu_usage", "status"],
228 - ["memory_breakdown", "status"]
229 - ]
230 -}
231 -```
232 -
233 -**Chart Options:**
234 -- **`groupBy`**:
235 - - `"column"` (default): Group by selected filter column
236 - - `"all"`: Aggregate all data together
237 -- **`aggregation`**: `sum`, `mean`, `max`, `min`, `count`
238 -
239 -**How It Works:**
240 -1. Frontend takes your table data
241 -2. Groups by selected column (e.g., "service type")
242 -3. Aggregates values using specified method (e.g., sum CPU values)
243 -4. Renders chart with one bar/slice per group
244 -
245 -### Table Row Grouping
246 -
247 -Simple tables support grouping rows with customizable aggregation. When users group by a column, rows with the same value are combined using the aggregation method you specify.
248 -
249 -**Enable Grouping Options:**
250 -```json
251 -{
252 - "group_by": {
253 - "aggregated": [{
254 - "id": "by_status",
255 - "name": "By Status",
256 - "column": "status"
257 - }, {
258 - "id": "by_user",
259 - "name": "By User",
260 - "column": "user"
261 - }]
262 - }
263 -}
264 -```
265 -
266 -**Define Column Aggregation:**
267 -Each column needs a summary type to control how values are aggregated when grouping:
268 -
269 -```c
270 -// In your C backend code
271 -buffer_rrdf_table_add_field(
272 - wb, field_id++, "cpu_percent", "CPU %",
273 - RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
274 - RRDF_FIELD_VISUAL_BAR,
275 - RRDF_FIELD_TRANSFORM_NUMBER,
276 - 2, "%", 100.0, RRDF_FIELD_SORT_DESCENDING, NULL,
277 - RRDF_FIELD_SUMMARY_SUM, // Sum CPU values when grouping
278 - RRDF_FIELD_FILTER_RANGE,
279 - RRDF_FIELD_OPTS_VISIBLE, NULL
280 -);
281 -
282 -buffer_rrdf_table_add_field(
283 - wb, field_id++, "process_count", "Processes",
284 - RRDF_FIELD_TYPE_INTEGER,
285 - RRDF_FIELD_VISUAL_VALUE,
286 - RRDF_FIELD_TRANSFORM_NUMBER,
287 - 0, "processes", NAN, RRDF_FIELD_SORT_DESCENDING, NULL,
288 - RRDF_FIELD_SUMMARY_COUNT, // Count processes when grouping
289 - RRDF_FIELD_FILTER_RANGE,
290 - RRDF_FIELD_OPTS_VISIBLE, NULL
291 -);
292 -```
293 -
294 -**Available Summary/Aggregation Types:**
295 -- `RRDF_FIELD_SUMMARY_COUNT` - Count rows in group
296 -- `RRDF_FIELD_SUMMARY_SUM` - Sum numeric values
297 -- `RRDF_FIELD_SUMMARY_MEAN` - Average values
298 -- `RRDF_FIELD_SUMMARY_MIN` - Minimum value
299 -- `RRDF_FIELD_SUMMARY_MAX` - Maximum value
300 -- `RRDF_FIELD_SUMMARY_UNIQUECOUNT` - Count unique values
301 -- `RRDF_FIELD_SUMMARY_MEDIAN` - Median value
302 -
303 -**Example Result:**
304 -When user groups by "service type", rows like:
305 -```
306 -nginx_worker1 25% Running
307 -nginx_worker2 30% Running
308 -mysql_main 45% Running
309 -```
310 -
311 -Become:
312 -```
313 -nginx 55% 2 processes (25% + 30% CPU, 2 processes counted)
314 -mysql 45% 1 process (45% CPU, 1 process counted)
315 -```
316 -
317 ----
318 -
319 -# Part 2: Log Explorer Functions
320 -
321 -Log explorer functions (`has_history: true`) provide advanced log analysis with full-text search, faceted filtering, time navigation, and histograms. Perfect for systemd journals, event logs, and audit trails.
322 -
323 -**Implementation Architecture:**
324 -- **Backend handles**: Query processing, pattern matching, facet counting, filtering
325 -- **Frontend provides**: UI for facets, histogram display, infinite scroll
326 -- **Performance**: Scales to millions of records with sampling and server-side filtering
327 -- **Query parameter**: Processed by backend using Netdata simple patterns
328 -- **Histograms**: Optional, generated by backend in Netdata chart format
329 -- **Uses facets library**: All processing leverages `libnetdata/facets/`
330 -
331 -### Complete Log Explorer Example
332 -
333 -```json
334 -{
335 - "status": 200,
336 - "type": "table",
337 - "has_history": true,
338 - "help": "System journal with real-time updates",
339 - "accepted_params": [
340 - "info", "after", "before", "direction", "last", "anchor",
341 - "query", "facets", "histogram", "if_modified_since",
342 - "data_only", "delta", "tail", "sampling"
343 - ],
344 - "table": {"id": "journal", "has_history": true},
345 - "columns": {
346 - "timestamp": {
347 - "index": 0,
348 - "name": "Time",
349 - "type": "timestamp",
350 - "transform": "datetime_usec",
351 - "sort": "descending|fixed",
352 - "sticky": true
353 - },
354 - "priority": {
355 - "index": 1,
356 - "name": "Level",
357 - "type": "string",
358 - "visualization": "pill",
359 - "filter": "facet",
360 - "options": ["facet", "visible", "sticky"]
361 - },
362 - "message": {
363 - "index": 2,
364 - "name": "Message",
365 - "type": "string",
366 - "full_width": true,
367 - "options": ["full_width", "wrap", "visible", "main_text", "fts"]
368 - }
369 - },
370 - "data": [
371 - [1697644320000000, {"rowOptions": {"severity": "error"}}, "ERROR", "Service failed"],
372 - [1697644319000000, {"rowOptions": {"severity": "normal"}}, "INFO", "Service started"]
373 - ],
374 - "facets": [
375 - {
376 - "id": "priority",
377 - "name": "Log Level",
378 - "order": 1,
379 - "defaultExpanded": true,
380 - "options": [
381 - {"id": "ERROR", "name": "ERROR", "count": 45, "order": 1},
382 - {"id": "INFO", "name": "INFO", "count": 234, "order": 3}
383 - ]
384 - }
385 - ],
386 - "items": {
387 - "evaluated": 50000,
388 - "matched": 2520,
389 - "returned": 100,
390 - "max_to_return": 100
391 - },
392 - "anchor": {
393 - "last_modified": 1697644320000000,
394 - "direction": "backward"
395 - }
396 -}
397 -```
398 -
399 -**Key Features Demonstrated:**
400 -- `has_history: true` - Enables log explorer UI with infinite scroll
401 -- `accepted_params` - Full parameter support for advanced features
402 -- **Faceted search** - Real-time counts computed by backend
403 -- **Anchor pagination** - Efficient navigation through large datasets
404 -- **Microsecond timestamps** - High precision for log entries
405 -- **Full-text search** - Backend pattern matching with `"fts"` fields
406 -- **Row coloring** - Severity-based visual indicators
407 -
408 -## Essential Log Explorer Features
409 -
410 -### 1. Faceted Search Sidebar
411 -
412 -Facets provide dynamic filtering with real-time counts computed by the backend:
413 -
414 -```json
415 -{
416 - "facets": [
417 - {
418 - "id": "level",
419 - "name": "Log Level",
420 - "order": 1,
421 - "defaultExpanded": true,
422 - "options": [
423 - {"id": "ERROR", "name": "ERROR", "count": 45, "order": 1},
424 - {"id": "WARN", "name": "WARN", "count": 123, "order": 2},
425 - {"id": "INFO", "name": "INFO", "count": 2341, "order": 3}
426 - ]
427 - }
428 - ]
429 -}
430 -```
431 -
432 -**Important**: These counts are calculated by the backend facets library during query execution, not by the frontend. The backend scans through matching records and maintains counters for each facet value.
433 -
434 -### 2. Full-Text Search
435 -
436 -Enable powerful query search across all fields (processed by backend):
437 -
438 -```json
439 -{
440 - "columns": {
441 - "message": {
442 - "options": ["fts"], // Make field full-text searchable
443 - // ... other options
444 - }
445 - }
446 -}
447 -```
448 -
449 -**Important**: Unlike simple tables, the query parameter is sent to the backend and processed server-side using the facets library. The backend performs pattern matching and only returns matching records.
450 -
451 -**Query Patterns** (using Netdata simple patterns):
452 -- `"error"` - Find "error" anywhere (substring match)
453 -- `"error|warning"` - Find "error" OR "warning" (pipe separator)
454 -- `"!debug"` - Exclude logs containing "debug"
455 -- `"!*debugging*|*debug*"` - Include "debug" but exclude "debugging"
456 -- `"connection failed"` - Find exact phrase (spaces are literal)
457 -
458 -**Pattern Evaluation Rules:**
459 -1. **Within field**: Left-to-right, first match wins
460 -2. **Across fields**: ALL fields evaluated, ANY negative match excludes row
461 -3. **Case-insensitive** matching throughout
462 -
463 -### 3. Time-Based Histograms
464 -
465 -Visualize log distribution over time (log explorers only):
466 -
467 -```json
468 -{
469 - "available_histograms": [
470 - {"id": "level", "name": "Log Level", "order": 1},
471 - {"id": "source", "name": "Source", "order": 2}
472 - ],
473 - "histogram": {
474 - "id": "level",
475 - "name": "Log Level",
476 - "chart": {
477 - "result": {
478 - "labels": ["time", "ERROR", "WARN", "INFO"],
479 - "data": [
480 - [1697644200, 5, 12, 234],
481 - [1697644260, 3, 8, 198]
482 - ]
483 - }
484 - }
485 - }
486 -}
487 -```
488 -
489 -**Important**: Histograms are NOT supported for simple tables (`has_history: false`). They are optional for log explorers and use the same format as Netdata's `/api/v3/data` endpoint. The backend generates histogram data using the facets library.
490 -
491 -### 4. Anchor-Based Navigation
492 -
493 -Handle large datasets with efficient pagination:
494 -
495 -```json
496 -{
497 - "items": {
498 - "evaluated": 50000, // Total scanned
499 - "matched": 2520, // Match filters
500 - "returned": 100, // In this response
501 - "max_to_return": 100
502 - },
503 - "anchor": {
504 - "last_modified": 1697644320000000,
505 - "direction": "backward"
506 - }
507 -}
508 -```
509 -
510 -### Parameter Integration Patterns
511 -
512 -**Basic Monitoring Function:**
513 -```json
514 -{
515 - "accepted_params": ["info", "after", "before"]
516 -}
517 -```
518 -
519 -**Advanced Log Function:**
520 -```json
521 -{
522 - "accepted_params": [
523 - "info", "after", "before", "direction", "last", "anchor",
524 - "query", "facets", "histogram", "if_modified_since",
525 - "data_only", "delta", "tail", "sampling", "slice"
526 - ]
527 -}
528 -```
529 -
530 -**UI Feature Enablement:**
531 -- `"slice"` → Enables "Full data queries" toggle
532 -- `"direction"` → Enables bidirectional pagination
533 -- `"tail"` → Enables streaming mode
534 -- `"delta"` → Enables incremental updates
535 -- `"query"` → Enables full-text search
536 -
537 -## Advanced Log Explorer Features
538 -
539 -### Column Options for Logs
540 -
541 -Log explorers support special column options:
542 -
543 -| Option | Effect |
544 -|--------|--------|
545 -| `"fts"` | Full-text searchable by query |
546 -| `"facet"` | Appears in sidebar filters with counts |
547 -| `"main_text"` | Primary content (usually message) |
548 -| `"rich_text"` | May contain formatting |
549 -| `"hidden"` | Hide by default |
550 -
551 -Example:
552 -```json
553 -{
554 - "message": {
555 - "type": "string",
556 - "full_width": true,
557 - "options": ["full_width", "wrap", "visible", "main_text", "fts", "rich_text"]
558 - }
559 -}
560 -```
561 -
562 -### Log-Specific UI Behavior
563 -
564 -When `has_history: true`:
565 -- **Sidebar**: Shows faceted filters instead of simple filters
566 -- **Search box**: Queries all `fts` fields using pattern matching
567 -- **Infinite scroll**: Loads more data as you scroll
568 -- **Time navigation**: Jump to specific time periods
569 -- **Sampling**: For very large datasets
570 -
571 ----
572 -
573 -# Part 3: Complete Options Reference
574 -
575 -This section documents every field type, option, and behavior for quick reference while developing.
576 -
577 -## Field Types and UI Rendering
578 -
579 -### Text and Categories
580 -
581 -```json
582 -{
583 - "type": "string",
584 - "visualization": "value" // Default: plain text, left-aligned
585 -}
586 -```
587 -
588 -```json
589 -{
590 - "type": "string",
591 - "visualization": "pill" // Colored badges for status/categories
592 -}
593 -```
594 -
595 -### Numbers and Metrics
596 -
597 -```json
598 -{
599 - "type": "integer", // Right-aligned numbers
600 - "transform": "number", // Respect decimal_points
601 - "decimal_points": 2
602 -}
603 -```
604 -
605 -```json
606 -{
607 - "type": "bar-with-integer", // Progress bars with values
608 - "max": 100, // Required for bars
609 - "units": "%"
610 -}
611 -```
612 -
613 -### Timestamp Format Requirements
614 -
615 -**Simple Tables**: Use milliseconds with `datetime` transform
616 -```json
617 -{
618 - "type": "timestamp",
619 - "transform": "datetime", // Expects milliseconds
620 - "data": [1697644320000] // JavaScript Date format
621 -}
622 -```
623 -
624 -**Log Explorers**: Use microseconds with `datetime_usec` transform
625 -```json
626 -{
627 - "type": "timestamp",
628 - "transform": "datetime_usec", // Expects microseconds
629 - "data": [1697644320000000] // Microsecond precision
630 -}
631 -```
632 -
633 -**Frontend Conversion:**
634 -```javascript
635 -// datetime_usec automatically converts to milliseconds
636 -if (usec) {
637 - epoch = epoch ? Math.floor(epoch / 1000) : epoch
638 -}
639 -```
640 -
641 -**API Parameters**: `after` and `before` are automatically converted from milliseconds to seconds when sent to functions.
642 -
643 -```json
644 -{
645 - "type": "duration",
646 - "transform": "duration_s" // Formats seconds as "1d 2h 3m"
647 -}
648 -```
649 -
650 -### Rich Content
651 -
652 -```json
653 -{
654 - "type": "feedTemplate", // Full-width rich content
655 - "full_width": true // Automatically applied
656 -}
657 -```
658 -
659 -## Essential Column Options
660 -
661 -### Layout Control
662 -
663 -```json
664 -{
665 - "unique_key": true, // Row identifier (exactly one required)
666 - "sticky": true, // Pin when scrolling horizontally
667 - "visible": true, // Show by default
668 - "full_width": true, // Expand to fill available space
669 - "wrap": true // Enable text wrapping
670 -}
671 -```
672 -
673 -### Filtering Options
674 -
675 -```json
676 -{
677 - "filter": "multiselect" // Checkboxes (default for simple tables)
678 -}
679 -```
680 -
681 -```json
682 -{
683 - "filter": "range" // Min/max sliders for numbers
684 -}
685 -```
686 -
687 -```json
688 -{
689 - "filter": "facet" // Sidebar with counts (log explorers only)
690 -}
691 -```
692 -
693 -### Sorting Options
694 -
695 -```json
696 -{
697 - "sort": "descending", // Default sort direction
698 - "sortable": true // Allow user sorting (default)
699 -}
700 -```
701 -
702 -```json
703 -{
704 - "sort": "descending|fixed", // Prevent user from changing sort
705 - "sortable": false
706 -}
707 -```
708 -
709 -## Row-Level Features
710 -
711 -### Row Coloring by Severity
712 -
713 -Add row coloring by including `rowOptions` as the last element:
714 -
715 -```json
716 -{
717 - "data": [
718 - ["normal data", "values", {"rowOptions": {"severity": "normal"}}],
719 - ["warning data", "values", {"rowOptions": {"severity": "warning"}}],
720 - ["error data", "values", {"rowOptions": {"severity": "error"}}],
721 - ["notice data", "values", {"rowOptions": {"severity": "notice"}}]
722 - ]
723 -}
724 -```
725 -
726 -**Severity Levels:**
727 -- `"normal"` - Default appearance
728 -- `"warning"` - Yellow background
729 -- `"error"` - Red background
730 -- `"notice"` - Blue background
731 -
732 -## Chart and Grouping Configuration
733 -
734 -### Chart Definition
735 -
736 -```json
737 -{
738 - "charts": {
739 - "resource_usage": {
740 - "name": "Resource Usage",
741 - "type": "stacked-bar",
742 - "columns": ["cpu", "memory", "disk"],
743 - "groupBy": "column",
744 - "aggregation": "sum"
745 - },
746 - "status_distribution": {
747 - "name": "Status Distribution",
748 - "type": "doughnut",
749 - "columns": ["count"],
750 - "groupBy": "all",
751 - "aggregation": "count"
752 - }
753 - },
754 - "default_charts": [
755 - ["resource_usage", "service_type"],
756 - ["status_distribution", "status"]
757 - ]
758 -}
759 -```
760 -
761 -### Grouping Configuration
762 -
763 -```json
764 -{
765 - "group_by": {
766 - "aggregated": [
767 - {
768 - "id": "by_service",
769 - "name": "By Service Type",
770 - "column": "service_type"
771 - },
772 - {
773 - "id": "by_status",
774 - "name": "By Status",
775 - "column": "status"
776 - }
777 - ]
778 - }
779 -}
780 -```
781 -
782 -## Backend Implementation Examples
783 -
784 -### C Code for Simple Tables
785 -
786 -```c
787 -// Add a progress bar column
788 -buffer_rrdf_table_add_field(
789 - wb, field_id++, "cpu", "CPU Usage",
790 - RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
791 - RRDF_FIELD_VISUAL_BAR,
792 - RRDF_FIELD_TRANSFORM_NUMBER,
793 - 2, "%", 100.0, RRDF_FIELD_SORT_DESCENDING, NULL,
794 - RRDF_FIELD_SUMMARY_SUM,
795 - RRDF_FIELD_FILTER_RANGE,
796 - RRDF_FIELD_OPTS_VISIBLE, NULL
797 -);
798 -
799 -// Add row coloring
800 -buffer_json_add_array_item_object(wb);
801 -buffer_json_member_add_object(wb, "rowOptions");
802 -buffer_json_member_add_string(wb, "severity", "error");
803 -buffer_json_object_close(wb);
804 -buffer_json_object_close(wb);
805 -```
806 -
807 -### Using the Facets Library (Log Explorers)
808 -
809 -```c
810 -// Initialize facets for log exploration
811 -FACETS *facets = facets_create(...);
812 -
813 -// Set up query search
814 -facets_set_query(facets, query_string);
815 -
816 -// Add a faceted field
817 -facets_register_facet_id(facets, "level",
818 - FACET_KEY_OPTION_FACET | FACET_KEY_OPTION_FTS | FACET_KEY_OPTION_REORDER);
819 -
820 -// Generate the response
821 -facets_table_config(facets, wb);
822 -```
823 -
824 -## Error Handling
825 -
826 -Always handle the info request first:
827 -
828 -```json
829 -{
830 - "v": 3, // Enable POST requests
831 - "status": 200,
832 - "type": "table",
833 - "has_history": false, // or true for log explorers
834 - "help": "Function description",
835 - "accepted_params": [...],
836 - "required_params": [...]
837 -}
838 -```
839 -
840 -### Complete Info Response Example
841 -
842 -```json
843 -{
844 - "v": 3,
845 - "status": 200,
846 - "type": "table",
847 - "has_history": true,
848 - "help": "System log explorer with faceted search",
849 - "accepted_params": [
850 - "info", "after", "before", "direction", "last", "anchor",
851 - "query", "facets", "histogram", "if_modified_since",
852 - "data_only", "delta", "tail", "sampling"
853 - ],
854 - "required_params": [
855 - {
856 - "id": "unit",
857 - "name": "System Unit",
858 - "type": "select",
859 - "options": [
860 - {"id": "nginx.service", "name": "Nginx"},
861 - {"id": "mysql.service", "name": "MySQL"}
862 - ]
863 - }
864 - ]
865 -}
866 -```
867 -
868 -**Critical Fields:**
869 -- `"v": 3` enables POST requests with JSON payloads
870 -- `accepted_params` determines which parameters the function accepts
871 -- `required_params` generates filter UI and validates execution
872 -
873 -For errors, return:
874 -
875 -```json
876 -{
877 - "status": 400,
878 - "error_message": "Descriptive error message"
879 -}
880 -```
881 -
882 -### Performance Optimization
883 -
884 -**For Large Datasets:**
885 -
886 -1. **Enable Sampling**:
887 -```json
888 -{
889 - "accepted_params": ["sampling"],
890 - "sampling": 10 // 1 in 10 sampling
891 -}
892 -```
893 -
894 -2. **Use Delta Updates**:
895 -```json
896 -{
897 - "if_modified_since": 1697644320000000,
898 - "delta": true,
899 - "data_only": true
900 -}
901 -```
902 -
903 -3. **Implement Tail Limiting**:
904 -```c
905 -// Limit tail data to prevent memory issues
906 -if (tail_mode) {
907 - limit_results_to(500);
908 -}
909 -```
910 -
911 -4. **Support Anchor Pagination**:
912 -```json
913 -{
914 - "pagination": {
915 - "enabled": true,
916 - "column": "timestamp",
917 - "key": "anchor",
918 - "units": "timestamp_usec"
919 - }
920 -}
921 -```
922 -
923 -## Best Practices Summary
924 -
925 -### For Simple Tables
926 -1. Always include one `unique_key` column
927 -2. Use `bar-with-integer` for metrics with `max` values
928 -3. Add `filter: "range"` for numbers, `"multiselect"` for categories
929 -4. Use `rowOptions` for status indication
930 -5. Set meaningful `default_sort_column`
931 -6. Define column `summary` types for grouping (SUM for metrics, COUNT for processes)
932 -7. Add charts for key metrics with appropriate `groupBy` and `aggregation`
933 -8. Include `group_by` options for common analysis patterns
934 -
935 -### For Log Explorers
936 -1. Set `has_history: true`
937 -2. Use microsecond timestamps with `datetime_usec`
938 -3. Mark important fields with `"fts"` for search
939 -4. Use `filter: "facet"` instead of `"multiselect"`
940 -5. Include proper facets with counts
941 -6. Implement anchor-based pagination
942 -
943 -### General
944 -1. Test with `?info` requests first
945 -2. Include helpful `help` text
946 -3. Handle errors gracefully
947 -4. Use appropriate `units` for clarity
948 -5. Follow existing function patterns in your codebase
949 -
950 -
951 ----
952 -
953 -This guide covers everything you need to build both simple monitoring functions and advanced log explorers. For implementation details and edge cases, see [FUNCTIONS_REFERENCE.md](FUNCTIONS_REFERENCE.md).
src/crates/netdata-plugin/docs/FUNCTION_UI_REFERENCE.md deleted
-1677
@@ -1,1677 +0,0 @@
1 -# Netdata Functions v3 Protocol - Technical Reference
2 -
3 -> **Note**: This is the technical specification. For a practical guide to implementing functions, see [FUNCTIONS_DEVELOPER_GUIDE.md](FUNCTIONS_DEVELOPER_GUIDE.md).
4 -
5 -## Overview
6 -
7 -This document provides the complete technical reference for Netdata Functions protocol v3, combining all information about simple tables (has_history=false) and log explorers (has_history=true). This is the authoritative internal documentation for maintaining and extending Netdata functions.
8 -
9 -## Table of Contents
10 -
11 -1. [Protocol Overview](#protocol-overview)
12 -2. [Request Flow](#request-flow)
13 -3. [Simple Table Format](#simple-table-format)
14 -4. [Log Explorer Format](#log-explorer-format)
15 -5. [Field Types and Enumerations](#field-types-and-enumerations)
16 -6. [UI Implementation](#ui-implementation)
17 -7. [Backend Implementation](#backend-implementation)
18 -8. [Best Practices](#best-practices)
19 -9. [Known Functions](#known-functions)
20 -10. [Development Checklist](#development-checklist)
21 -
22 -## Protocol Overview
23 -
24 -Netdata Functions allow collectors/plugins to expose interactive data through a streaming protocol. The protocol has evolved from GET-based CLI parameters (legacy) to POST-based JSON payloads (modern v3).
25 -
26 -### Function Types
27 -
28 -1. **Simple Table View** (`has_history: false`)
29 - - Basic tabular data display with frontend-side filtering and search
30 - - Examples: `processes`, `network-connections`, `block-devices`
31 -
32 -2. **Log Explorer Format** (`has_history: true`)
33 - - Advanced table with backend-powered faceted search, histograms, and infinite scroll
34 - - Examples: `systemd-journal`, `windows-events`
35 -
36 -### Critical Implementation Differences
37 -
38 -| Aspect | Simple Tables | Log Explorers |
39 -|--------|---------------|---------------|
40 -| **Data Processing** | All data sent to frontend | Backend filters before sending |
41 -| **Facet Counts** | Frontend counts occurrences in received data | Backend computes counts during query execution |
42 -| **Full-Text Search** | Frontend substring search across visible data | Backend pattern matching with facets library |
43 -| **Histograms** | Not supported - no time-based visualization | Optional - backend generates Netdata chart format |
44 -| **Performance** | Limited by browser memory and processing | Scales to millions of records with sampling |
45 -| **Query Parameter** | Ignored by backend (frontend only) | Processed by backend using simple patterns |
46 -
47 -## Request Flow
48 -
49 -### Modern Flow (v:3)
50 -
51 -1. **Info Request** (Always GET)
52 - ```
53 - GET /api/v3/function?function=systemd-journal info after:1234567890 before:1234567890
54 - ```
55 -
56 -2. **Info Response**
57 - ```json
58 - {
59 - "v": 3, // Indicates POST should be used for data requests
60 - "accepted_params": [...],
61 - "required_params": [...],
62 - "status": 200,
63 - "type": "table",
64 - "has_history": false
65 - }
66 - ```
67 -
68 -3. **Data Request** (POST when v=3)
69 - ```json
70 - {
71 - "query": "*error* !*debug*",
72 - "selections": {
73 - "priority": ["error", "warning"],
74 - "unit": ["nginx.service"]
75 - },
76 - "after": 1234567890,
77 - "before": 1234567890,
78 - "last": 100
79 - }
80 - ```
81 -
82 -### Preflight Info Request Details
83 -
84 -**Request Format:**
85 -```
86 -GET /api/v3/function?function=systemd-journal info after:1234567890 before:1234567890
87 -```
88 -
89 -**Required Response Fields:**
90 -```json
91 -{
92 - "v": 3,
93 - "status": 200,
94 - "type": "table",
95 - "has_history": false,
96 - "accepted_params": ["info", "after", "before", "direction", "last"],
97 - "required_params": [
98 - {
99 - "id": "priority",
100 - "name": "Log Level",
101 - "type": "select",
102 - "options": [
103 - {"id": "error", "name": "Error", "defaultSelected": true},
104 - {"id": "warn", "name": "Warning"}
105 - ]
106 - }
107 - ],
108 - "help": "Function description"
109 -}
110 -```
111 -
112 -**Frontend Processing:**
113 -- `accepted_params`: Validates which parameters can be sent to function
114 -- `required_params`: Generates filter UI, prevents execution if missing
115 -- `v: 3`: Enables POST requests with JSON payloads
116 -- Missing required parameters show user-friendly error messages
117 -
118 -### Backend Implementation
119 -
120 -```c
121 -// Simplified from logs_query_status.h
122 -if(payload) {
123 - // POST request - parse JSON payload
124 - facets_use_hashes_for_ids(facets, false); // Use plain field names
125 - rq->fields_are_ids = false;
126 -} else {
127 - // GET request - parse CLI parameters (legacy)
128 - facets_use_hashes_for_ids(facets, true); // Use hash IDs
129 - rq->fields_are_ids = true;
130 -}
131 -```
132 -
133 -### Key Differences
134 -
135 -| Aspect | GET (Legacy) | POST (Modern v3) |
136 -|--------|--------------|------------------|
137 -| Version | v \< 3 | v = 3 |
138 -| Field IDs | 11-char hashes | Plain names |
139 -| Parameters | URL encoded | JSON body |
140 -| Facet filters | `field_hash:value1,value2` | `{"field": ["value1", "value2"]}` |
141 -| Full-text search | `query:search terms` | `{"query": "search terms"}` |
142 -
143 -### Standard Accepted Parameters
144 -
145 -**Core Parameters (All Functions):**
146 -- `"info"` - Function info requests (always supported)
147 -- `"after"` - Time range start (seconds epoch)
148 -- `"before"` - Time range end (seconds epoch)
149 -
150 -**Log Explorer Parameters (has_history=true):**
151 -- `"direction"` - Query direction: `"backward"` | `"forward"`
152 -- `"last"` - Result limit (default: 200)
153 -- `"anchor"` - Pagination cursor (timestamp or row identifier)
154 -- `"query"` - Full-text search using Netdata simple patterns
155 -- `"facets"` - Facet selection: `"field1,field2"`
156 -- `"histogram"` - Histogram field selection
157 -- `"if_modified_since"` - Conditional updates (microseconds epoch)
158 -- `"data_only"` - Skip metadata in response (boolean)
159 -- `"delta"` - Incremental responses (boolean)
160 -- `"tail"` - Streaming mode (boolean)
161 -- `"sampling"` - Data sampling control
162 -
163 -**UI Feature Parameters:**
164 -- `"slice"` - Enables "Full data queries" toggle in UI
165 -
166 -**Frontend Behavior:**
167 -```javascript
168 -// Only accepted parameters are sent to functions
169 -const allowedFilterIds = [...selectedFacets, ...requiredParamIds, ...acceptedParams]
170 -filtersToSend = allowedFilterIds.reduce((acc, filterId) => {
171 - if (filterId in filters) acc[filterId] = filters[filterId]
172 - return acc
173 -}, {})
174 -```
175 -
176 -### Query Parameter (Netdata Simple Patterns)
177 -
178 -The `query` parameter uses Netdata's simple pattern matching for full-text search across all fields:
179 -
180 -**Pattern Syntax:**
181 -- `|` - Pattern separator (OR logic between patterns)
182 -- `*` - Wildcard matching any number of characters
183 -- `!` - Negates the pattern (exclude matches)
184 -- Spaces are matched literally (not pattern separators)
185 -- Case-insensitive matching
186 -- Default behavior is substring matching (no wildcards needed)
187 -
188 -**Matching Modes:**
189 -- `pattern` - Substring match (default) - finds "pattern" anywhere
190 -- `*pattern` - Suffix match - finds strings ending with "pattern"
191 -- `pattern*` - Prefix match - finds strings starting with "pattern"
192 -- `*pattern*` - Substring match (explicit) - same as default
193 -
194 -**Examples:**
195 -```json
196 -{
197 - "query": "error" // Finds "error" anywhere (substring)
198 - "query": "error|warning" // Finds "error" OR "warning"
199 - "query": "error|warning|critical" // Multiple OR patterns
200 - "query": "!debug" // Exclude ALL rows containing "debug"
201 - "query": "!*debugging*|*debug*" // Include "debug" but exclude "debugging"
202 - "query": "connection failed" // Find exact phrase (spaces included)
203 - "query": "*error" // Find strings ending with "error"
204 - "query": "nginx*" // Find strings starting with "nginx"
205 -}
206 -```
207 -
208 -**Pattern Evaluation Rules:**
209 -
210 -1. **Within a field**: Left-to-right, first match wins
211 - - `"!*debugging*|*debug*"` - If text contains "debugging", it's negative match. Otherwise, if contains "debug", it's positive match.
212 -
213 -2. **Across all fields**: ALL fields are evaluated (no short-circuit)
214 - - Every field with FTS enabled is checked against the pattern
215 - - Positive and negative matches are counted separately
216 - - Field evaluation order doesn't affect the outcome
217 -
218 -3. **Row decision (after all fields evaluated)**:
219 - - **Excluded if**: ANY field has a negative match (regardless of positive matches)
220 - - **Included if**: At least one positive match AND zero negative matches
221 - - **Excluded if**: No positive matches found
222 -
223 -**Example**: Query `"!*debugging*|*debug*"`
224 -```
225 -Row 1: message="debug info", category="debugging tips"
226 - → message: positive match (debug)
227 - → category: negative match (debugging)
228 - → Result: EXCLUDED (has negative match)
229 -
230 -Row 2: message="debug info", category="testing"
231 - → message: positive match (debug)
232 - → category: no match
233 - → Result: INCLUDED (positive match, no negative)
234 -
235 -Row 3: message="error info", category="testing"
236 - → message: no match
237 - → category: no match
238 - → Result: EXCLUDED (no positive matches)
239 -```
240 -
241 -**Key Point**: The order fields are evaluated doesn't matter - the same counters are updated and the same decision is made regardless of whether positive or negative matches are found first.
242 -
243 -**Common Use Cases:**
244 -- `"timeout|failed|refused"` - Find various connection issues
245 -- `"!trace|!debug|nginx|apache"` - Find web server logs, but exclude debug/trace
246 -- `"error code 500"` - Find exact phrase with spaces
247 -- `"critical|fatal|emergency"` - Find severe log levels
248 -- `"!*test*|!*debug*|*"` - Include everything except test and debug content
249 -
250 -**Implementation Details:**
251 -- Uses `simple_pattern_create(query, "|", SIMPLE_PATTERN_SUBSTRING, false)`
252 -- Searches all fields marked with `FACET_KEY_OPTION_FTS` or when `FACETS_OPTION_ALL_KEYS_FTS` is set
253 -- No support for `?` single-character wildcards
254 -- Escaping with `\` is supported for literal matches
255 -
256 -**Important**: Hash IDs (like `priority_hash`) are obsolete and only exist for backward compatibility with GET requests. All new functions should use v:3 with plain field names.
257 -
258 -## Simple Table Format
259 -
260 -### Complete Structure
261 -
262 -```json
263 -{
264 - // Required fields
265 - "status": 200,
266 - "type": "table",
267 - "has_history": false,
268 - "data": [
269 - [value1, value2, value3, ...], // Row 1
270 - [value1, value2, value3, ...], // Row 2
271 - // Special rowOptions as last element
272 - [..., {"rowOptions": {"severity": "warning|error|notice|normal"}}]
273 - ],
274 - "columns": {
275 - "column_name": {
276 - // Required
277 - "index": 0, // Position in data array
278 - "name": "Display Name", // Column header
279 - "type": "string", // Field type
280 -
281 - // Optional
282 - "unique_key": false, // Row identifier (exactly one required)
283 - "visible": true, // Default visibility
284 - "sticky": false, // Pin when scrolling
285 - "visualization": "value", // How to render
286 - "transform": "none", // Value transformation
287 - "decimal_points": 2, // For numbers
288 - "units": "bytes", // Display units
289 - "max": 100, // For bar types
290 - "sort": "descending", // Default sort
291 - "sortable": true, // User can sort
292 - "filter": "multiselect", // Filter type
293 - "full_width": false, // Expand to fill
294 - "wrap": false, // Text wrapping
295 - "summary": "sum" // Aggregation (backend only)
296 - }
297 - },
298 -
299 - // Optional extensions
300 - "help": "Function description",
301 - "update_every": 1,
302 - "expires": 1234567890,
303 - "default_sort_column": "column_name",
304 - "group_by": {
305 - "aggregated": [{
306 - "id": "group_id",
307 - "name": "Group Name",
308 - "column": "column_to_group_by"
309 - }]
310 - },
311 - "charts": {
312 - "chart_id": {
313 - "name": "Chart Name",
314 - "type": "stacked|bar",
315 - "columns": ["col1", "col2"]
316 - }
317 - },
318 - "accepted_params": [{
319 - "id": "param_id",
320 - "name": "Parameter Name",
321 - "type": "string|integer|boolean",
322 - "default": "default_value"
323 - }],
324 - "required_params": ["param1", "param2"]
325 -}
326 -```
327 -
328 -### Row Options
329 -
330 -Special last element in data array for row styling:
331 -
332 -```json
333 -[..., {"rowOptions": {"severity": "error"}}] // Red background
334 -[..., {"rowOptions": {"severity": "warning"}}] // Yellow background
335 -[..., {"rowOptions": {"severity": "notice"}}] // Blue background
336 -[..., {"rowOptions": {"severity": "normal"}}] // Default appearance
337 -```
338 -
339 -## Log Explorer Format
340 -
341 -### Complete Structure
342 -
343 -```json
344 -{
345 - // Basic fields (same as simple table)
346 - "status": 200,
347 - "type": "table",
348 - "has_history": true, // REQUIRED: Enables log explorer UI
349 - "help": "System log explorer",
350 - "update_every": 1,
351 -
352 - // Table metadata
353 - "table": {
354 - "id": "logs",
355 - "has_history": true,
356 - "pin_alert": false
357 - },
358 -
359 - // Faceted filters (dynamic with counts)
360 - "facets": [
361 - {
362 - "id": "priority", // Plain field name (not hash)
363 - "name": "Priority",
364 - "order": 1,
365 - "defaultExpanded": true,
366 - "options": [
367 - {
368 - "id": "ERROR",
369 - "name": "ERROR",
370 - "count": 45, // Real-time count
371 - "order": 1
372 - }
373 - ]
374 - }
375 - ],
376 -
377 - // Enhanced columns
378 - "columns": {
379 - "timestamp": {
380 - "index": 0,
381 - "id": "timestamp",
382 - "name": "Time",
383 - "type": "timestamp",
384 - "transform": "datetime_usec",
385 - "sort": "descending|fixed",
386 - "sortable": false,
387 - "sticky": true
388 - },
389 - "level": {
390 - "index": 1,
391 - "id": "priority", // Links to facet
392 - "name": "Level",
393 - "type": "string",
394 - "visualization": "pill",
395 - "filter": "facet", // Not multiselect!
396 - "options": ["facet", "visible", "sticky"]
397 - },
398 - "message": {
399 - "index": 3,
400 - "id": "message",
401 - "name": "Message",
402 - "type": "string",
403 - "full_width": true,
404 - "options": [
405 - "full_width",
406 - "wrap",
407 - "visible",
408 - "main_text", // Primary content
409 - "fts", // Full-text searchable
410 - "rich_text" // May contain formatting
411 - ]
412 - }
413 - },
414 -
415 - // Data with microsecond timestamps
416 - "data": [
417 - [
418 - 1697644320000000, // Microseconds
419 - {"severity": "error"}, // rowOptions
420 - "ERROR", // level
421 - "nginx", // source
422 - "Connection failed" // message
423 - ]
424 - ],
425 -
426 - // Histogram configuration
427 - "available_histograms": [
428 - {"id": "priority", "name": "Priority", "order": 1},
429 - {"id": "source", "name": "Source", "order": 2}
430 - ],
431 - "histogram": {
432 - "id": "priority",
433 - "name": "Priority",
434 - "chart": {
435 - "summary": {/* Netdata chart metadata */},
436 - "result": {
437 - "labels": ["time", "ERROR", "WARN", "INFO"],
438 - "data": [
439 - [1697644200, 5, 12, 234],
440 - [1697644260, 3, 8, 198]
441 - ]
442 - }
443 - }
444 - },
445 -
446 - // Pagination metadata
447 - "items": {
448 - "evaluated": 50000, // Total scanned
449 - "matched": 2520, // Match filters
450 - "unsampled": 100, // Skipped (sampling)
451 - "estimated": 0, // Statistical estimate
452 - "returned": 100, // In this response
453 - "max_to_return": 100,
454 - "before": 0,
455 - "after": 2420
456 - },
457 -
458 - // Navigation anchor
459 - "anchor": {
460 - "last_modified": 1697644320000000,
461 - "direction": "backward" // or "forward"
462 - },
463 -
464 - // Request echo (optional)
465 - "request": {
466 - "query": "*error* *warning*",
467 - "filters": ["priority:error,warning"],
468 - "histogram": "priority"
469 - },
470 -
471 - // Additional metadata
472 - "expires": 1697644920000,
473 - "sampling": 10 // 1 in N sampling
474 -}
475 -```
476 -
477 -### Key Differences from Simple Tables
478 -
479 -| Feature | Simple Table | Log Explorer |
480 -|---------|--------------|--------------|
481 -| **Facet counts** | Frontend computes from data | Backend computes in facets library |
482 -| **Full-text search** | Frontend substring matching | Backend pattern matching |
483 -| **Histograms** | Not supported | Optional (backend generated) |
484 -| Filtering | Static multiselect | Dynamic facets with counts |
485 -| Pagination | All data at once | Anchor-based infinite scroll |
486 -| Time visualization | None | Histogram chart |
487 -| Navigation | None | Bi-directional with timestamps |
488 -| Performance | All data loaded | Sampling for large datasets |
489 -
490 -### Log-Specific Column Options
491 -
492 -| Option | UI Effect |
493 -|--------|-----------|
494 -| `"facet"` | Field is filterable via facets |
495 -| `"fts"` | Full-text searchable |
496 -| `"main_text"` | Primary content field |
497 -| `"rich_text"` | May contain formatting |
498 -| `"pretty_xml"` | Format as XML |
499 -| `"hidden"` | Hide by default |
500 -
501 -## Facet Value Pills and Aggregated Counts
502 -
503 -### Overview
504 -
505 -Facet values in the sidebar display pills with counts that change based on the function's aggregation mode. The UI uses a smart component that displays different formats:
506 -
507 -1. **Simple Counts**: Just the number of matching rows
508 -2. **Aggregated Counts**: Shows both aggregated count and original count with a union symbol
509 -
510 -### UI Implementation
511 -
512 -The pills are rendered using this logic:
513 -
514 -```javascript
515 -{!!actualCount && <TextSmall>{actualCount} &#8835;&nbsp;</TextSmall>}
516 -<TextSmall>{(pill || count).toString()}</TextSmall>
517 -```
518 -
519 -**Symbol**: `&#8835;` renders as `⊃` (superset symbol, looks like rotated 'u')
520 -
521 -**Display Formats**:
522 -- Simple: `"42"` (just the count)
523 -- Aggregated: `"15 ⊃ 42"` (15 aggregated items containing 42 total)
524 -
525 -### Backend Configuration
526 -
527 -Functions enable aggregated counts by including an `aggregated_view` object in their response:
528 -
529 -```json
530 -{
531 - "aggregated_view": {
532 - "column": "Count",
533 - "results_label": "unique combinations",
534 - "aggregated_label": "sockets"
535 - }
536 -}
537 -```
538 -
539 -**Example from network-connections function**:
540 -```c
541 -// In network-viewer.c when aggregated=true
542 -buffer_json_member_add_object(wb, "aggregated_view");
543 -{
544 - buffer_json_member_add_string(wb, "column", "Count");
545 - buffer_json_member_add_string(wb, "results_label", "unique combinations");
546 - buffer_json_member_add_string(wb, "aggregated_label", "sockets");
547 -}
548 -buffer_json_object_close(wb);
549 -```
550 -
551 -## Charts Configuration
552 -
553 -Functions can provide both standard charts (computed by frontend) and custom visualizations.
554 -
555 -### Standard Charts
556 -
557 -**Configuration:**
558 -```json
559 -{
560 - "charts": {
561 - "cpu_usage": {
562 - "name": "CPU Usage by Service",
563 - "type": "stacked-bar",
564 - "columns": ["user_cpu", "system_cpu"],
565 - "groupBy": "column",
566 - "aggregation": "sum"
567 - }
568 - },
569 - "default_charts": [
570 - ["cpu_usage", "service_type"]
571 - ]
572 -}
573 -```
574 -
575 -**Supported Types:**
576 -- `"bar"` - Basic bar chart
577 -- `"stacked-bar"` - Multi-column stacked bars
578 -- `"doughnut"` - Pie/doughnut chart
579 -- `"value"` - Simple numeric display
580 -
581 -**GroupBy Options:**
582 -- `"column"` (default) - Group by selected filter column
583 -- `"all"` - Aggregate all data together
584 -
585 -### Custom Charts
586 -
587 -For specialized visualizations beyond standard chart types, some functions may use predefined custom chart types:
588 -
589 -```json
590 -{
591 - "customCharts": {
592 - "network_topology": {
593 - "type": "network-viewer",
594 - "config": {
595 - "layout": "force",
596 - "showLabels": true
597 - }
598 - }
599 - }
600 -}
601 -```
602 -
603 -**Available Custom Types:**
604 -- `"network-viewer"` - Interactive network topology (for network-connections function)
605 -
606 -### Frontend Processing
607 -
608 -Charts are computed from table data:
609 -1. Frontend groups data by selected column
610 -2. Applies aggregation function (sum, count, avg, etc.)
611 -3. Renders chart with grouped results
612 -
613 -### Table Row Grouping
614 -
615 -Simple tables support grouping rows with backend-defined aggregation rules.
616 -
617 -**Backend Summary Types:**
618 -```c
619 -typedef enum {
620 - RRDF_FIELD_SUMMARY_COUNT, // Count rows in group
621 - RRDF_FIELD_SUMMARY_UNIQUECOUNT, // Count unique values
622 - RRDF_FIELD_SUMMARY_SUM, // Sum numeric values
623 - RRDF_FIELD_SUMMARY_MIN, // Minimum value
624 - RRDF_FIELD_SUMMARY_MAX, // Maximum value
625 - RRDF_FIELD_SUMMARY_MEAN, // Average value
626 - RRDF_FIELD_SUMMARY_MEDIAN, // Median value
627 -} RRDF_FIELD_SUMMARY;
628 -```
629 -
630 -**Column Summary Configuration:**
631 -```c
632 -buffer_rrdf_table_add_field(
633 - wb, field_id++, "cpu", "CPU Usage",
634 - RRDF_FIELD_TYPE_INTEGER,
635 - RRDF_FIELD_VISUAL_VALUE,
636 - RRDF_FIELD_TRANSFORM_NUMBER,
637 - 2, "%", NAN, RRDF_FIELD_SORT_DESCENDING, NULL,
638 - RRDF_FIELD_SUMMARY_SUM, // How to aggregate when grouping
639 - RRDF_FIELD_FILTER_RANGE,
640 - RRDF_FIELD_OPTS_VISIBLE, NULL
641 -);
642 -```
643 -
644 -**Group By Support:**
645 -```json
646 -{
647 - "group_by": {
648 - "aggregated": [{
649 - "id": "by_status",
650 - "name": "By Status",
651 - "column": "status"
652 - }]
653 - }
654 -}
655 -```
656 -
657 -### Frontend Processing
658 -
659 -The frontend processes table data to generate facet counts:
660 -
661 -```javascript
662 -const getFilterTableOptions = (data, { param, columns, aggregatedView } = {}) =>
663 - Object.entries(
664 - data.reduce((h, fn) => {
665 - h[fn[param]] = {
666 - count: (h[fn[param]]?.count || 0) + (fn.hidden ? 0 : 1),
667 - ...(aggregatedView && {
668 - actualCount: (h[fn[param]]?.actualCount || 0) +
669 - (fn.hidden ? 0 : fn[aggregatedView.column] || 1),
670 - actualCountLabel: aggregatedView.aggregatedLabel,
671 - countLabel: aggregatedView.resultsLabel,
672 - }),
673 - }
674 - return h
675 - }, {})
676 - ).map(([id, values]) => ({ id, ...values }))
677 -```
678 -
679 -### How It Works
680 -
681 -The aggregated count system uses an existing data column to track how many items were aggregated into each row:
682 -
683 -1. **Backend declares** which column contains the aggregation count:
684 - ```json
685 - "aggregated_view": {
686 - "column": "Count" // Use the "Count" column from data rows
687 - }
688 - ```
689 -
690 -2. **Frontend calculates** two values for each facet value:
691 - - **`count`**: How many rows have this facet value
692 - - **`actualCount`**: Sum of the aggregation column for all rows with this facet value
693 -
694 -3. **Example**: Network connections aggregated by protocol
695 -
696 - **Data returned by backend:**
697 - ```
698 - Direction | Protocol | LocalPort | RemotePort | Count
699 - ---------|----------|-----------|------------|-------
700 - Inbound | TCP | * | * | 5
701 - Inbound | UDP | * | * | 3
702 - Outbound | TCP | * | * | 7
703 - Outbound | UDP | * | * | 2
704 - ```
705 -
706 - **Facet pills displayed in UI:**
707 - - Direction facet:
708 - - Inbound: `8 ⊃ 2` (8 connections aggregated into 2 rows)
709 - - Outbound: `9 ⊃ 2` (9 connections aggregated into 2 rows)
710 - - Protocol facet:
711 - - TCP: `12 ⊃ 2` (12 connections aggregated into 2 rows)
712 - - UDP: `5 ⊃ 2` (5 connections aggregated into 2 rows)
713 -
714 - **Reading the pills**: `12 ⊃ 2` means "12 original items shown in 2 table rows"
715 -
716 -### Tooltip Content
717 -
718 -The tooltip provides human-readable context:
719 -- Simple mode: `"42 results"`
720 -- Aggregated mode: `"15 sockets aggregated in 42 unique combinations"`
721 -
722 -### Current Implementations
723 -
724 -| Function | Aggregated Mode | Trigger | Count Meaning |
725 -|----------|----------------|---------|---------------|
726 -| `network-connections` | `sockets:aggregated` | Parameter | Sockets → unique combinations |
727 -| Other functions | N/A | None currently | Single count only |
728 -
729 -### Adding Aggregated Counts
730 -
731 -To add aggregated count support to a function:
732 -
733 -1. **Backend**: Add `aggregated_view` object to response when in aggregated mode
734 -2. **Data**: Include aggregation column with numeric values
735 -3. **Frontend**: No changes needed - automatically processes based on `aggregated_view` presence
736 -
737 -## Field Types and Enumerations
738 -
739 -### Field Types (RRDF_FIELD_TYPE_*)
740 -
741 -#### Implemented in UI
742 -
743 -| Type | UI Component | Use Case | Notes |
744 -|------|--------------|----------|-------|
745 -| `string` | ValueCell | Text, names, categories | Left-aligned |
746 -| `integer` | ValueCell | Numbers, counts, IDs | Right-aligned |
747 -| `bar-with-integer` | BarCell | Percentages, metrics | Requires `max` |
748 -| `duration` | BarCell | Time intervals | Auto-formats seconds |
749 -| `timestamp` | DatetimeCell* | Date/time points | *UI maps to datetime |
750 -| `feedTemplate` | FeedTemplateCell | Rich content | Auto full_width |
751 -
752 -#### Fallback Implementation
753 -
754 -| Type | Behavior | Notes |
755 -|------|----------|-------|
756 -| `boolean` | ValueCell | No special boolean UI |
757 -| `detail-string` | ValueCell | No expandable functionality |
758 -| `array` | ValueCell | Works with `pill` visualization |
759 -| `none` | ValueCell | Avoid using |
760 -
761 -### Visual Types (RRDF_FIELD_VISUAL_*)
762 -
763 -| Type | UI Component | Use Case |
764 -|------|--------------|----------|
765 -| `value` | ValueCell | Standard text display (default) |
766 -| `bar` | BarCell | Progress bar without text |
767 -| `pill` | PillCell | Badge/tag display |
768 -| `richValue` | RichValueCell | Enhanced value display |
769 -| `feedTemplate` | FeedTemplateCell | Full-width template |
770 -| `rowOptions` | null | Special row configuration |
771 -
772 -**Fallback Behavior**: `gauge` is not a recognized visualization. If used, it is ignored, and the renderer falls back to using the field's `type` to select a component.
773 -
774 -### Transform Types (RRDF_FIELD_TRANSFORM_*)
775 -
776 -| Type | Input | Output | Notes |
777 -|------|-------|--------|-------|
778 -| `none` | Any | Unchanged | Default |
779 -| `number` | Number | Formatted with decimals | Uses `decimal_points` |
780 -| `duration` | Seconds | "Xd Yh Zm" | Human-readable |
781 -| `datetime` | Epoch ms | Localized date/time | |
782 -| `datetime_usec` | Epoch μs | Localized date/time | For logs |
783 -| `xml` | XML string | Formatted XML | No specialized UI |
784 -
785 -### Conditional Patterns and Dependencies
786 -
787 -#### Type and Transform Compatibility
788 -
789 -Not all `transform` values are compatible with all `type` values. The backend enforces the following compatibility rules:
790 -
791 -| Field Type (`type`) | Compatible Transforms (`transform`) |
792 -|---|---|
793 -| `timestamp` | `datetime_ms`, `datetime_usec` |
794 -| `duration` | `duration_s` |
795 -| `integer`, `bar-with-integer` | `number` |
796 -| `string`, `boolean`, `array` | `none`, `xml` |
797 -
798 -Using an incompatible transform will result in unexpected behavior or errors.
799 -
800 -#### `rowOptions` Dummy Column
801 -
802 -To add `rowOptions` for row-level severity styling, a special "dummy" column must be added to the `columns` definition. This column is not displayed in the UI but provides the necessary metadata. It must be created with this specific combination of values:
803 -
804 -* **type**: `none` (`RRDF_FIELD_TYPE_NONE`)
805 -* **visualization**: `rowOptions` (`RRDF_FIELD_VISUAL_ROW_OPTIONS`)
806 -* **options flag**: `dummy` (`RRDF_FIELD_OPTS_DUMMY`)
807 -
808 -**Example C code:**
809 -```c
810 -buffer_rrdf_table_add_field(wb, field_id++, "row_options", "Row Options",
811 - RRDF_FIELD_TYPE_NONE, RRDF_FIELD_VISUAL_ROW_OPTIONS, RRDF_FIELD_TRANSFORM_NONE,
812 - 0, NULL, NAN, RRDF_FIELD_SORT_ASCENDING, NULL,
813 - RRDF_FIELD_SUMMARY_COUNT, RRDF_FIELD_FILTER_NONE, RRDF_FIELD_OPTS_DUMMY, NULL);
814 -```
815 -
816 -### Filter Types (RRDF_FIELD_FILTER_*)
817 -
818 -| Type | UI Component | Use Case | Location |
819 -|------|--------------|----------|----------|
820 -| `multiselect` | Checkboxes | Column filtering (default) | Dynamic filters |
821 -| `range` | RangeFilter | Numeric min/max | Dynamic filters |
822 -| `facet` | Facets component | With counts (logs) | Sidebar |
823 -
824 -### Field Options (RRDF_FIELD_OPTS_*)
825 -
826 -| Option | Bit | UI Effect |
827 -|--------|-----|-----------|
828 -| `unique_key` | 0x01 | Row identifier (one required) |
829 -| `visible` | 0x02 | Show by default |
830 -| `sticky` | 0x04 | Pin column when scrolling |
831 -| `full_width` | 0x08 | Expand to fill space |
832 -| `wrap` | 0x10 | Enable text wrapping |
833 -| `dummy` | 0x20 | Internal use only |
834 -| `expanded_filter` | 0x40 | Expand filter by default |
835 -
836 -### Sort Options (RRDF_FIELD_SORT_*)
837 -
838 -- `ascending` - Sort low to high
839 -- `descending` - Sort high to low
840 -- Fixed sort (0x80) - Prevent user sorting
841 -
842 -### Summary Types (RRDF_FIELD_SUMMARY_*)
843 -
844 -Backend calculates but UI doesn't display directly:
845 -- `count`, `sum`, `min`, `max`, `mean`, `median`
846 -- `uniqueCount` - Number of unique values
847 -- `extent` - Range [min, max] (UI supported)
848 -- `unique` - List of unique values (UI supported)
849 -
850 -## UI Implementation
851 -
852 -### Column Width System
853 -
854 -The UI automatically sizes columns based on metadata:
855 -
856 -| Size | Pixels | Applied To |
857 -|------|--------|------------|
858 -| xxxs | 90px | unique_key fields, bar types |
859 -| xxs | 110px | Default for most types |
860 -| xs | 130px | Available but rarely used |
861 -| sm | 160px | timestamp, datetime types |
862 -| md-xl | 190-290px | Available but rarely used |
863 -| xxl | 1000px | feedTemplate with full_width |
864 -
865 -**Algorithm**:
866 -1. If `full_width`: → xxl with expansion
867 -2. If `unique_key`: → xxxs
868 -3. By visualization: bar → xxxs
869 -4. By type: feedTemplate → xxl, timestamp → sm
870 -5. Default → xxs
871 -
872 -### Component Mapping
873 -
874 -```javascript
875 -// Field type → Component
876 -componentByType = {
877 - "bar": BarCell,
878 - "bar-with-integer": BarCell,
879 - "duration": BarCell,
880 - "pill": PillCell,
881 - "feedTemplate": FeedTemplateCell,
882 - "datetime": DatetimeCell,
883 - // Others → ValueCell
884 -}
885 -
886 -// Visualization → Component
887 -componentByVisualization = {
888 - "bar": BarCell,
889 - "pill": PillCell,
890 - "richValue": RichValueCell,
891 - "feedTemplate": FeedTemplateCell,
892 - "rowOptions": null, // Skip rendering
893 - // Others → ValueCell
894 -}
895 -```
896 -
897 -### UI Component Architecture
898 -
899 -The frontend uses a modular architecture with:
900 -- **Value Components**: Handle different field type rendering
901 -- **Table Normalizer**: Processes function responses into UI-ready format
902 -- **Filter Components**: Implement multiselect, range, and facet filtering
903 -- **Chart Components**: Standard chart rendering (bar, stacked-bar, doughnut)
904 -- **Custom Visualizations**: Extensible system for specialized charts
905 -
906 -## Backend Implementation
907 -
908 -### Key Functions
909 -
910 -```c
911 -// Add a field to the table
912 -buffer_rrdf_table_add_field(
913 - BUFFER *wb,
914 - size_t field_id,
915 - const char *key,
916 - const char *name,
917 - RRDF_FIELD_TYPE type,
918 - RRDF_FIELD_VISUAL visual,
919 - RRDF_FIELD_TRANSFORM transform,
920 - size_t decimal_points,
921 - const char *units,
922 - NETDATA_DOUBLE max,
923 - RRDF_FIELD_SORT sort,
924 - const char *pointer_to_dim_in_rrdr,
925 - RRDF_FIELD_SUMMARY summary,
926 - RRDF_FIELD_FILTER filter,
927 - RRDF_FIELD_OPTS options,
928 - const char *default_value
929 -);
930 -```
931 -
932 -### Real Examples
933 -
934 -```c
935 -// CPU usage with progress bar
936 -buffer_rrdf_table_add_field(
937 - wb, field_id++, "CPU", "CPU %",
938 - RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
939 - RRDF_FIELD_VISUAL_BAR,
940 - RRDF_FIELD_TRANSFORM_NUMBER,
941 - 2, "%", 100.0, RRDF_FIELD_SORT_DESCENDING, NULL,
942 - RRDF_FIELD_SUMMARY_SUM,
943 - RRDF_FIELD_FILTER_RANGE,
944 - RRDF_FIELD_OPTS_VISIBLE, NULL
945 -);
946 -
947 -// Process name (unique key)
948 -buffer_rrdf_table_add_field(
949 - wb, field_id++, "Name", "Name",
950 - RRDF_FIELD_TYPE_STRING,
951 - RRDF_FIELD_VISUAL_VALUE,
952 - RRDF_FIELD_TRANSFORM_NONE,
953 - 0, NULL, NAN, RRDF_FIELD_SORT_ASCENDING, NULL,
954 - RRDF_FIELD_SUMMARY_COUNT,
955 - RRDF_FIELD_FILTER_MULTISELECT,
956 - RRDF_FIELD_OPTS_VISIBLE | RRDF_FIELD_OPTS_STICKY | RRDF_FIELD_OPTS_UNIQUE_KEY,
957 - NULL
958 -);
959 -
960 -// Row options in data
961 -buffer_json_add_array_item_object(wb);
962 -buffer_json_member_add_object(wb, "rowOptions");
963 -buffer_json_member_add_string(wb, "severity", "error");
964 -buffer_json_object_close(wb);
965 -buffer_json_object_close(wb);
966 -```
967 -
968 -### Key Backend Files
969 -
970 -- **Enums**: `netdata/src/libnetdata/buffer/functions_fields.h`
971 -- **Implementation**: `netdata/src/libnetdata/buffer/functions_fields.c`
972 -- **Facets Library**: `netdata/src/libnetdata/facets/`
973 -- **Example Functions**:
974 - - `apps.plugin/apps_functions.c`
975 - - `network-viewer.plugin/network-viewer.c`
976 - - `systemd-journal.plugin/systemd-journal.c`
977 -
978 -## Best Practices
979 -
980 -### Always Include
981 -- One field with `unique_key` option
982 -- Meaningful `name` for display
983 -- Appropriate `type` for the data
984 -- Info response with `"v": 3`
985 -
986 -### For Numeric Data
987 -- Use `bar-with-integer` for percentages
988 -- Set appropriate `max` value
989 -- Include `units` for clarity
990 -- Use `range` filter
991 -
992 -### For Time Data
993 -- `duration` type for intervals
994 -- `timestamp` type for points in time
995 -- Use appropriate transform
996 -
997 -### For Status/Severity
998 -- Include `rowOptions` as last array element
999 -- Use standard severity levels: error, warning, notice, normal
1000 -
1001 -### For Filtering
1002 -- `multiselect` (default) for categories
1003 -- `range` for numeric data
1004 -- `facet` only for has_history=true
1005 -
1006 -### Common Patterns
1007 -- Numeric metrics → `bar-with-integer` + `range` filter
1008 -- Categories → `string` + `multiselect` filter
1009 -- Time intervals → `duration` + `duration` transform
1010 -- Status → `string` + `pill` visualization
1011 -- Row coloring → `rowOptions` with severity
1012 -
1013 -## Known Functions
1014 -
1015 -### Simple Table Functions (has_history=false)
1016 -
1017 -| Function | Plugin | Key Features |
1018 -|----------|--------|--------------|
1019 -| processes | apps.plugin | CPU/memory bars, grouping |
1020 -| socket | ebpf.plugin | Complex filters, charts |
1021 -| network-connections | network-viewer.plugin | Aggregated views, severity |
1022 -| systemd-list-units | systemd-units.plugin | Unit status, severity |
1023 -| ipmi-sensors | freeipmi.plugin | Hardware monitoring |
1024 -| block-devices | proc.plugin | I/O statistics, charts |
1025 -| network-interfaces | proc.plugin | Network stats, severity |
1026 -| mount-points | diskspace.plugin | Filesystem usage |
1027 -| cgroup-top | cgroups.plugin | Container metrics |
1028 -| systemd-top | cgroups.plugin | Service metrics |
1029 -| metrics-cardinality | web api | Dynamic columns |
1030 -| streaming | web api | Replication status |
1031 -| all-queries | web api | Monitors the progress of in-flight queries |
1032 -
1033 -### Log Explorer Functions (has_history=true)
1034 -
1035 -| Function | Plugin | Key Features |
1036 -|----------|--------|--------------|
1037 -| systemd-journal | systemd-journal.plugin | System logs, faceted search |
1038 -| windows-events | windows-events.plugin | Windows logs, faceted search |
1039 -
1040 -
1041 -## Development Checklist
1042 -
1043 -### Creating a New Function
1044 -
1045 -- [ ] Choose function type (simple table or log explorer)
1046 -- [ ] Implement info response with `"v": 3`
1047 -- [ ] Define columns with appropriate types and options
1048 -- [ ] Include one `unique_key` field
1049 -- [ ] Add proper error handling
1050 -- [ ] Test with POST requests
1051 -- [ ] Document accepted/required parameters
1052 -
1053 -### Format Validation
1054 -
1055 -- [ ] Verify JSON structure matches specification
1056 -- [ ] Check all required fields are present
1057 -- [ ] Test empty result sets
1058 -- [ ] Test large datasets
1059 -- [ ] Verify error responses
1060 -- [ ] Test special characters in data
1061 -- [ ] Check numeric precision
1062 -- [ ] Verify date/time formatting
1063 -- [ ] Test sorting and filtering
1064 -
1065 -### UI Integration
1066 -
1067 -- [ ] Confirm field types map to UI components
1068 -- [ ] Verify filters work correctly
1069 -- [ ] Check column widths display properly
1070 -- [ ] Test row coloring with severity
1071 -- [ ] Verify transforms apply correctly
1072 -- [ ] Check responsive behavior
1073 -
1074 -### Performance
1075 -
1076 -- [ ] Handle large datasets efficiently
1077 -- [ ] Implement sampling for logs if needed
1078 -- [ ] Set appropriate `update_every`
1079 -- [ ] Consider pagination/anchoring
1080 -- [ ] Test with concurrent requests
1081 -
1082 -## Corner Cases and Edge Handling
1083 -
1084 -### Empty Result Sets
1085 -
1086 -The protocol handles empty results gracefully:
1087 -
1088 -```json
1089 -{
1090 - "status": 200,
1091 - "type": "table",
1092 - "has_history": false,
1093 - "columns": {...}, // Full column definitions
1094 - "data": [] // Empty array is valid
1095 -}
1096 -```
1097 -
1098 -- Minimum valid response: `{"status": 200, "type": "table", "columns": {}, "data": []}`
1099 -- UI displays "No data available" message
1100 -- Column headers still render for context
1101 -
1102 -### Null and Missing Values
1103 -
1104 -- `null` values in data arrays are rendered as empty cells
1105 -- Missing array elements default to `null`
1106 -- `NaN` for numeric fields: For fields with `transform: "number"`, `NaN` values will be displayed as the string "NaN". For `timestamp` fields with `datetime` or `datetime_usec` transforms, `NaN` epoch values will display as empty cells.
1107 -- Empty strings render as empty cells
1108 -- Backend uses `NAN` constant for missing numeric values
1109 -
1110 -### Special Characters
1111 -
1112 -The protocol properly escapes:
1113 -- JSON special characters (`"`, `\`, control chars)
1114 -- HTML entities (escaped by React components): React automatically escapes HTML content to prevent XSS attacks. Any HTML tags or entities in the data will be displayed as literal text, not rendered as HTML.
1115 -- Unicode characters (UTF-8 support throughout)
1116 -- SQL injection protection in queries
1117 -
1118 -### Numeric Precision
1119 -
1120 -- `decimal_points` field controls display precision, using JavaScript's `toFixed()` method.
1121 -- Backend uses `NETDATA_DOUBLE` type for floating-point numbers.
1122 -- Frontend's number formatting:
1123 - - Some components use `Intl.NumberFormat` with a fixed locale (e.g., "en-US").
1124 - - Others use `toLocaleString()` which respects the browser's locale.
1125 - - Therefore, locale-specific formatting is applied in some, but not all, cases.
1126 -- Very large numbers: Due to the use of `toFixed()`, very large numbers will be displayed as a long string with the specified decimal places, not automatically in scientific notation.
1127 -- Infinity/NaN handling:
1128 - - `NaN` values in numeric fields will be displayed as the string "NaN".
1129 - - `Infinity` and `-Infinity` values will be displayed as the strings "Infinity" and "-Infinity" respectively.
1130 - - For `timestamp` fields with `datetime` or `datetime_usec` transforms, `NaN` epoch values will display as empty cells.
1131 -
1132 -### Timestamp Format Requirements
1133 -
1134 -**Simple Tables**: Use milliseconds with `datetime` transform
1135 -```json
1136 -{
1137 - "type": "timestamp",
1138 - "transform": "datetime", // Expects milliseconds
1139 - "data": [1697644320000] // JavaScript Date format
1140 -}
1141 -```
1142 -
1143 -**Log Explorers**: Use microseconds with `datetime_usec` transform
1144 -```json
1145 -{
1146 - "type": "timestamp",
1147 - "transform": "datetime_usec", // Expects microseconds
1148 - "data": [1697644320000000] // Microsecond precision
1149 -}
1150 -```
1151 -
1152 -**Frontend Conversion:**
1153 -```javascript
1154 -// datetime_usec automatically converts to milliseconds
1155 -if (usec) {
1156 - epoch = epoch ? Math.floor(epoch / 1000) : epoch
1157 -}
1158 -```
1159 -
1160 -**API Parameters**: `after` and `before` are automatically converted from milliseconds to seconds when sent to functions.
1161 -
1162 -**Display Format:**
1163 -- Timezone: Determined by URL parameter (`utc`) or system default
1164 -- Format: Uses `Intl.DateTimeFormat` with browser's locale
1165 -- Both types render as localized date/time strings with seconds precision
1166 -
1167 -### Sorting Capabilities
1168 -
1169 -- **Backend Control**: Each column defines default sort with `RRDF_FIELD_SORT_*`
1170 -- **User Control**: `sortable: true` enables UI sorting (default)
1171 -- **Fixed Sort**: Bit flag 0x80 prevents user sorting
1172 -- **Initial Sort**: `default_sort_column` specifies startup sort
1173 -- **Multi-Column**: UI supports sorting by any sortable column
1174 -- **Performance**: Client-side sorting for simple tables
1175 -
1176 -### Filtering Capabilities
1177 -
1178 -- **Multiselect**: Default filter type with checkboxes
1179 - - Shows all unique values from the column
1180 - - Multiple selections allowed
1181 - - OR logic between selections
1182 -- **Range**: Numeric filters with min/max sliders
1183 - - Requires numeric field type
1184 - - Auto-detects min/max from data
1185 - - Inclusive filtering
1186 -- **Facet**: Advanced filtering for log explorer
1187 - - Shows counts next to each option
1188 - - Dynamic updates with other filters
1189 - - Indexed for performance
1190 -
1191 -## Anchor-Based Pagination
1192 -
1193 -Log explorer functions use anchor-based pagination for efficient navigation through large datasets.
1194 -
1195 -### Configuration
1196 -
1197 -```json
1198 -{
1199 - "pagination": {
1200 - "enabled": true,
1201 - "column": "timestamp",
1202 - "key": "anchor",
1203 - "units": "timestamp_usec"
1204 - }
1205 -}
1206 -```
1207 -
1208 -### Frontend Implementation
1209 -
1210 -**Anchor Management:**
1211 -```javascript
1212 -// Frontend calculates anchors from data boundaries
1213 -anchorBefore: latestData[latestData.length - 1][pagination.column],
1214 -anchorAfter: latestData[0][pagination.column],
1215 -anchorUnits: pagination.units
1216 -```
1217 -
1218 -**Infinite Scroll Navigation:**
1219 -- **Backward**: Scroll down loads older data using `anchorBefore`
1220 -- **Forward**: Scroll up loads newer data using `anchorAfter`
1221 -- **State Tracking**: `hasNextPage`, `hasPrevPage` control load triggers
1222 -
1223 -**Required Parameters:**
1224 -- `anchor: {VALUE}` - Pagination cursor value
1225 -- `direction: "backward"|"forward"` - Navigation direction
1226 -- `last: NUMBER` - Page size (default: 200)
1227 -
1228 -### PLAY Mode Integration
1229 -
1230 -When in PLAY mode (`after < 0`), pagination automatically coordinates with real-time updates:
1231 -
1232 -```javascript
1233 -{
1234 - direction: "forward",
1235 - merge: true,
1236 - tail: true,
1237 - delta: true,
1238 - anchor: anchorAfter
1239 -}
1240 -```
1241 -
1242 -### Large Data Sets
1243 -
1244 -Simple tables load all data at once, but handle large sets efficiently:
1245 -- Virtual scrolling for thousands of rows
1246 -- Client-side filtering/sorting
1247 -- No built-in pagination (all data in response)
1248 -
1249 -Log explorer uses anchor-based pagination:
1250 -- Efficient navigation through millions of records
1251 -- Configurable page size (`last` parameter)
1252 -- Bi-directional infinite scrolling
1253 -- Sampling for very large sets
1254 -
1255 -## Incremental Updates (Delta Mode)
1256 -
1257 -Delta mode enables efficient real-time updates by sending only changes since the last request.
1258 -
1259 -### When Delta is Enabled
1260 -
1261 -```javascript
1262 -{
1263 - if_modified_since: 1697644320000000, // Previous modification timestamp
1264 - direction: "forward",
1265 - merge: true,
1266 - tail: true,
1267 - delta: true,
1268 - data_only: true,
1269 - anchor: anchorAfter
1270 -}
1271 -```
1272 -
1273 -### Delta Response Types
1274 -
1275 -**Facets Delta:**
1276 -```json
1277 -{
1278 - "facetsDelta": [
1279 - {
1280 - "id": "priority",
1281 - "options": [
1282 - {"id": "ERROR", "count": 5}, // Incremental counts
1283 - {"id": "WARN", "count": 12}
1284 - ]
1285 - }
1286 - ]
1287 -}
1288 -```
1289 -
1290 -**Histogram Delta:**
1291 -```json
1292 -{
1293 - "histogramDelta": {
1294 - "chart": {
1295 - "result": {
1296 - "labels": ["time", "ERROR", "WARN"],
1297 - "data": [
1298 - [1697644320, 3, 8] // New data points only
1299 - ]
1300 - }
1301 - }
1302 - }
1303 -}
1304 -```
1305 -
1306 -### Data Merging
1307 -
1308 -Frontend merges delta responses with existing data:
1309 -- **Facet counts**: Accumulated using `count = (existing || 0) + (delta || 0)`
1310 -- **Table data**: Appended/prepended based on `direction`
1311 -- **Histogram data**: New data points added to existing chart
1312 -
1313 -## Real-Time Updates (PLAY Mode)
1314 -
1315 -PLAY mode enables live data streaming with efficient polling and conditional updates.
1316 -
1317 -### PLAY Mode Detection
1318 -
1319 -- **PLAY Mode**: `after < 0` (relative time from now)
1320 -- **PAUSE Mode**: `after > 0` (absolute timestamp)
1321 -
1322 -### Parameter Coordination
1323 -
1324 -When `if_modified_since` is present, the system automatically includes:
1325 -
1326 -```json
1327 -{
1328 - "if_modified_since": 1697644320000000,
1329 - "direction": "forward",
1330 - "merge": true,
1331 - "tail": true,
1332 - "delta": true,
1333 - "data_only": true,
1334 - "anchor": "anchorAfter"
1335 -}
1336 -```
1337 -
1338 -### Error Handling
1339 -
1340 -**304 Not Modified**: Indicates no new data available
1341 -```json
1342 -{
1343 - "status": 304
1344 -}
1345 -```
1346 -
1347 -Frontend handles 304 responses gracefully without showing errors to users.
1348 -
1349 -### Polling Behavior
1350 -
1351 -- **Polling Interval**: Based on function's `update_every` value
1352 -- **Auto-Pause**: When window loses focus or user hovers over data
1353 -- **Conditional Requests**: Uses `if_modified_since` to avoid unnecessary data transfers
1354 -
1355 -### Required vs Optional Fields
1356 -
1357 -**Minimum Required Fields:**
1358 -```json
1359 -{
1360 - "status": 200, // Required
1361 - "type": "table", // Required
1362 - "columns": {}, // Required (can be empty)
1363 - "data": [] // Required (can be empty)
1364 -}
1365 -```
1366 -
1367 -**Common Optional Fields:**
1368 -- `has_history`: Default false
1369 -- `help`: Documentation text
1370 -- `update_every`: Default 1
1371 -- `expires`: Cache control
1372 -- `default_sort_column`: Initial sort
1373 -- All column options except `index`, `name`, `type`
1374 -
1375 -## Error Handling
1376 -
1377 -### Error Response Format
1378 -
1379 -When a function encounters an error, the backend returns a JSON object. The primary error generation function (`rrd_call_function_error`) produces the following minimal format:
1380 -
1381 -```json
1382 -{
1383 - "status": 400, // The HTTP status code (e.g., 400, 404, 500)
1384 - "error_message": "A descriptive error message" // A human-readable message explaining the error
1385 -}
1386 -```
1387 -
1388 -**Frontend Consumption and Interpretation:**
1389 -The frontend is designed to handle a more comprehensive error structure, allowing for richer error display and localization. When an error occurs, the frontend will attempt to extract information from the received error object using the following hierarchy:
1390 -
1391 -* **`status`**: The HTTP status code, used for general error classification (e.g., 400 for bad request, 404 for not found).
1392 -* **`error_message`**: The primary detailed message from the backend. This is the most consistently populated field by current backend functions.
1393 -* **`error`**: A short, machine-readable error identifier (e.g., "MissingParameter"). While not consistently generated by `rrd_call_function_error`, other parts of the system or future backend implementations might provide this.
1394 -* **`message`**: A user-friendly message. The frontend often maps `error_message` or an internal `errorMsgKey` to this for display.
1395 -* **`help`**: Optional additional guidance for resolving the error. This field is not currently generated by `rrd_call_function_error`.
1396 -
1397 -**Example of Frontend Interpretation (Conceptual):**
1398 -The frontend might internally map specific `error_message` strings to predefined `errorMsgKey` values to provide localized or more context-specific messages to the user. For instance, a backend `error_message` like "The 'time_range' parameter is required" might be mapped to an `errorMsgKey` of "ErrMissingTimeRange" in the frontend, which then displays a user-friendly message like "Please specify a time range for this function."
1399 -
1400 -Therefore, while the backend currently provides `status` and `error_message`, developers should be aware that the frontend's error handling is capable of utilizing the more detailed fields (`error`, `message`, `help`) if they are provided by the backend in the future or by other API endpoints.
1401 -
1402 -**Common Error Codes:**
1403 -
1404 -
1405 -### Common Error Codes
1406 -
1407 -| Code | Use Case | Example |
1408 -|------|----------|---------|
1409 -| 400 | Bad input | Missing/invalid parameters |
1410 -| 401 | Auth required | User not logged in |
1411 -| 403 | Forbidden | Insufficient permissions |
1412 -| 404 | Not found | No data matches query |
1413 -| 500 | Server error | Internal failures |
1414 -| 503 | Unavailable | Service overloaded |
1415 -
1416 -## Protocol Integration
1417 -
1418 -### PLUGINSD Commands
1419 -
1420 -Functions integrate with Netdata through the PLUGINSD protocol:
1421 -
1422 -**Registration:**
1423 -```
1424 -FUNCTION "function_name" timeout "help text" "tags" "http_access" priority version
1425 -```
1426 -
1427 -**Execution Flow:**
1428 -1. **Collector → Agent**: `FUNCTION` registers the function
1429 -2. **Agent → Collector**: `FUNCTION_CALL` with transaction ID
1430 -3. **Collector → Agent**: `FUNCTION_RESULT_BEGIN` status format expires
1431 -4. **Collector → Agent**: Response payload
1432 -5. **Collector → Agent**: `FUNCTION_RESULT_END`
1433 -
1434 -**With Payload:**
1435 -```
1436 -FUNCTION_PAYLOAD_BEGIN transaction timeout function access source content_type
1437 -<payload data>
1438 -FUNCTION_PAYLOAD_END
1439 -```
1440 -
1441 -**Cancellation:**
1442 -```
1443 -FUNCTION_CANCEL transaction_id
1444 -```
1445 -
1446 -**Progress Updates:**
1447 -```
1448 -FUNCTION_PROGRESS transaction_id done total
1449 -```
1450 -
1451 -### Streaming Protocol
1452 -
1453 -Functions support streaming for real-time updates:
1454 -
1455 -```c
1456 -// Transaction management
1457 -dictionary_set(parser->inflight.functions, transaction_str, &function_data);
1458 -
1459 -// Timeout handling
1460 -if (*pf->stop_monotonic_ut + RRDFUNCTIONS_TIMEOUT_EXTENSION_UT < now_ut) {
1461 - // Function timed out
1462 -}
1463 -
1464 -// Progress callback
1465 -if(stream_has_capability(s, STREAM_CAP_PROGRESS)) {
1466 - // Enable progress updates
1467 -}
1468 -```
1469 -
1470 -### Key Files
1471 -- `pluginsd_functions.c`: Function execution and management
1472 -- `stream-sender-execute.c`: Streaming function calls
1473 -- `plugins.d/README.md`: Protocol documentation
1474 -- `plugins.d/functions-table.md`: Table format specification
1475 -
1476 -## Migration Guide
1477 -
1478 -### Upgrading from GET to POST (v:3)
1479 -
1480 -1. Update info response to include `"v": 3`
1481 -2. Change field IDs from hashes to plain names
1482 -3. Test with JSON POST payloads
1483 -4. Remove hash generation code
1484 -5. Update documentation
1485 -
1486 -### Adding Log Explorer Features
1487 -
1488 -1. Set `has_history: true`
1489 -2. Implement facets with counts
1490 -3. Add timestamp column with microseconds
1491 -4. Support anchor-based pagination
1492 -5. Add histogram data
1493 -6. Implement full-text search
1494 -
1495 -## Status and Next Steps
1496 -
1497 -This document represents the complete v3 protocol specification. Key areas for future development:
1498 -
1499 -1. **Protocol Extensions**
1500 - - Streaming updates for real-time data
1501 - - Aggregation pipelines
1502 - - Custom visualization types
1503 -
1504 -2. **UI Enhancements**
1505 - - Additional field types (gauge, sparkline)
1506 - - Custom column renderers
1507 - - Advanced filtering options
1508 -
1509 -3. **Performance Optimizations**
1510 - - Server-side pagination for simple tables
1511 - - Incremental updates
1512 - - Result caching
1513 -
1514 ----
1515 -
1516 -## Appendix A: Validation Checklist and Progress
1517 -
1518 -*This section tracks the validation work completed and remaining tasks*
1519 -
1520 -### Format Discovery
1521 -- [x] Analyze `processes` function implementation in apps.plugin
1522 -- [x] Analyze `network-connections` function in network-viewer.plugin
1523 -- [x] Identify common patterns between implementations
1524 -- [x] Extract all enum definitions used in responses
1525 -- [x] Document each field type and possible values
1526 -- [x] Identify optional vs required fields
1527 -
1528 -### Enumeration Completeness
1529 -- [x] Find all enum definitions in netdata C code
1530 -- [x] Map enum values to their string representations
1531 -- [x] Document the purpose of each enum value
1532 -- [ ] Check for any conditional enum values
1533 -
1534 -### Function Coverage
1535 -- [x] Scan all collectors/plugins for function implementations
1536 -- [x] List all simple table functions (has_history=false)
1537 -- [x] Verify format consistency across all functions
1538 -- [x] Document any function-specific extensions
1539 -
1540 -### UI Mapping
1541 -- [x] Analyze cloud-frontend code for function rendering
1542 -- [x] Map each format field to UI component
1543 -- [x] Document how enum values affect UI behavior
1544 -- [x] Identify any frontend-specific transformations
1545 -
1546 -### Corner Cases
1547 -- [x] Empty result sets
1548 -- [x] Large data sets (pagination?)
1549 -- [x] Error responses
1550 -- [x] Null/missing values
1551 -- [x] Special characters in data
1552 -- [x] Numeric precision/formatting
1553 -- [x] Date/time formatting
1554 -- [x] Sorting capabilities
1555 -- [x] Filtering capabilities
1556 -
1557 -### Protocol Validation
1558 -- [x] Request format documentation
1559 -- [x] Response format documentation
1560 -- [x] Error handling patterns
1561 -- [x] Streaming protocol integration
1562 -
1563 -### Cross-Reference Checks
1564 -- [x] Compare documented format with actual implementations
1565 -- [x] Verify all functions conform to documented format
1566 -- [x] Check for undocumented features in UI
1567 -- [x] Validate against any existing documentation
1568 -
1569 ----
1570 -
1571 -## Appendix B: Investigation History
1572 -
1573 -### Analysis Log
1574 -
1575 -*This section documents the investigation process to avoid repeating work*
1576 -
1577 -#### Session 1 - Initial Setup and Analysis
1578 -- Created FUNCTIONS.md structure
1579 -- Established checklist for comprehensive validation
1580 -- Analyzed `processes` function in apps.plugin
1581 - - Location: `/netdata/src/collectors/apps.plugin/apps_functions.c`
1582 - - Registration: `apps_plugin.c:752`
1583 - - Uses standard table format with array-based data rows
1584 -- Analyzed `network-connections` function in network-viewer.plugin
1585 - - Location: `/netdata/src/collectors/network-viewer.plugin/network-viewer.c`
1586 - - Registration via PLUGINSD protocol
1587 - - Supports aggregated and detailed views
1588 -- Extracted all RRDF enum definitions from:
1589 - - `/netdata/src/libnetdata/buffer/functions_fields.h`
1590 - - `/netdata/src/libnetdata/buffer/functions_fields.c`
1591 -- Documented complete enum value mappings for field types, visualizations, transforms, etc.
1592 -
1593 -#### Session 2 - Protocol Integration and Corner Cases
1594 -- Investigated PLUGINSD protocol integration
1595 - - Found in `pluginsd_functions.c` and `stream-sender-execute.c`
1596 - - Commands: FUNCTION, FUNCTION_CALL, FUNCTION_RESULT_BEGIN/END
1597 - - Support for payloads, cancellation, and progress updates
1598 -- Analyzed corner case handling:
1599 - - Empty results: `{"status": 200, "type": "table", "columns": {}, "data": []}`
1600 - - Null values: Rendered as empty cells in UI
1601 - - Special characters: Proper JSON escaping throughout
1602 - - Numeric precision: Controlled by `decimal_points` field
1603 - - Date/time: Millisecond epochs for tables, microsecond for logs
1604 -- Identified required vs optional fields:
1605 - - Required: status, type, columns, data
1606 - - Optional: Everything else (has_history, help, etc.)
1607 -- Documented sorting and filtering capabilities:
1608 - - Sorting: Backend-controlled defaults, user-sortable columns
1609 - - Filtering: Multiselect (default), range (numeric), facet (logs)
1610 - - Large datasets: Virtual scrolling for tables, pagination for logs
1611 -
1612 -### Key Findings
1613 -1. **Data Format**: Simple tables use array-of-arrays for data rows
1614 -2. **Column Definition**: Each column has extensive metadata controlling display and behavior
1615 -3. **Common Functions**: Both implement `buffer_rrdf_table_add_field()` for column definitions
1616 -4. **Response Builder**: Uses `buffer_json_*` functions to build JSON responses
1617 -5. **Field Options**: Bit flags allow combining multiple options per field
1618 -6. **Protocol Evolution**: GET with hashes → POST with plain field names (v:3)
1619 -7. **UI Mapping**: Comprehensive component mapping based on type/visualization
1620 -8. **Error Handling**: Standardized error response format with HTTP codes
1621 -9. **Performance**: Client-side operations for tables, server-side for logs
1622 -10. **Extensibility**: Optional fields allow function-specific features
1623 -
1624 -### Format Compliance Summary
1625 -- **All 12 simple table functions are fully compliant** with the documented format
1626 -- **Common extensions** found:
1627 - - `rowOptions` field for row severity/status (5/12 functions)
1628 - - `charts` and `default_charts` definitions (most functions)
1629 - - `group_by` aggregation options (most functions)
1630 - - `accepted_params` and `required_params` (metrics-cardinality)
1631 - - `default_sort_column` for initial sorting
1632 -- **No breaking deviations** - all extensions are additive and optional
1633 -
1634 ----
1635 -
1636 -## Appendix C: Log Explorer UI Features Detail
1637 -
1638 -*Detailed UI behaviors when has_history=true*
1639 -
1640 -### UI Features Enabled
1641 -
1642 -When properly formatted, the log explorer UI provides:
1643 -
1644 -1. **Sidebar with Faceted Filters**
1645 - - Shows facets with real-time counts
1646 - - Multi-select filtering
1647 - - Collapsible sections
1648 - - Search within facets
1649 -
1650 -2. **Time-based Histogram**
1651 - - Visual log distribution over time
1652 - - Click and drag to select time ranges
1653 - - Switch between different histogram fields
1654 - - Auto-updates with filters
1655 -
1656 -3. **Advanced Table Features**
1657 - - Infinite scroll using anchor navigation
1658 - - No manual pagination controls
1659 - - Automatic row coloring by severity
1660 - - Full-width message display
1661 - - Column pinning and resizing
1662 -
1663 -4. **Search and Navigation**
1664 - - Full-text search box
1665 - - Bi-directional navigation (forward/backward)
1666 - - Jump to specific time
1667 - - Export filtered results
1668 -
1669 -5. **Live Features**
1670 - - Tail mode for real-time updates
1671 - - Auto-refresh based on `update_every`
1672 - - Delta updates for efficiency
1673 - - Notification of new entries
1674 -
1675 ----
1676 -
1677 -*Last Updated: Based on analysis of Netdata codebase and cloud-frontend implementation*
src/crates/netdata-plugin/docs/README.md deleted
-868
@@ -1,868 +0,0 @@
1 -# External plugins
2 -
3 -`plugins.d` is the Netdata internal plugin that collects metrics
4 -from external processes, thus allowing Netdata to use **external plugins**.
5 -
6 -## Provided External Plugins
7 -
8 -| plugin | language | O/S | description |
9 -|:------------------------------------------------------------------------------------------------------:|:--------:|:--------------:|:----------------------------------------------------------------------------------------------------------------------------------------|
10 -| [apps.plugin](/src/collectors/apps.plugin/README.md) | `C` | linux, freebsd | monitors the whole process tree on Linux and FreeBSD and breaks down system resource usage by **process**, **user** and **user group**. |
11 -| [charts.d.plugin](/src/collectors/charts.d.plugin/README.md) | `BASH` | all | a **plugin orchestrator** for data collection modules written in `BASH` v4+. |
12 -| [cups.plugin](/src/collectors/cups.plugin/README.md) | `C` | all | monitors **CUPS** |
13 -| [ebpf.plugin](/src/collectors/ebpf.plugin/README.md) | `C` | linux | monitors different metrics on environments using kernel internal functions. |
14 -| [go.d.plugin](/src/go/plugin/go.d/README.md) | `GO` | all | collects metrics from the system, applications, or third-party APIs. |
15 -| [ioping.plugin](/src/collectors/ioping.plugin/README.md) | `C` | all | measures disk latency. |
16 -| [freeipmi.plugin](/src/collectors/freeipmi.plugin/README.md) | `C` | linux | collects metrics from enterprise hardware sensors, on Linux servers. |
17 -| [nfacct.plugin](/src/collectors/nfacct.plugin/README.md) | `C` | linux | collects netfilter firewall, connection tracker and accounting metrics using `libmnl` and `libnetfilter_acct`. |
18 -| [xenstat.plugin](/src/collectors/xenstat.plugin/README.md) | `C` | linux | collects XenServer and XCP-ng metrics using `lxenstat`. |
19 -| [perf.plugin](/src/collectors/perf.plugin/README.md) | `C` | linux | collects CPU performance metrics using performance monitoring units (PMU). |
20 -| [python.d.plugin](/src/collectors/python.d.plugin/README.md) | `python` | all | a **plugin orchestrator** for data collection modules written in `python` v2 or v3 (both are supported). |
21 -| [slabinfo.plugin](/src/collectors/slabinfo.plugin/README.md) | `C` | linux | collects kernel internal cache objects (SLAB) metrics. |
22 -
23 -Plugin orchestrators may also be described as **modular plugins**. They are modular since they accept custom made modules to be included. Writing modules for these plugins is easier than accessing the native Netdata API directly. You will find modules already available for each orchestrator under the directory of the particular modular plugin (e.g. under python.d.plugin for the python orchestrator).
24 -Each of these modular plugins has each own methods for defining modules. Please check the examples and their documentation.
25 -
26 -## Motivation
27 -
28 -This plugin allows Netdata to use **external plugins** for data collection:
29 -
30 -1. external data collection plugins may be written in any computer language.
31 -
32 -2. external data collection plugins may use O/S capabilities or `setuid` to
33 - run with escalated privileges (compared to the `netdata` daemon).
34 - The communication between the external plugin and Netdata is unidirectional
35 - (from the plugin to Netdata), so that Netdata cannot manipulate an external
36 - plugin running with escalated privileges.
37 -
38 -## Operation
39 -
40 -Each of the external plugins is expected to run forever.
41 -Netdata will start it when it starts and stop it when it exits.
42 -
43 -If the external plugin exits or crashes, Netdata will log an error.
44 -If the external plugin exits or crashes without pushing metrics to Netdata, Netdata will not start it again.
45 -
46 -- Plugins that exit with any value other than zero, will be disabled. Plugins that exit with zero, will be restarted after some time.
47 -- Plugins may also be disabled by Netdata if they output things that Netdata does not understand.
48 -
49 -The `stdout` of external plugins is connected to Netdata to receive metrics,
50 -with the API defined below.
51 -
52 -The `stderr` of external plugins is connected to Netdata's `error.log`.
53 -
54 -Plugins can create any number of charts with any number of dimensions each. Each chart can have its own characteristics independently of the others generated by the same plugin. For example, one chart may have an update frequency of 1 second, another may have 5 seconds and a third may have 10 seconds.
55 -
56 -## Configuration
57 -
58 -Netdata will supply the environment variables `NETDATA_USER_CONFIG_DIR` (for user supplied) and `NETDATA_STOCK_CONFIG_DIR` (for Netdata supplied) configuration files to identify the directory where configuration files are stored. It is up to the plugin to read the configuration it needs.
59 -
60 -The `netdata.conf` section `[plugins]` section contains a list of all the plugins found at the system where Netdata runs, with a boolean setting to enable them or not.
61 -
62 -Example:
63 -
64 -```
65 -[plugins]
66 - # enable running new plugins = yes
67 - # check for new plugins every = 60
68 -
69 - # charts.d = yes
70 - # ioping = yes
71 - # python.d = yes
72 -```
73 -
74 -The setting `enable running new plugins` sets the default behavior for all external plugins. It can be
75 -overridden for distinct plugins by modifying the appropriate plugin value configuration to either `yes` or `no`.
76 -
77 -The setting `check for new plugins every` sets the interval between scans of the directory
78 -`/usr/libexec/netdata/plugins.d`. New plugins can be added any time, and Netdata will detect them in a timely manner.
79 -
80 -For each of the external plugins enabled, another `netdata.conf` section
81 -is created, in the form of `[plugin:NAME]`, where `NAME` is the name of the external plugin.
82 -This section allows controlling the update frequency of the plugin and provide
83 -additional command line arguments to it.
84 -
85 -For example, for `apps.plugin` the following section is available:
86 -
87 -```
88 -[plugin:apps]
89 - # update every = 1
90 - # command options =
91 -```
92 -
93 -- `update every` controls the granularity of the external plugin.
94 -- `command options` allows giving additional command line options to the plugin.
95 -
96 -Netdata will provide to the external plugins the environment variable `NETDATA_UPDATE_EVERY`, in seconds (the default is 1). This is the **minimum update frequency** for all charts. A plugin that is updating values more frequently than this, is just wasting resources.
97 -
98 -Netdata will call the plugin with just one command line parameter: the number of seconds the user requested this plugin to update its data (by default is also 1).
99 -
100 -Other than the above, the plugin configuration is up to the plugin.
101 -
102 -Keep in mind, that the user may use Netdata configuration to overwrite chart and dimension parameters. This is transparent to the plugin.
103 -
104 -### Autoconfiguration
105 -
106 -Plugins should attempt to autoconfigure themselves when possible.
107 -
108 -For example, if your plugin wants to monitor `squid`, you can search for it on port `3128` or `8080`. If any succeeds, you can proceed. If it fails you can output an error (on stderr) saying that you cannot find `squid` running and giving instructions about the plugin configuration. Then you can stop (exit with non-zero value), so that Netdata will not attempt to start the plugin again.
109 -
110 -## External Plugins API
111 -
112 -Any program that can print a few values to its standard output can become a Netdata external plugin.
113 -
114 -Netdata parses lines starting with:
115 -
116 -- `CHART` - create or update a chart
117 -- `DIMENSION` - add or update a dimension to the chart just created
118 -- `VARIABLE` - define a variable (to be used in health calculations)
119 -- `CLABEL` - add a label to a chart
120 -- `CLABEL_COMMIT` - commit added labels to the chart
121 -- `FUNCTION` - define a function that can be called later to execute it
122 -- `BEGIN` - initialize data collection for a chart
123 -- `SET` - set the value of a dimension for the initialized chart
124 -- `END` - complete data collection for the initialized chart
125 -- `FLUSH` - ignore the last collected values
126 -- `DISABLE` - disable this plugin
127 -- `FUNCTION` - define functions
128 -- `FUNCTION_PROGRESS` - report the progress of a function execution
129 -- `FUNCTION_RESULT_BEGIN` - to initiate the transmission of function results
130 -- `FUNCTION_RESULT_END` - to end the transmission of function result
131 -- `CONFIG` - to define dynamic configuration entities
132 -
133 -a single program can produce any number of charts with any number of dimensions each.
134 -
135 -Charts can be added any time (not just the beginning).
136 -
137 -Netdata may send the following commands to the plugin's `stdin`:
138 -
139 -- `FUNCTION` - to call a specific function, with all parameters inline
140 -- `FUNCTION_PAYLOAD` - to call a specific function, with a payload of parameters
141 -- `FUNCTION_PAYLOAD_END` - to end the payload of parameters
142 -- `FUNCTION_CANCEL` - to cancel a running function transaction - no response is required
143 -- `FUNCTION_PROGRESS` - to report that a user asked the progress of running function call - no response is required
144 -
145 -### Command line parameters
146 -
147 -The plugin **MUST** accept just **one** parameter: **the number of seconds it is
148 -expected to update the values for its charts**. The value passed by Netdata
149 -to the plugin is controlled via its configuration file (so there is no need
150 -for the plugin to handle this configuration option).
151 -
152 -The external plugin can overwrite the update frequency. For example, the server may
153 -request per second updates, but the plugin may ignore it and update its charts
154 -every 5 seconds.
155 -
156 -### Environment variables
157 -
158 -There are a few environment variables that are set by `netdata` and are
159 -available for the plugin to use.
160 -
161 -| variable | description |
162 -|:--------------------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
163 -| `NETDATA_USER_CONFIG_DIR` | The directory where all Netdata-related user configuration should be stored. If the plugin requires custom user configuration, this is the place the user has saved it (normally under `/etc/netdata`). |
164 -| `NETDATA_STOCK_CONFIG_DIR` | The directory where all Netdata -related stock configuration should be stored. If the plugin is shipped with configuration files, this is the place they can be found (normally under `/usr/lib/netdata/conf.d`). |
165 -| `NETDATA_PLUGINS_DIR` | The directory where all Netdata plugins are stored. |
166 -| `NETDATA_USER_PLUGINS_DIRS` | The list of directories where custom plugins are stored. |
167 -| `NETDATA_WEB_DIR` | The directory where the web files of Netdata are saved. |
168 -| `NETDATA_CACHE_DIR` | The directory where the cache files of Netdata are stored. Use this directory if the plugin requires a place to store data. A new directory should be created for the plugin for this purpose, inside this directory. |
169 -| `NETDATA_LOG_DIR` | The directory where the log files are stored. By default the `stderr` output of the plugin will be saved in the `error.log` file of Netdata. |
170 -| `NETDATA_HOST_PREFIX` | This is used in environments where system directories like `/sys` and `/proc` have to be accessed at a different path. |
171 -| `NETDATA_DEBUG_FLAGS` | This is a number (probably in hex starting with `0x`), that enables certain Netdata debugging features. Check **\[[Tracing Options]]** for more information. |
172 -| `NETDATA_UPDATE_EVERY` | The minimum number of seconds between chart refreshes. This is like the **internal clock** of Netdata (it is user configurable, defaulting to `1`). There is no meaning for a plugin to update its values more frequently than this number of seconds. |
173 -| `NETDATA_INVOCATION_ID` | A random UUID in compact form, representing the unique invocation identifier of Netdata. When running under systemd, Netdata uses the `INVOCATION_ID` set by systemd. |
174 -| `NETDATA_LOG_METHOD` | One of `syslog`, `journal`, `stderr` or `none`, indicating the preferred log method of external plugins. |
175 -| `NETDATA_LOG_FORMAT` | One of `journal`, `logfmt` or `json`, indicating the format of the logs. Plugins can use the Netdata `systemd-cat-native` command to log always in `journal` format, and have it automatically converted to the format expected by netdata. |
176 -| `NETDATA_LOG_LEVEL` | One of `emergency`, `alert`, `critical`, `error`, `warning`, `notice`, `info`, `debug`. Plugins are expected to log events with the given priority and the more important ones. |
177 -| `NETDATA_SYSLOG_FACILITY` | Set only when the `NETDATA_LOG_METHOD` is `syslog`. Possible values are `auth`, `authpriv`, `cron`, `daemon`, `ftp`, `kern`, `lpr`, `mail`, `news`, `syslog`, `user`, `uucp` and `local0` to `local7` |
178 -| `NETDATA_ERRORS_THROTTLE_PERIOD` | The log throttling period in seconds. |
179 -| `NETDATA_ERRORS_PER_PERIOD` | The allowed number of log events per period. |
180 -| `NETDATA_SYSTEMD_JOURNAL_PATH` | When `NETDATA_LOG_METHOD` is set to `journal`, this is the systemd-journald socket path to use. |
181 -
182 -### The output of the plugin
183 -
184 -The plugin should output instructions for Netdata to its output (`stdout`). Since this uses pipes, please make sure you flush stdout after every iteration.
185 -
186 -#### DISABLE
187 -
188 -`DISABLE` will disable this plugin. This will prevent Netdata from restarting the plugin. You can also exit with the value `1` to have the same effect.
189 -
190 -#### HOST_DEFINE
191 -
192 -`HOST_DEFINE` defines a new (or updates an existing) virtual host.
193 -
194 -The template is:
195 -
196 -> HOST_DEFINE machine_guid hostname
197 -
198 -where:
199 -
200 -- `machine_guid`
201 -
202 - uniquely identifies the host, this is what will be needed to add charts to the host.
203 -
204 -- `hostname`
205 -
206 - is the hostname of the virtual host
207 -
208 -#### HOST_LABEL
209 -
210 -`HOST_LABEL` adds a key-value pair to the virtual host labels. It has to be given between `HOST_DEFINE` and `HOST_DEFINE_END`.
211 -
212 -The template is:
213 -
214 -> HOST_LABEL key value
215 -
216 -where:
217 -
218 -- `key`
219 -
220 - uniquely identifies the key of the label
221 -
222 -- `value`
223 -
224 - is the value associated with this key
225 -
226 -There are a few special keys that are used to define the system information of the monitored system:
227 -
228 -- `_cloud_provider_type`
229 -- `_cloud_instance_type`
230 -- `_cloud_instance_region`
231 -- `_os_name`
232 -- `_os_version`
233 -- `_kernel_version`
234 -- `_system_cores`
235 -- `_system_cpu_freq`
236 -- `_system_ram_total`
237 -- `_system_disk_space`
238 -- `_architecture`
239 -- `_virtualization`
240 -- `_container`
241 -- `_container_detection`
242 -- `_virt_detection`
243 -- `_is_k8s_node`
244 -- `_install_type`
245 -- `_prebuilt_arch`
246 -- `_prebuilt_dist`
247 -
248 -#### HOST_DEFINE_END
249 -
250 -`HOST_DEFINE_END` commits the host information, creating a new host entity, or updating an existing one with the same `machine_guid`.
251 -
252 -#### HOST
253 -
254 -`HOST` switches data collection between hosts.
255 -
256 -The template is:
257 -
258 -> HOST machine_guid
259 -
260 -where:
261 -
262 -- `machine_guid`
263 -
264 - is the UUID of the host to switch to. After this command, every other command following it is assumed to be associated with this host.
265 - Setting machine_guid to `localhost` switches data collection to the local host.
266 -
267 -#### CHART
268 -
269 -`CHART` defines a new chart.
270 -
271 -the template is:
272 -
273 -> CHART type.id name title units \[family \[context \[charttype \[priority \[update_every \[options \[plugin [module]]]]]]]]
274 -
275 - where:
276 -
277 -- `type.id`
278 -
279 - uniquely identifies the chart,
280 - this is what will be needed to add values to the chart
281 -
282 - the `type` part controls the menu the charts will appear in
283 -
284 -- `name`
285 -
286 - is the name that will be presented to the user instead of `id` in `type.id`. This means that only the `id` part of
287 - `type.id` is changed. When a name has been given, the chart is indexed (and can be referred) as both `type.id` and
288 - `type.name`. You can set name to `''`, or `null`, or `(null)` to disable it. If a chart with the same name already
289 - exists, a serial number is automatically attached to the name to avoid naming collisions.
290 -
291 -- `title`
292 -
293 - the text above the chart
294 -
295 -- `units`
296 -
297 - the label of the vertical axis of the chart,
298 - all dimensions added to a chart should have the same units
299 - of measurement
300 -
301 -- `family`
302 -
303 - is used to group charts together
304 - (for example all eth0 charts should say: eth0),
305 - if empty or missing, the `id` part of `type.id` will be used
306 -
307 - this controls the sub-menu on the dashboard
308 -
309 -- `context`
310 -
311 - the context is giving the template of the chart. For example, if multiple charts present the same information for a different family, they should have the same `context`
312 -
313 - this is used for looking up rendering information for the chart (colors, sizes, informational texts) and also apply alerts to it
314 -
315 -- `charttype`
316 -
317 - one of `line`, `area`, `stacked` or `heatmap`,
318 - if empty or missing, the `line` will be used
319 -
320 -- `priority`
321 -
322 - is the relative priority of the charts as rendered on the web page,
323 - lower numbers make the charts appear before the ones with higher numbers,
324 - if empty or missing, `1000` will be used
325 -
326 -- `update_every`
327 -
328 - overwrite the update frequency set by the server,
329 - if empty or missing, the user configured value will be used
330 -
331 -- `options`
332 -
333 - a space separated list of options, enclosed in quotes. The following options are currently supported: `obsolete` to mark a chart as obsolete (Netdata will hide it and delete it after some time), `store_first` to make Netdata store the first collected value, assuming there was an invisible previous value set to zero (this is used by statsd charts - if the first data collected value of incremental dimensions is not zero based, unrealistic spikes will appear with this option set) and `hidden` to perform all operations on a chart, but do not offer it on dashboards (the chart will be send to external databases). `CHART` options have been added in Netdata v1.7 and the `hidden` option was added in 1.10.
334 -
335 -- `plugin` and `module`
336 -
337 - both are just names that are used to let the user identify the plugin and the module that generated the chart. If `plugin` is unset or empty, Netdata will automatically set the filename of the plugin that generated the chart. `module` has not default.
338 -
339 -#### DIMENSION
340 -
341 -`DIMENSION` defines a new dimension for the chart
342 -
343 -the template is:
344 -
345 -> DIMENSION id \[name \[algorithm \[multiplier \[divisor [options]]]]]
346 -
347 - where:
348 -
349 -- `id`
350 -
351 - the `id` of this dimension (it is a text value, not numeric),
352 - this will be needed later to add values to the dimension
353 -
354 - We suggest to avoid using `.` in dimension ids. External databases expect metrics to be `.` separated and people will get confused if a dimension id contains a dot.
355 -
356 -- `name`
357 -
358 - the name of the dimension as it will appear at the legend of the chart,
359 - if empty or missing the `id` will be used
360 -
361 -- `algorithm`
362 -
363 - one of:
364 -
365 - - `absolute`
366 -
367 - the value is to drawn as-is (interpolated to second boundary),
368 - if `algorithm` is empty, invalid or missing, `absolute` is used
369 -
370 - - `incremental`
371 -
372 - the value increases over time,
373 - the difference from the last value is presented in the chart,
374 - the server interpolates the value and calculates a per second figure
375 -
376 - - `percentage-of-absolute-row`
377 -
378 - the % of this value compared to the total of all dimensions
379 -
380 - - `percentage-of-incremental-row`
381 -
382 - the % of this value compared to the incremental total of
383 - all dimensions
384 -
385 -- `multiplier`
386 -
387 - an integer value to multiply the collected value,
388 - if empty or missing, `1` is used
389 -
390 -- `divisor`
391 -
392 - an integer value to divide the collected value,
393 - if empty or missing, `1` is used
394 -
395 -- `options`
396 -
397 - a space separated list of options, enclosed in quotes. Options supported: `obsolete` to mark a dimension as obsolete (Netdata will delete it after some time) and `hidden` to make this dimension hidden, it will take part in the calculations but will not be presented in the chart.
398 -
399 -#### VARIABLE
400 -
401 -> VARIABLE [SCOPE] name = value
402 -
403 -`VARIABLE` defines a variable that can be used in alerts. This is to used for setting constants (like the max connections a server may accept).
404 -
405 -Variables support 2 scopes:
406 -
407 -- `GLOBAL` or `HOST` to define the variable at the host level.
408 -- `LOCAL` or `CHART` to define the variable at the chart level. Use chart-local variables when the same variable may exist for different charts (i.e. Netdata monitors 2 mysql servers, and you need to set the `max_connections` each server accepts). Using chart-local variables is the ideal to build alert templates.
409 -
410 -The position of the `VARIABLE` line, sets its default scope (in case you do not specify a scope). So, defining a `VARIABLE` before any `CHART`, or between `END` and `BEGIN` (outside any chart), sets `GLOBAL` scope, while defining a `VARIABLE` just after a `CHART` or a `DIMENSION`, or within the `BEGIN` - `END` block of a chart, sets `LOCAL` scope.
411 -
412 -These variables can be set and updated at any point.
413 -
414 -Variable names should use alphanumeric characters, the `.` and the `_`.
415 -
416 -The `value` is floating point (Netdata used `long double`).
417 -
418 -Variables are transferred to upstream Netdata servers (streaming and database replication).
419 -
420 -#### CLABEL
421 -
422 -> CLABEL name value source
423 -
424 -`CLABEL` defines a label used to organize and identify a chart.
425 -
426 -Name and value accept characters according to the following table:
427 -
428 -| Character | Symbol | Label Name | Label Value |
429 -|---------------------|:------:|:----------:|:-----------:|
430 -| UTF-8 character | UTF-8 | _ | keep |
431 -| Lower case letter | [a-z] | keep | keep |
432 -| Upper case letter | [A-Z] | keep | [a-z] |
433 -| Digit | [0-9] | keep | keep |
434 -| Underscore | _ | keep | keep |
435 -| Minus | - | keep | keep |
436 -| Plus | + | _ | keep |
437 -| Colon | : | _ | keep |
438 -| Semicolon | ; | _ | : |
439 -| Equal | = | _ | : |
440 -| Period | . | keep | keep |
441 -| Comma | , | . | . |
442 -| Slash | / | keep | keep |
443 -| Backslash | \ | / | / |
444 -| At | @ | _ | keep |
445 -| Space | ' ' | _ | keep |
446 -| Opening parenthesis | ( | _ | keep |
447 -| Closing parenthesis | ) | _ | keep |
448 -| Anything else | | _ | _ |
449 -
450 -The `source` is an integer field that can have the following values:
451 -- `1`: The value was set automatically.
452 -- `2`: The value was set manually.
453 -- `4`: This is a K8 label.
454 -- `8`: This is a label defined using `netdata` Agent-Cloud link.
455 -
456 -#### CLABEL_COMMIT
457 -
458 -`CLABEL_COMMIT` indicates that all labels were defined and the chart can be updated.
459 -
460 -#### FUNCTION
461 -
462 -The plugin can register functions to Netdata, like this:
463 -
464 -> FUNCTION [GLOBAL] "name and parameters of the function" timeout "help string for users" "tags" "access" priority version
465 -
466 -- Tags currently recognized are either `top` or `logs` (or both, space separated).
467 -- Access is one of `any`, `member`, or `admin`:
468 - - `any` to offer the function to all users of Netdata, even if they are not authenticated.
469 - - `member` to offer the function to all authenticated members of Netdata.
470 - - `admin` to offer the function only to authenticated administrators.
471 -- Priority defines the position of the function relative to the other functions (default is 100).
472 -- Version defines the version of the function (default is 0).
473 -
474 -Users can use a function to ask for more information from the collector. Netdata maintains a registry of functions in 2 levels:
475 -
476 -- per node
477 -- per chart
478 -
479 -Both node and chart functions are exactly the same, but chart functions allow Netdata to relate functions with charts and therefore present a context-sensitive menu of functions related to the chart the user is using.
480 -
481 -Users can get a list of all the registered functions using the `/api/v1/functions` endpoint of Netdata and call functions using the `/api/v1/function` API call of Netdata.
482 -
483 -Once a function is called, the plugin will receive at its standard input a command that looks like this:
484 -
485 -```
486 -FUNCTION transaction_id timeout "name and parameters of the function as one quoted parameter" "user permissions value" "source of request"
487 -```
488 -
489 -When the function to be called is to receive a payload of parameters, the call looks like this:
490 -
491 -```
492 -FUNCTION_PAYLOAD transaction_id timeout "name and parameters of the function as one quoted parameter" "user permissions value" "source of request" "content/type"
493 -body of the payload, formatted according to content/type
494 -FUNCTION PAYLOAD END
495 -```
496 -
497 -In this case, Netdata will send:
498 -
499 -- A line starting with `FUNCTION_PAYLOAD` together with the required metadata for the function, like the transaction id, the function name and its parameters, the timeout and the content type. This line ends with a newline.
500 -- Then, the payload itself (which may or may not have newlines in it). The payload should be parsed according to the content type parameter.
501 -- Finally, a line starting with `FUNCTION_PAYLOAD_END`, so it is expected like `\nFUNCTION_PAYLOAD_END\n`.
502 -
503 -Note 1: The plugins.d protocol allows parameters without single or double quotes if they don't contain spaces. However, the plugin should be able to parse parameters even if they are enclosed in single or double quotes. If the first character of a parameter is a single quote, its last character should also be a single quote too, and similarly for double quotes.
504 -
505 -Note 2: Netdata always sends the function and its parameters enclosed in double quotes. If the function command and its parameters contain quotes, they are converted to single quotes.
506 -
507 -The plugin is expected to parse and validate `name and parameters of the function as one quotes parameter`. Netdata allows the user interface to manipulate this string by appending more parameters.
508 -
509 -If the plugin rejects the request, it should respond with this:
510 -
511 -```
512 -FUNCTION_RESULT_BEGIN transaction_id 400 application/json
513 -{
514 - "status": 400,
515 - "error_message": "description of the rejection reasons"
516 -}
517 -FUNCTION_RESULT_END
518 -```
519 -
520 -If the plugin prepares a response, it should send (via its standard output, together with the collected data, but not interleaved with them):
521 -
522 -```
523 -FUNCTION_RESULT_BEGIN transaction_id http_response_code content_type expiration
524 -```
525 -
526 -Where:
527 -
528 - - `transaction_id` is the transaction id that Netdata sent for this function execution
529 - - `http_response_code` is the http error code Netdata should respond with, 200 is the "ok" response
530 - - `content_type` is the content type of the response
531 - - `expiration` is the absolute timestamp (number, unix epoch) this response expires
532 -
533 -Immediately after this, all text is assumed to be the response content.
534 -The content is text and line oriented. The maximum line length accepted is 15kb. Longer lines will be truncated.
535 -The type of the context itself depends on the plugin and the UI.
536 -
537 -To terminate the message, Netdata seeks a line with just this:
538 -
539 -```
540 -FUNCTION_RESULT_END
541 -```
542 -
543 -This defines the end of the message. `FUNCTION_RESULT_END` should appear in a line alone, without any other text, so it is wise to add `\n` before and after it.
544 -
545 -After this line, Netdata resumes processing collected metrics from the plugin.
546 -
547 -The maximum uncompressed payload size Netdata will accept is 100MB.
548 -
549 -##### Functions cancellation
550 -
551 -Netdata is able to detect when a user made an API request, but abandoned it before it was completed. If this happens to an API called for a function served by the plugin, Netdata will generate a `FUNCTION_CANCEL` request to let the plugin know that it can stop processing the query.
552 -
553 -After receiving such a command, the plugin **must still send a response for the original function request**, to wake up any waiting threads before they timeout. The http response code is not important, since the response will be discarded, however for auditing reasons we suggest to send back a 499 http response code. This is not a standard response code according to the HTTP protocol, but web servers like `nginx` are using it to indicate that a request was abandoned by a user.
554 -
555 -##### Functions progress
556 -
557 -When a request takes too long to be processed, Netdata allows the plugin to report progress to Netdata, which in turn will report progress to the caller.
558 -
559 -The plugin can send `FUNCTION_PROGRESS` like this:
560 -
561 -```
562 -FUNCTION_PROGRESS transaction_id done all
563 -```
564 -
565 -Where:
566 -
567 -- `transaction_id` is the transaction id of the function request
568 -- `done` is an integer value indicating the amount of work done
569 -- `all` is an integer value indicating the total amount of work to be done
570 -
571 -Netdata supports two kinds of progress:
572 -- progress as a percentage, which is calculated as `done * 100 / all`
573 -- progress without knowing the total amount of work to be done, which is enabled when the plugin reports `all` as zero.
574 -
575 -##### Functions timeout
576 -
577 -All functions calls specify a timeout, at which all the intermediate routing nodes (parents, web server threads) will time out and abort the call.
578 -
579 -However, all intermediate routing nodes are configured to extend the timeout when the caller asks for progress. This works like this:
580 -
581 -When a progress request is received, if the expected timeout of the request is less than or equal to 10 seconds, the expected timeout is extended by 10 seconds.
582 -
583 -Usually, the user interface asks for a progress every second. So, during the last 10 seconds of the timeout, every progress request made shifts the timeout 10 seconds to the future.
584 -
585 -To accomplish this, when Netdata receives a progress request by a user, it generates progress requests to the plugin, updating all the intermediate nodes to extend their timeout if necessary.
586 -
587 -The plugin will receive progress requests like this:
588 -
589 -```
590 -FUNCTION_PROGRESS transaction_id
591 -```
592 -
593 -There is no need to respond to this command. It is only there to let the plugin know that a user is still waiting for the query to finish.
594 -
595 -#### CONFIG
596 -
597 -`CONFIG` commands sent from the plugin to Netdata define dynamic configuration entities. These configurable entities are exposed to the user interface, allowing users to change configuration at runtime.
598 -
599 -Dynamically configurations made this way are saved to disk by Netdata and are replayed automatically when Netdata or the plugin restarts.
600 -
601 -`CONFIG` commands look like this:
602 -
603 -```
604 -CONFIG id action ...
605 -```
606 -
607 -Where:
608 -
609 -- `id` is a unique identifier for the configurable entity. This should by design be unique across Netdata. It should be something like `plugin:module:jobs`, e.g. `go.d:postgresql:jobs:masterdb`. This is assumed to be colon-separated with the last part (`masterdb` in our example), being the one displayed to users when there ano conflicts under the same configuration path.
610 -- `action` can be:
611 - - `create`, to declare the dynamic configuration entity
612 - - `delete`, to delete the dynamic configuration entity - this does not delete user configuration, we if an entity with the same id is created in the future, the saved configuration will be given to it.
613 - - `status`, to update the dynamic configuration entity status
614 -
615 -> IMPORTANT:<br/>
616 -> The plugin should blindly create, delete and update the status of its dynamic configuration entities, without any special logic applied to it. Netdata needs to be updated of what is actually happening at the plugin. Keep in mind that creating dynamic configuration entities triggers responses from Netdata, depending on its type and status. Re-creating a job, triggers the same responses every time, so make sure you create jobs only when you add jobs.
617 -
618 -When the `action` is `create`, the following additional parameters are expected:
619 -
620 -```
621 -CONFIG id action status type "path" source_type "source" "supported commands" "view permissions" "edit permissions"
622 -```
623 -
624 -Where:
625 -
626 -- `action` should be `create`
627 -- `status` can be:
628 - - `accepted`, the plugin accepted the configuration, but it is not running yet.
629 - - `running`, the plugin accepted and runs the configuration.
630 - - `failed`, the plugin tries to run the configuration but it fails.
631 - - `incomplete`, the plugin needs additional settings to run this configuration. This is usually used for the cases the plugin discovered a job, but important information is missing for it to work.
632 - - `disabled`, the configuration has been disabled by a user.
633 - - `orphan`, the configuration is not claimed by any plugin. This is used internally by Netdata to mark the configuration nodes available, for which there is no plugin related to them. Do not use in plugins directly.
634 -- `type` can be `single`, `template` or `job`:
635 - - `single` is used when the configurable entity is fixed and users should never be able to add or delete it.
636 - - `template` is used to define a template based on which users can add multiple configurations, like adding data collection jobs. So, the plugin defines the template of the jobs and users are presented with a `[+]` button to add such configuration jobs. The plugin can define multiple templates by giving different `id`s to them.
637 - - `job` is used to define a job of a template. The plugin should always add all its jobs, independently of the way they have been discovered. It is important to note the relation between `template` and `job` when it comes it the `id`: The `id` of the template should be the prefix of the `job`'s `id`. For example, if the template is `go.d:postgresql:jobs`, then all its jobs be like `go.d:postgresql:jobs:jobname`.
638 -- `path` is the absolute path of the configurable entity inside the tree of Netdata configurations. Usually, this is should be `/collectors`.
639 -- `source` can be `internal`, `stock`, `user`, `discovered` or `dyncfg`:
640 - - `internal` is used for configurations that are based on internal code settings
641 - - `stock` is used for default configurations
642 - - `discovered` is used for dynamic configurations the plugin discovers by its own
643 - - `user` is used for user configurations, usually via a configuration file
644 - - `dyncfg` is used for configuration received via this dynamic configuration mechanism
645 -- `source` should provide more details about the exact source of the configuration, like `line@file`, or `user@ip`, etc.
646 -- `supported_commands` is a space separated list of the following keywords, enclosed in single or double quotes. These commands are used by the user interface to determine the actions the users can take:
647 - - `schema`, to expose the JSON schema for the user interface. This is mandatory for all configurable entities. When `schema` requests are received, Netdata will first attempt to load the schema from `/etc/netdata/schema.d/` and `/var/lib/netdata/conf.d/schema.d`. For jobs, it will serve the schema of their template. If no schema is found for the required `id`, the `schema` request will be forwarded to the plugin, which is expected to send back the relevant schema.
648 - - `get`, to expose the current configuration values, according the schema defined. `templates` cannot support `get`, since they don't maintain any data.
649 - - `update`, to receive configuration updates for this entity. `templates` cannot support `update`, since they don't maintain any data.
650 - - `test`, like `update` but only test the configuration and report success or failure.
651 - - `add`, to receive job creation commands for templates. Only `templates` should support this command.
652 - - `remove`, to remove a configuration. Only `jobs` should support this command.
653 - - `enable` and `disable`, to receive user requests to enable and disable this entity. Adding only one of `enable` or `disable` to the supported commands, Netdata will add both of them. The plugin should expose these commands on `templates` only when it wants to receive `enable` and `disable` commands for all the `jobs` of this `template`.
654 - - `restart`, to restart a job.
655 -- `view permissions` and `edit permissions` are bitmaps of the Netdata permission system to control access to the configuration. If set to zero, Netdata will require a signed in user with view and edit permissions to the Netdata's configuration system.
656 -
657 -The plugin receives commands as if it had exposed a `FUNCTION` named `config`. Netdata formats all these calls like this:
658 -
659 -```
660 -config id command
661 -```
662 -
663 -Where `id` is the unique id of the configurable entity and `command` is one of the supported commands the plugin sent to Netdata.
664 -
665 -The plugin will receive (for commands: `schema`, `get`, `remove`, `enable`, `disable` and `restart`):
666 -
667 -```
668 -FUNCTION transaction_id timeout "config id command" "user permissions value" "source string"
669 -```
670 -
671 -or (for commands: `update`, `add` and `test`):
672 -
673 -```
674 -FUNCTION_PAYLOAD transaction_id timeout "config id command" "user permissions value" "source string" "content/type"
675 -body of the payload formatted according to content/type
676 -FUNCTION_PAYLOAD_END
677 -```
678 -
679 -Once received, the plugin should process it and respond accordingly.
680 -
681 -Immediately after the plugin adds a configuration entity, if the commands `enable` and `disable` are supported by it, Netdata will send either `enable` or `disable` for it, based on the last user action, which has been persisted to disk.
682 -
683 -Plugin responses follow the same format `FUNCTIONS` do:
684 -
685 -```
686 -FUNCTION_RESULT_BEGIN transaction_id http_response_code content/type expiration
687 -body of the response formatted according to content/type
688 -FUNCTION_RESULT_END
689 -```
690 -
691 -Successful responses (HTTP response code 200) to `schema` and `get` should send back the relevant JSON object.
692 -All other responses should have the following response body:
693 -
694 -```json
695 -{
696 - "status" : 404,
697 - "message" : "some text"
698 -}
699 -```
700 -
701 -The user interface presents the message to users, even when the response is successful (HTTP code 200).
702 -
703 -When responding to additions and updates, Netdata uses the following success response codes to derive additional information:
704 -
705 -- `200`, responding with 200, means the configuration has been accepted and it is running.
706 -- `202`, responding with 202, means the configuration has been accepted but it is not yet running. A subsequent `status` action will update it.
707 -- `298`, responding with 298, means the configuration has been accepted but it is disabled for some reason (probably because it matches nothing or the contents are not useful - use the `message` to provide additional information).
708 -- `299`, responding with 299, means the configuration has been accepted but a restart is required to apply it.
709 -
710 -## Data collection
711 -
712 -data collection is defined as a series of `BEGIN` -> `SET` -> `END` lines
713 -
714 -> BEGIN type.id [microseconds]
715 -
716 -- `type.id`
717 -
718 - is the unique identification of the chart (as given in `CHART`)
719 -
720 -- `microseconds`
721 -
722 - is the number of microseconds since the last update of the chart. It is optional.
723 -
724 - Under heavy system load, the system may have some latency transferring
725 - data from the plugins to Netdata via the pipe. This number improves
726 - accuracy significantly, since the plugin is able to calculate the
727 - duration between its iterations better than Netdata.
728 -
729 - The first time the plugin is started, no microseconds should be given
730 - to Netdata.
731 -
732 -> SET id = value
733 -
734 -- `id`
735 -
736 - is the unique identification of the dimension (of the chart just began)
737 -
738 -- `value`
739 -
740 - is the collected value, only integer values are collected. If you want to push fractional values, multiply this value by 100 or 1000 and set the `DIMENSION` divider to 1000.
741 -
742 -> END
743 -
744 - END does not take any parameters, it commits the collected values for all dimensions to the chart. If a dimensions was not `SET`, its value will be empty for this commit.
745 -
746 -More `SET` lines may appear to update all the dimensions of the chart.
747 -All of them in one `BEGIN` -> `END` block.
748 -
749 -All `SET` lines within a single `BEGIN` -> `END` block have to refer to the
750 -same chart.
751 -
752 -If more charts need to be updated, each chart should have its own
753 -`BEGIN` -> `SET` -> `END` block.
754 -
755 -If, for any reason, a plugin has issued a `BEGIN` but wants to cancel it,
756 -it can issue a `FLUSH`. The `FLUSH` command will instruct Netdata to ignore
757 -all the values collected since the last `BEGIN` command.
758 -
759 -If a plugin does not behave properly (outputs invalid lines, or does not
760 -follow these guidelines), will be disabled by Netdata.
761 -
762 -### collected values
763 -
764 -Netdata will collect any **signed** value in the 64bit range:
765 -`-9.223.372.036.854.775.808` to `+9.223.372.036.854.775.807`
766 -
767 -If a value is not collected, leave it empty, like this:
768 -
769 -`SET id =`
770 -
771 -or do not output the line at all.
772 -
773 -## Modular Plugins
774 -
775 -1. **python**, use `python.d.plugin`, there are many examples in the [python.d
776 - directory](/src/collectors/python.d.plugin/README.md)
777 -
778 - python is ideal for Netdata plugins. It is a simple, yet powerful way to collect data, it has a very small memory footprint, although it is not the most CPU efficient way to do it.
779 -
780 -2. **BASH**, use `charts.d.plugin`, there are many examples in the [charts.d
781 - directory](/src/collectors/charts.d.plugin/README.md)
782 -
783 - BASH is the simplest scripting language for collecting values. It is the less efficient though in terms of CPU resources. You can use it to collect data quickly, but extensive use of it might use a lot of system resources.
784 -
785 -3. **C**
786 -
787 - Of course, C is the most efficient way of collecting data. This is why Netdata itself is written in C.
788 -
789 -## Writing Plugins Properly
790 -
791 -There are a few rules for writing plugins properly:
792 -
793 -1. Respect system resources
794 -
795 - Pay special attention to efficiency:
796 -
797 - - Initialize everything once, at the beginning. Initialization is not an expensive operation. Your plugin will most probably be started once and run forever. So, do whatever heavy operation is needed at the beginning, just once.
798 - - Do the absolutely minimum while iterating to collect values repeatedly.
799 - - If you need to connect to another server to collect values, avoid re-connects if possible. Connect just once, with keep-alive (for HTTP) enabled and collect values using the same connection.
800 - - Avoid any CPU or memory heavy operation while collecting data. If you control memory allocation, avoid any memory allocation while iterating to collect values.
801 - - Avoid running external commands when possible. If you are writing shell scripts avoid especially pipes (each pipe is another fork, a very expensive operation).
802 -
803 -2. The best way to iterate at a constant pace is this pseudo code:
804 -
805 -```js
806 - var update_every = argv[1] * 1000; /* seconds * 1000 = milliseconds */
807 -
808 - readConfiguration();
809 -
810 - if(!verifyWeCanCollectValues()) {
811 - print("DISABLE");
812 - exit(1);
813 - }
814 -
815 - createCharts(); /* print CHART and DIMENSION statements */
816 -
817 - var loops = 0;
818 - var last_run = 0;
819 - var next_run = 0;
820 - var dt_since_last_run = 0;
821 - var now = 0;
822 -
823 - while(true) {
824 - /* find the current time in milliseconds */
825 - now = currentTimeStampInMilliseconds();
826 -
827 - /*
828 - * find the time of the next loop
829 - * this makes sure we are always aligned
830 - * with the Netdata daemon
831 - */
832 - next_run = now - (now % update_every) + update_every;
833 -
834 - /*
835 - * wait until it is time
836 - * it is important to do it in a loop
837 - * since many wait functions can be interrupted
838 - */
839 - while( now < next_run ) {
840 - sleepMilliseconds(next_run - now);
841 - now = currentTimeStampInMilliseconds();
842 - }
843 -
844 - /* calculate the time passed since the last run */
845 - if ( loops > 0 )
846 - dt_since_last_run = (now - last_run) * 1000; /* in microseconds */
847 -
848 - /* prepare for the next loop */
849 - last_run = now;
850 - loops++;
851 -
852 - /* do your magic here to collect values */
853 - collectValues();
854 -
855 - /* send the collected data to Netdata */
856 - printValues(dt_since_last_run); /* print BEGIN, SET, END statements */
857 - }
858 -```
859 -
860 - Using the above procedure, your plugin will be synchronized to start data collection on steps of `update_every`. There will be no need to keep track of latencies in data collection.
861 -
862 - Netdata interpolates values to second boundaries, so even if your plugin is not perfectly aligned it does not matter. Netdata will find out. When your plugin works in increments of `update_every`, there will be no gaps in the charts due to the possible cumulative micro-delays in data collection. Gaps will only appear if the data collection is really delayed.
863 -
864 -3. If you are not sure of memory leaks, exit every one hour. Netdata will re-start your process.
865 -
866 -4. If possible, try to autodetect if your plugin should be enabled, without any configuration.
867 -
868 -
src/go/cmd/godplugin/main.go
+264
@@ -3,11 +3,16 @@
3 package main
4
5 import (
6 + "context"
7 + "encoding/json"
8 "fmt"
9 + "io"
10 "log/slog"
11 "os"
12 "os/user"
13 + "strconv"
14 "strings"
15 + "time"
16
17 "go.uber.org/automaxprocs/maxprocs"
18 "golang.org/x/net/http/httpproxy"
@@ -16,8 +21,17 @@ import (
21 "github.com/netdata/netdata/go/plugins/pkg/buildinfo"
22 "github.com/netdata/netdata/go/plugins/pkg/cli"
23 "github.com/netdata/netdata/go/plugins/pkg/executable"
24 + "github.com/netdata/netdata/go/plugins/pkg/multipath"
25 + "github.com/netdata/netdata/go/plugins/pkg/netdataapi"
26 "github.com/netdata/netdata/go/plugins/pkg/pluginconfig"
27 "github.com/netdata/netdata/go/plugins/plugin/go.d/agent"
28 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/confgroup"
29 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/discovery/dummy"
30 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/discovery/file"
31 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/dyncfg"
32 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/functions"
33 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/jobmgr"
34 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
35 _ "github.com/netdata/netdata/go/plugins/plugin/go.d/collector"
36 )
37
@@ -40,6 +54,10 @@ func main() {
54
55 pluginconfig.MustInit(opts)
56
57 + if opts.Function != "" {
58 + os.Exit(runFunctionCLI(opts))
59 + }
60 +
61 if lvl := pluginconfig.EnvLogLevel(); lvl != "" {
62 logger.Level.SetByName(lvl)
63 }
@@ -85,3 +103,249 @@ func parseCLI() *cli.Option {
103
104 return opt
105 }
106 +
107 +func runFunctionCLI(opts *cli.Option) int {
108 + functionName := strings.TrimSpace(opts.Function)
109 + if functionName == "" {
110 + writeFunctionError(400, "missing function name (expected module:method)")
111 + return 1
112 + }
113 +
114 + moduleName, methodID, err := functions.SplitFunctionName(functionName)
115 + if err != nil {
116 + writeFunctionError(400, "%v", err)
117 + return 1
118 + }
119 +
120 + creator, ok := module.DefaultRegistry.Lookup(moduleName)
121 + if !ok {
122 + writeFunctionError(404, "unknown module '%s'", moduleName)
123 + return 1
124 + }
125 + if creator.Methods == nil {
126 + writeFunctionError(404, "module '%s' does not expose functions", moduleName)
127 + return 1
128 + }
129 + if methodID == "" {
130 + writeFunctionError(400, "missing method name in function '%s'", functionName)
131 + return 1
132 + }
133 +
134 + payloadBytes, payloadTimeout, err := readFunctionPayload(opts.FunctionPayload)
135 + if err != nil {
136 + writeFunctionError(400, "%v", err)
137 + return 1
138 + }
139 +
140 + timeout, err := resolveFunctionTimeout(opts.FunctionTimeout, payloadTimeout)
141 + if err != nil {
142 + writeFunctionError(400, "%v", err)
143 + return 1
144 + }
145 +
146 + reg := confgroup.Registry{}
147 + reg.Register(moduleName, confgroup.Default{
148 + MinUpdateEvery: opts.UpdateEvery,
149 + UpdateEvery: creator.UpdateEvery,
150 + AutoDetectionRetry: creator.AutoDetectionRetry,
151 + Priority: creator.Priority,
152 + })
153 +
154 + groups, err := loadConfigGroups(moduleName, reg, pluginconfig.CollectorsDir())
155 + if err != nil {
156 + writeFunctionError(500, "%v", err)
157 + return 1
158 + }
159 + if len(groups) == 0 {
160 + writeFunctionError(404, "no configs found for module '%s'", moduleName)
161 + return 1
162 + }
163 +
164 + ctx, cancel := context.WithCancel(context.Background())
165 + defer cancel()
166 +
167 + jobMgr := jobmgr.New()
168 + // Force-enable configs in function CLI runs (non-TTY by default).
169 + jobMgr.PluginName = "nodyncfg"
170 + jobMgr.Out = io.Discard
171 + jobMgr.VarLibDir = pluginconfig.VarLibDir()
172 + jobMgr.Modules = module.Registry{moduleName: creator}
173 + jobMgr.ConfigDefaults = reg
174 + jobMgr.FnReg = functions.NewManager()
175 + jobMgr.FunctionJSONWriter = func(payload []byte, _ int) {
176 + _, _ = os.Stdout.Write(payload)
177 + _, _ = os.Stdout.Write([]byte("\n"))
178 + }
179 + jobMgr.SetDyncfgResponder(dyncfg.NewResponder(netdataapi.New(io.Discard)))
180 +
181 + in := make(chan []*confgroup.Group, 1)
182 + go jobMgr.Run(ctx, in)
183 +
184 + startCtx, startCancel := context.WithTimeout(ctx, time.Second*10)
185 + defer startCancel()
186 + if ok := jobMgr.WaitStarted(startCtx); !ok {
187 + writeFunctionError(503, "job manager failed to start")
188 + return 1
189 + }
190 +
191 + in <- groups
192 +
193 + if err := waitForJobs(startCtx, jobMgr, moduleName); err != nil {
194 + writeFunctionError(503, "%v", err)
195 + return 1
196 + }
197 +
198 + fn := functions.Function{
199 + Name: functionName,
200 + Args: opts.FunctionArgs,
201 + Payload: payloadBytes,
202 + Timeout: timeout,
203 + ContentType: "application/json",
204 + }
205 + jobMgr.ExecuteFunction(functionName, fn)
206 +
207 + return 0
208 +}
209 +
210 +func readFunctionPayload(raw string) ([]byte, time.Duration, error) {
211 + if raw == "" {
212 + return nil, 0, nil
213 + }
214 +
215 + var data []byte
216 + var err error
217 + if strings.HasPrefix(raw, "@") {
218 + data, err = os.ReadFile(strings.TrimPrefix(raw, "@"))
219 + } else {
220 + data = []byte(raw)
221 + }
222 + if err != nil {
223 + return nil, 0, fmt.Errorf("read payload: %w", err)
224 + }
225 +
226 + var payload map[string]any
227 + if err := json.Unmarshal(data, &payload); err != nil {
228 + return nil, 0, fmt.Errorf("parse payload JSON: %w", err)
229 + }
230 +
231 + timeoutMs, ok, err := parsePayloadTimeout(payload)
232 + if err != nil {
233 + return nil, 0, err
234 + }
235 + if ok {
236 + return data, time.Duration(timeoutMs) * time.Millisecond, nil
237 + }
238 + return data, 0, nil
239 +}
240 +
241 +func parsePayloadTimeout(payload map[string]any) (int64, bool, error) {
242 + if payload == nil {
243 + return 0, false, nil
244 + }
245 + raw, ok := payload["timeout"]
246 + if !ok {
247 + return 0, false, nil
248 + }
249 + switch v := raw.(type) {
250 + case float64:
251 + return int64(v), true, nil
252 + case int:
253 + return int64(v), true, nil
254 + case int64:
255 + return v, true, nil
256 + case string:
257 + if v == "" {
258 + return 0, false, nil
259 + }
260 + n, err := strconv.ParseInt(v, 10, 64)
261 + if err != nil {
262 + return 0, false, fmt.Errorf("invalid payload timeout '%s'", v)
263 + }
264 + return n, true, nil
265 + default:
266 + return 0, false, fmt.Errorf("invalid payload timeout type %T", raw)
267 + }
268 +}
269 +
270 +func resolveFunctionTimeout(flagValue string, payloadTimeout time.Duration) (time.Duration, error) {
271 + if flagValue != "" {
272 + d, err := time.ParseDuration(flagValue)
273 + if err == nil {
274 + return d, nil
275 + }
276 + secs, err2 := strconv.ParseInt(flagValue, 10, 64)
277 + if err2 != nil {
278 + return 0, fmt.Errorf("invalid function-timeout '%s'", flagValue)
279 + }
280 + return time.Duration(secs) * time.Second, nil
281 + }
282 + if payloadTimeout > 0 {
283 + return payloadTimeout, nil
284 + }
285 + return time.Minute, nil
286 +}
287 +
288 +func loadConfigGroups(moduleName string, reg confgroup.Registry, collectors multipath.MultiPath) ([]*confgroup.Group, error) {
289 + if path, err := collectors.Find(moduleName + ".conf"); err == nil && path != "" {
290 + reader := file.NewReader(reg, []string{path})
291 + return runDiscoverer(reader)
292 + }
293 +
294 + disc, err := dummy.NewDiscovery(dummy.Config{
295 + Registry: reg,
296 + Names: []string{moduleName},
297 + })
298 + if err != nil {
299 + return nil, err
300 + }
301 + return runDiscoverer(disc)
302 +}
303 +
304 +type discoverer interface {
305 + Run(ctx context.Context, in chan<- []*confgroup.Group)
306 +}
307 +
308 +func runDiscoverer(d discoverer) ([]*confgroup.Group, error) {
309 + ctx, cancel := context.WithTimeout(context.Background(), time.Second*10)
310 + defer cancel()
311 +
312 + ch := make(chan []*confgroup.Group, 1)
313 + go d.Run(ctx, ch)
314 +
315 + select {
316 + case groups, ok := <-ch:
317 + if !ok {
318 + return nil, fmt.Errorf("discoverer returned no groups")
319 + }
320 + return groups, nil
321 + case <-ctx.Done():
322 + return nil, fmt.Errorf("discoverer timeout")
323 + }
324 +}
325 +
326 +func waitForJobs(ctx context.Context, mgr *jobmgr.Manager, moduleName string) error {
327 + for {
328 + if len(mgr.GetJobNames(moduleName)) > 0 {
329 + return nil
330 + }
331 + select {
332 + case <-ctx.Done():
333 + return fmt.Errorf("no jobs started for module '%s'", moduleName)
334 + case <-time.After(100 * time.Millisecond):
335 + }
336 + }
337 +}
338 +
339 +func writeFunctionError(status int, format string, args ...any) {
340 + resp := map[string]any{
341 + "status": status,
342 + "errorMessage": fmt.Sprintf(format, args...),
343 + }
344 + data, err := json.Marshal(resp)
345 + if err != nil {
346 + _, _ = fmt.Fprintf(os.Stdout, "{\"status\":%d,\"errorMessage\":\"%s\"}\n", status, "failed to encode error response")
347 + return
348 + }
349 + _, _ = os.Stdout.Write(data)
350 + _, _ = os.Stdout.Write([]byte("\n"))
351 +}
src/go/go.mod
+1
@@ -69,6 +69,7 @@ require (
69 github.com/alexbrainman/odbc v0.0.0-20250601004241-49e6b2bc0cf0
70 github.com/ibm-messaging/mq-golang/v5 v5.7.0
71 github.com/microsoft/go-mssqldb v1.9.6
72 + github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
73 gopkg.in/yaml.v3 v3.0.1
74 )
75
src/go/go.sum
+4
@@ -90,6 +90,8 @@ github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/r
90 github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
91 github.com/distribution/reference v0.5.0 h1:/FUIFXtfc/x2gpa5/VGfiGLuOIdYa1t65IKK2OFGvA0=
92 github.com/distribution/reference v0.5.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
93 +github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
94 +github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
95 github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM=
96 github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
97 github.com/docker/go-connections v0.4.0 h1:El9xVISelRB7BuFusrZozjnkIM5YnzCViNKohAFqRJQ=
@@ -341,6 +343,8 @@ github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0
343 github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
344 github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
345 github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
346 +github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 h1:KRzFb2m7YtdldCEkzs6KqmJw4nqEVZGK7IN2kJkjTuQ=
347 +github.com/santhosh-tekuri/jsonschema/v6 v6.0.2/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
348 github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4/go.mod h1:C1a7PQSMz9NShzorzCiG2fk9+xuCgLkPeCvMHYR2OWg=
349 github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp81k=
350 github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
src/go/pkg/cli/cli.go
+14 -10
@@ -12,16 +12,20 @@ import (
12
13 // Option defines command line options.
14 type Option struct {
15 - UpdateEvery int
16 - Module string `short:"m" long:"modules" description:"module name to run" default:"all"`
17 - Job []string `short:"j" long:"job" description:"job name to run"`
18 - ConfDir []string `short:"c" long:"config-dir" description:"config dir to read"`
19 - WatchPath []string `short:"w" long:"watch-path" description:"config path to watch"`
20 - Debug bool `short:"d" long:"debug" description:"debug mode"`
21 - Version bool `short:"v" long:"version" description:"display the version and exit"`
22 - DumpMode string `long:"dump" description:"run in dump mode for specified duration (e.g. 30s, 5m) and analyze metric structure"`
23 - DumpSummary bool `long:"dump-summary" description:"show consolidated summary across all jobs in dump mode"`
24 - DumpDataDir string `long:"dump-data" description:"write structured dump artifacts for the selected module to the given directory"`
15 + UpdateEvery int
16 + Module string `short:"m" long:"modules" description:"module name to run" default:"all"`
17 + Job []string `short:"j" long:"job" description:"job name to run"`
18 + ConfDir []string `short:"c" long:"config-dir" description:"config dir to read"`
19 + WatchPath []string `short:"w" long:"watch-path" description:"config path to watch"`
20 + Debug bool `short:"d" long:"debug" description:"debug mode"`
21 + Version bool `short:"v" long:"version" description:"display the version and exit"`
22 + DumpMode string `long:"dump" description:"run in dump mode for specified duration (e.g. 30s, 5m) and analyze metric structure"`
23 + DumpSummary bool `long:"dump-summary" description:"show consolidated summary across all jobs in dump mode"`
24 + DumpDataDir string `long:"dump-data" description:"write structured dump artifacts for the selected module to the given directory"`
25 + Function string `long:"function" description:"execute function once (module name)"`
26 + FunctionArgs []string `long:"function-args" description:"function args (repeatable, e.g. info)"`
27 + FunctionPayload string `long:"function-payload" description:"function payload JSON or @file.json"`
28 + FunctionTimeout string `long:"function-timeout" description:"function timeout (e.g. 60s)"`
29 }
30
31 // Parse returns parsed command-line flags in Option struct
src/go/pkg/funcapi/columns.go new
+78
@@ -0,0 +1,78 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package funcapi
4 +
5 +// ValueOptions defines per-column formatting settings.
6 +type ValueOptions struct {
7 + Transform FieldTransform
8 + DecimalPoints int
9 + DefaultValue any
10 +}
11 +
12 +// Column defines a table column for function responses.
13 +type Column struct {
14 + Index int
15 + Name string
16 + Type FieldType
17 + Units string
18 + Visualization FieldVisual
19 + Sort FieldSort
20 + Sortable bool
21 + Sticky bool
22 + Summary FieldSummary
23 + Filter FieldFilter
24 + FullWidth bool
25 + Wrap bool
26 + DefaultExpandedFilter bool
27 + UniqueKey bool
28 + Visible bool
29 + ValueOptions ValueOptions
30 + Max *float64
31 + PointerTo string
32 + Dummy bool
33 +}
34 +
35 +// BuildColumn converts a Column definition to the JSON map used by the UI.
36 +func (c Column) BuildColumn() map[string]any {
37 + col := map[string]any{
38 + "index": c.Index,
39 + "unique_key": c.UniqueKey,
40 + "name": c.Name,
41 + "visible": c.Visible,
42 + "type": c.Type.String(),
43 + "visualization": c.Visualization.String(),
44 + "sort": c.Sort.String(),
45 + "sortable": c.Sortable,
46 + "sticky": c.Sticky,
47 + "summary": c.Summary.String(),
48 + "filter": c.Filter.String(),
49 + "full_width": c.FullWidth,
50 + "wrap": c.Wrap,
51 + "default_expanded_filter": c.DefaultExpandedFilter,
52 + }
53 +
54 + if c.Units != "" {
55 + col["units"] = c.Units
56 + }
57 + if c.Max != nil {
58 + col["max"] = *c.Max
59 + }
60 + if c.PointerTo != "" {
61 + col["pointer_to"] = c.PointerTo
62 + }
63 + if c.Dummy {
64 + col["dummy"] = true
65 + }
66 +
67 + valueOpts := map[string]any{
68 + "transform": c.ValueOptions.Transform.String(),
69 + "decimal_points": c.ValueOptions.DecimalPoints,
70 + "default_value": c.ValueOptions.DefaultValue,
71 + }
72 + if c.Units != "" {
73 + valueOpts["units"] = c.Units
74 + }
75 + col["value_options"] = valueOpts
76 +
77 + return col
78 +}
src/go/pkg/funcapi/columns_test.go new
+94
@@ -0,0 +1,94 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package funcapi
4 +
5 +import (
6 + "testing"
7 +
8 + "github.com/stretchr/testify/assert"
9 + "github.com/stretchr/testify/require"
10 +)
11 +
12 +func TestColumnBuildColumn_AllFields(t *testing.T) {
13 + max := 99.5
14 + col := Column{
15 + Index: 3,
16 + Name: "CPU",
17 + Type: FieldTypeInteger,
18 + Units: "ms",
19 + Visualization: FieldVisualBar,
20 + Sort: FieldSortDescending,
21 + Sortable: true,
22 + Sticky: true,
23 + Summary: FieldSummarySum,
24 + Filter: FieldFilterRange,
25 + FullWidth: true,
26 + Wrap: true,
27 + DefaultExpandedFilter: true,
28 + UniqueKey: true,
29 + Visible: true,
30 + Max: &max,
31 + PointerTo: "details",
32 + Dummy: true,
33 + ValueOptions: ValueOptions{
34 + Transform: FieldTransformDuration,
35 + DecimalPoints: 2,
36 + DefaultValue: 0,
37 + },
38 + }
39 +
40 + result := col.BuildColumn()
41 +
42 + assert.Equal(t, 3, result["index"])
43 + assert.Equal(t, "CPU", result["name"])
44 + assert.Equal(t, "integer", result["type"])
45 + assert.Equal(t, "bar", result["visualization"])
46 + assert.Equal(t, "descending", result["sort"])
47 + assert.Equal(t, true, result["sortable"])
48 + assert.Equal(t, true, result["sticky"])
49 + assert.Equal(t, "sum", result["summary"])
50 + assert.Equal(t, "range", result["filter"])
51 + assert.Equal(t, true, result["full_width"])
52 + assert.Equal(t, true, result["wrap"])
53 + assert.Equal(t, true, result["default_expanded_filter"])
54 + assert.Equal(t, true, result["unique_key"])
55 + assert.Equal(t, true, result["visible"])
56 + assert.Equal(t, "ms", result["units"])
57 + assert.Equal(t, max, result["max"])
58 + assert.Equal(t, "details", result["pointer_to"])
59 + assert.Equal(t, true, result["dummy"])
60 +
61 + valueOpts, ok := result["value_options"].(map[string]any)
62 + require.True(t, ok, "value_options should be a map")
63 + assert.Equal(t, "duration", valueOpts["transform"])
64 + assert.Equal(t, 2, valueOpts["decimal_points"])
65 + assert.Equal(t, 0, valueOpts["default_value"])
66 + assert.Equal(t, "ms", valueOpts["units"])
67 +}
68 +
69 +func TestColumnBuildColumn_NoOptionalFields(t *testing.T) {
70 + col := Column{
71 + Index: 0,
72 + Name: "Query",
73 + Type: FieldTypeString,
74 + ValueOptions: ValueOptions{
75 + Transform: FieldTransformNone,
76 + },
77 + }
78 +
79 + result := col.BuildColumn()
80 +
81 + _, hasUnits := result["units"]
82 + _, hasMax := result["max"]
83 + _, hasPointer := result["pointer_to"]
84 + _, hasDummy := result["dummy"]
85 + assert.False(t, hasUnits, "units should be omitted when empty")
86 + assert.False(t, hasMax, "max should be omitted when nil")
87 + assert.False(t, hasPointer, "pointer_to should be omitted when empty")
88 + assert.False(t, hasDummy, "dummy should be omitted when false")
89 +
90 + valueOpts, ok := result["value_options"].(map[string]any)
91 + require.True(t, ok, "value_options should be a map")
92 + _, hasValueUnits := valueOpts["units"]
93 + assert.False(t, hasValueUnits, "value_options.units should be omitted when Units empty")
94 +}
src/go/pkg/funcapi/enums.go new
+209
@@ -0,0 +1,209 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package funcapi
4 +
5 +import "encoding/json"
6 +
7 +// FieldType defines the column data type.
8 +type FieldType uint8
9 +
10 +const (
11 + FieldTypeNone FieldType = iota
12 + FieldTypeInteger
13 + FieldTypeFloat
14 + FieldTypeBoolean
15 + FieldTypeString
16 + FieldTypeDetailString
17 + FieldTypeBarWithInteger
18 + FieldTypeDuration
19 + FieldTypeTimestamp
20 + FieldTypeArray
21 + FieldTypeFeedTemplate
22 +)
23 +
24 +func (t FieldType) String() string {
25 + switch t {
26 + case FieldTypeInteger:
27 + return "integer"
28 + case FieldTypeFloat:
29 + return "float"
30 + case FieldTypeBoolean:
31 + return "boolean"
32 + case FieldTypeString:
33 + return "string"
34 + case FieldTypeDetailString:
35 + return "detail-string"
36 + case FieldTypeBarWithInteger:
37 + return "bar-with-integer"
38 + case FieldTypeDuration:
39 + return "duration"
40 + case FieldTypeTimestamp:
41 + return "timestamp"
42 + case FieldTypeArray:
43 + return "array"
44 + case FieldTypeFeedTemplate:
45 + return "feedTemplate"
46 + default:
47 + return "none"
48 + }
49 +}
50 +
51 +func (t FieldType) MarshalJSON() ([]byte, error) {
52 + return json.Marshal(t.String())
53 +}
54 +
55 +// FieldVisual defines how values are rendered.
56 +type FieldVisual uint8
57 +
58 +const (
59 + FieldVisualValue FieldVisual = iota
60 + FieldVisualBar
61 + FieldVisualPill
62 + FieldVisualRichValue
63 + FieldVisualFeedTemplate
64 + FieldVisualRowOptions
65 +)
66 +
67 +func (v FieldVisual) String() string {
68 + switch v {
69 + case FieldVisualBar:
70 + return "bar"
71 + case FieldVisualPill:
72 + return "pill"
73 + case FieldVisualRichValue:
74 + return "richValue"
75 + case FieldVisualFeedTemplate:
76 + return "feedTemplate"
77 + case FieldVisualRowOptions:
78 + return "rowOptions"
79 + default:
80 + return "value"
81 + }
82 +}
83 +
84 +func (v FieldVisual) MarshalJSON() ([]byte, error) {
85 + return json.Marshal(v.String())
86 +}
87 +
88 +// FieldTransform defines value formatting.
89 +type FieldTransform uint8
90 +
91 +const (
92 + FieldTransformNone FieldTransform = iota
93 + FieldTransformNumber
94 + FieldTransformDuration
95 + FieldTransformDatetime
96 + FieldTransformDatetimeUsec
97 + FieldTransformText
98 + FieldTransformXML
99 +)
100 +
101 +func (t FieldTransform) String() string {
102 + switch t {
103 + case FieldTransformNumber:
104 + return "number"
105 + case FieldTransformDuration:
106 + return "duration"
107 + case FieldTransformDatetime:
108 + return "datetime"
109 + case FieldTransformDatetimeUsec:
110 + return "datetime_usec"
111 + case FieldTransformText:
112 + return "text"
113 + case FieldTransformXML:
114 + return "xml"
115 + default:
116 + return "none"
117 + }
118 +}
119 +
120 +func (t FieldTransform) MarshalJSON() ([]byte, error) {
121 + return json.Marshal(t.String())
122 +}
123 +
124 +// FieldSort defines the sort direction for a column.
125 +type FieldSort uint8
126 +
127 +const (
128 + FieldSortAscending FieldSort = iota
129 + FieldSortDescending
130 +)
131 +
132 +func (s FieldSort) String() string {
133 + switch s {
134 + case FieldSortDescending:
135 + return "descending"
136 + default:
137 + return "ascending"
138 + }
139 +}
140 +
141 +func (s FieldSort) MarshalJSON() ([]byte, error) {
142 + return json.Marshal(s.String())
143 +}
144 +
145 +// FieldSummary defines aggregation behavior.
146 +type FieldSummary uint8
147 +
148 +const (
149 + FieldSummaryCount FieldSummary = iota
150 + FieldSummaryUniqueCount
151 + FieldSummarySum
152 + FieldSummaryMin
153 + FieldSummaryMax
154 + FieldSummaryMean
155 + FieldSummaryMedian
156 +)
157 +
158 +func (s FieldSummary) String() string {
159 + switch s {
160 + case FieldSummaryUniqueCount:
161 + return "uniqueCount"
162 + case FieldSummarySum:
163 + return "sum"
164 + case FieldSummaryMin:
165 + return "min"
166 + case FieldSummaryMax:
167 + return "max"
168 + case FieldSummaryMean:
169 + return "mean"
170 + case FieldSummaryMedian:
171 + return "median"
172 + default:
173 + return "count"
174 + }
175 +}
176 +
177 +func (s FieldSummary) MarshalJSON() ([]byte, error) {
178 + return json.Marshal(s.String())
179 +}
180 +
181 +// FieldFilter defines filter UI type.
182 +type FieldFilter uint8
183 +
184 +const (
185 + FieldFilterNone FieldFilter = iota
186 + FieldFilterRange
187 + FieldFilterMultiselect
188 + FieldFilterText
189 + FieldFilterFacet
190 +)
191 +
192 +func (f FieldFilter) String() string {
193 + switch f {
194 + case FieldFilterRange:
195 + return "range"
196 + case FieldFilterMultiselect:
197 + return "multiselect"
198 + case FieldFilterText:
199 + return "text"
200 + case FieldFilterFacet:
201 + return "facet"
202 + default:
203 + return "none"
204 + }
205 +}
206 +
207 +func (f FieldFilter) MarshalJSON() ([]byte, error) {
208 + return json.Marshal(f.String())
209 +}
src/go/pkg/funcapi/enums_test.go new
+180
@@ -0,0 +1,180 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package funcapi
4 +
5 +import (
6 + "encoding/json"
7 + "testing"
8 +
9 + "github.com/stretchr/testify/assert"
10 + "github.com/stretchr/testify/require"
11 +)
12 +
13 +func TestFieldType_StringAndJSON(t *testing.T) {
14 + cases := []struct {
15 + value FieldType
16 + expected string
17 + }{
18 + {FieldTypeNone, "none"},
19 + {FieldTypeInteger, "integer"},
20 + {FieldTypeFloat, "float"},
21 + {FieldTypeBoolean, "boolean"},
22 + {FieldTypeString, "string"},
23 + {FieldTypeDetailString, "detail-string"},
24 + {FieldTypeBarWithInteger, "bar-with-integer"},
25 + {FieldTypeDuration, "duration"},
26 + {FieldTypeTimestamp, "timestamp"},
27 + {FieldTypeArray, "array"},
28 + {FieldTypeFeedTemplate, "feedTemplate"},
29 + {FieldType(250), "none"},
30 + }
31 +
32 + for _, tc := range cases {
33 + t.Run(tc.expected, func(t *testing.T) {
34 + assert.Equal(t, tc.expected, tc.value.String())
35 + data, err := tc.value.MarshalJSON()
36 + require.NoError(t, err)
37 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
38 + data, err = json.Marshal(tc.value)
39 + require.NoError(t, err)
40 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
41 + })
42 + }
43 +}
44 +
45 +func TestFieldVisual_StringAndJSON(t *testing.T) {
46 + cases := []struct {
47 + value FieldVisual
48 + expected string
49 + }{
50 + {FieldVisualValue, "value"},
51 + {FieldVisualBar, "bar"},
52 + {FieldVisualPill, "pill"},
53 + {FieldVisualRichValue, "richValue"},
54 + {FieldVisualFeedTemplate, "feedTemplate"},
55 + {FieldVisualRowOptions, "rowOptions"},
56 + {FieldVisual(250), "value"},
57 + }
58 +
59 + for _, tc := range cases {
60 + t.Run(tc.expected, func(t *testing.T) {
61 + assert.Equal(t, tc.expected, tc.value.String())
62 + data, err := json.Marshal(tc.value)
63 + require.NoError(t, err)
64 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
65 + })
66 + }
67 +}
68 +
69 +func TestFieldTransform_StringAndJSON(t *testing.T) {
70 + cases := []struct {
71 + value FieldTransform
72 + expected string
73 + }{
74 + {FieldTransformNone, "none"},
75 + {FieldTransformNumber, "number"},
76 + {FieldTransformDuration, "duration"},
77 + {FieldTransformDatetime, "datetime"},
78 + {FieldTransformDatetimeUsec, "datetime_usec"},
79 + {FieldTransformText, "text"},
80 + {FieldTransformXML, "xml"},
81 + {FieldTransform(250), "none"},
82 + }
83 +
84 + for _, tc := range cases {
85 + t.Run(tc.expected, func(t *testing.T) {
86 + assert.Equal(t, tc.expected, tc.value.String())
87 + data, err := json.Marshal(tc.value)
88 + require.NoError(t, err)
89 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
90 + })
91 + }
92 +}
93 +
94 +func TestFieldSort_StringAndJSON(t *testing.T) {
95 + cases := []struct {
96 + value FieldSort
97 + expected string
98 + }{
99 + {FieldSortAscending, "ascending"},
100 + {FieldSortDescending, "descending"},
101 + {FieldSort(250), "ascending"},
102 + }
103 +
104 + for _, tc := range cases {
105 + t.Run(tc.expected, func(t *testing.T) {
106 + assert.Equal(t, tc.expected, tc.value.String())
107 + data, err := json.Marshal(tc.value)
108 + require.NoError(t, err)
109 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
110 + })
111 + }
112 +}
113 +
114 +func TestFieldSummary_StringAndJSON(t *testing.T) {
115 + cases := []struct {
116 + value FieldSummary
117 + expected string
118 + }{
119 + {FieldSummaryCount, "count"},
120 + {FieldSummaryUniqueCount, "uniqueCount"},
121 + {FieldSummarySum, "sum"},
122 + {FieldSummaryMin, "min"},
123 + {FieldSummaryMax, "max"},
124 + {FieldSummaryMean, "mean"},
125 + {FieldSummaryMedian, "median"},
126 + {FieldSummary(250), "count"},
127 + }
128 +
129 + for _, tc := range cases {
130 + t.Run(tc.expected, func(t *testing.T) {
131 + assert.Equal(t, tc.expected, tc.value.String())
132 + data, err := json.Marshal(tc.value)
133 + require.NoError(t, err)
134 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
135 + })
136 + }
137 +}
138 +
139 +func TestFieldFilter_StringAndJSON(t *testing.T) {
140 + cases := []struct {
141 + value FieldFilter
142 + expected string
143 + }{
144 + {FieldFilterNone, "none"},
145 + {FieldFilterRange, "range"},
146 + {FieldFilterMultiselect, "multiselect"},
147 + {FieldFilterText, "text"},
148 + {FieldFilterFacet, "facet"},
149 + {FieldFilter(250), "none"},
150 + }
151 +
152 + for _, tc := range cases {
153 + t.Run(tc.expected, func(t *testing.T) {
154 + assert.Equal(t, tc.expected, tc.value.String())
155 + data, err := json.Marshal(tc.value)
156 + require.NoError(t, err)
157 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
158 + })
159 + }
160 +}
161 +
162 +func TestParamSelection_StringAndJSON(t *testing.T) {
163 + cases := []struct {
164 + value ParamSelection
165 + expected string
166 + }{
167 + {ParamSelect, "select"},
168 + {ParamMultiSelect, "multiselect"},
169 + {ParamSelection(250), "select"},
170 + }
171 +
172 + for _, tc := range cases {
173 + t.Run(tc.expected, func(t *testing.T) {
174 + assert.Equal(t, tc.expected, tc.value.String())
175 + data, err := json.Marshal(tc.value)
176 + require.NoError(t, err)
177 + assert.Equal(t, `"`+tc.expected+`"`, string(data))
178 + })
179 + }
180 +}
src/go/pkg/funcapi/params.go new
+254
@@ -0,0 +1,254 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package funcapi
4 +
5 +import "encoding/json"
6 +
7 +// ParamSelection defines the selection mode for required params.
8 +type ParamSelection uint8
9 +
10 +const (
11 + ParamSelect ParamSelection = iota
12 + ParamMultiSelect
13 +)
14 +
15 +func (p ParamSelection) String() string {
16 + switch p {
17 + case ParamMultiSelect:
18 + return "multiselect"
19 + default:
20 + return "select"
21 + }
22 +}
23 +
24 +func (p ParamSelection) MarshalJSON() ([]byte, error) {
25 + return json.Marshal(p.String())
26 +}
27 +
28 +// ParamOption defines a single option for a required param.
29 +// Column is not serialized and can be used for safe SQL mapping (e.g., __sort).
30 +type ParamOption struct {
31 + ID string
32 + Name string
33 + Default bool
34 + Disabled bool
35 + Sort *FieldSort
36 + Column string
37 +}
38 +
39 +// ParamConfig defines a required param and its available options.
40 +type ParamConfig struct {
41 + ID string
42 + Name string
43 + Help string
44 + Selection ParamSelection
45 + Options []ParamOption
46 + UniqueView bool
47 +}
48 +
49 +// RequiredParam converts ParamConfig to the wire format used by required_params.
50 +func (p ParamConfig) RequiredParam() map[string]any {
51 + out := map[string]any{
52 + "id": p.ID,
53 + "name": p.Name,
54 + "type": p.Selection.String(),
55 + "options": buildParamOptions(p.Options),
56 + }
57 + if p.Help != "" {
58 + out["help"] = p.Help
59 + }
60 + if p.UniqueView {
61 + out["unique_view"] = true
62 + }
63 + return out
64 +}
65 +
66 +func buildParamOptions(opts []ParamOption) []map[string]any {
67 + if len(opts) == 0 {
68 + return []map[string]any{}
69 + }
70 +
71 + hasDefault := false
72 + for _, opt := range opts {
73 + if opt.Default {
74 + hasDefault = true
75 + break
76 + }
77 + }
78 +
79 + options := make([]map[string]any, 0, len(opts))
80 + for i, opt := range opts {
81 + o := map[string]any{
82 + "id": opt.ID,
83 + "name": opt.Name,
84 + }
85 + if opt.Disabled {
86 + o["disabled"] = true
87 + }
88 + if opt.Sort != nil {
89 + o["sort"] = opt.Sort.String()
90 + }
91 + if opt.Default || (!hasDefault && i == 0) {
92 + o["defaultSelected"] = true
93 + }
94 + options = append(options, o)
95 + }
96 + return options
97 +}
98 +
99 +// ResolvedParam holds resolved values for a required param.
100 +type ResolvedParam struct {
101 + IDs []string
102 + Options []ParamOption
103 +}
104 +
105 +// GetOne returns the first selected ID.
106 +func (p ResolvedParam) GetOne() string {
107 + if len(p.IDs) > 0 {
108 + return p.IDs[0]
109 + }
110 + return ""
111 +}
112 +
113 +// ResolvedParams maps param ID to resolved values.
114 +type ResolvedParams map[string]ResolvedParam
115 +
116 +// Get returns all selected IDs for the param.
117 +func (p ResolvedParams) Get(id string) []string {
118 + if p == nil {
119 + return nil
120 + }
121 + return p[id].IDs
122 +}
123 +
124 +// GetOne returns the first selected ID for the param.
125 +func (p ResolvedParams) GetOne(id string) string {
126 + if p == nil {
127 + return ""
128 + }
129 + if v, ok := p[id]; ok && len(v.IDs) > 0 {
130 + return v.IDs[0]
131 + }
132 + return ""
133 +}
134 +
135 +// Option returns the first selected option for the param.
136 +func (p ResolvedParams) Option(id string) (ParamOption, bool) {
137 + if p == nil {
138 + return ParamOption{}, false
139 + }
140 + if v, ok := p[id]; ok && len(v.Options) > 0 {
141 + return v.Options[0], true
142 + }
143 + return ParamOption{}, false
144 +}
145 +
146 +// Column returns the column mapping for the selected option (used by __sort).
147 +// If the option has no Column mapping, the selected ID is returned.
148 +func (p ResolvedParams) Column(id string) string {
149 + opt, ok := p.Option(id)
150 + if !ok {
151 + return ""
152 + }
153 + if opt.Column != "" {
154 + return opt.Column
155 + }
156 + return opt.ID
157 +}
158 +
159 +// ResolveParam resolves user values against a ParamConfig, applying defaults as needed.
160 +func ResolveParam(cfg ParamConfig, values []string) ResolvedParam {
161 + byID := make(map[string]ParamOption, len(cfg.Options))
162 + for _, opt := range cfg.Options {
163 + byID[opt.ID] = opt
164 + }
165 +
166 + var selected []ParamOption
167 +
168 + switch cfg.Selection {
169 + case ParamMultiSelect:
170 + for _, val := range values {
171 + if opt, ok := byID[val]; ok {
172 + selected = append(selected, opt)
173 + }
174 + }
175 + default:
176 + if len(values) > 0 {
177 + if opt, ok := byID[values[0]]; ok {
178 + selected = []ParamOption{opt}
179 + }
180 + }
181 + }
182 +
183 + if len(selected) == 0 {
184 + selected = defaultOptions(cfg)
185 + }
186 +
187 + resolved := ResolvedParam{}
188 + if len(selected) > 0 {
189 + resolved.Options = selected
190 + resolved.IDs = make([]string, 0, len(selected))
191 + for _, opt := range selected {
192 + resolved.IDs = append(resolved.IDs, opt.ID)
193 + }
194 + }
195 + return resolved
196 +}
197 +
198 +// ResolveParams resolves multiple ParamConfig entries.
199 +func ResolveParams(cfgs []ParamConfig, values map[string][]string) ResolvedParams {
200 + resolved := ResolvedParams{}
201 + for _, cfg := range cfgs {
202 + resolved[cfg.ID] = ResolveParam(cfg, values[cfg.ID])
203 + }
204 + return resolved
205 +}
206 +
207 +func defaultOptions(cfg ParamConfig) []ParamOption {
208 + var defaults []ParamOption
209 + for _, opt := range cfg.Options {
210 + if opt.Default {
211 + defaults = append(defaults, opt)
212 + }
213 + }
214 + if len(defaults) > 0 {
215 + if cfg.Selection == ParamMultiSelect {
216 + return defaults
217 + }
218 + return []ParamOption{defaults[0]}
219 + }
220 + if len(cfg.Options) == 0 {
221 + return nil
222 + }
223 + return []ParamOption{cfg.Options[0]}
224 +}
225 +
226 +// MergeParamConfigs replaces base configs with overrides by ID, preserving base order.
227 +func MergeParamConfigs(base, overrides []ParamConfig) []ParamConfig {
228 + if len(overrides) == 0 {
229 + return base
230 + }
231 +
232 + overrideByID := make(map[string]ParamConfig, len(overrides))
233 + for _, cfg := range overrides {
234 + overrideByID[cfg.ID] = cfg
235 + }
236 +
237 + seen := make(map[string]bool, len(base)+len(overrides))
238 + merged := make([]ParamConfig, 0, len(base)+len(overrides))
239 + for _, cfg := range base {
240 + if override, ok := overrideByID[cfg.ID]; ok {
241 + merged = append(merged, override)
242 + seen[cfg.ID] = true
243 + continue
244 + }
245 + merged = append(merged, cfg)
246 + seen[cfg.ID] = true
247 + }
248 + for _, cfg := range overrides {
249 + if !seen[cfg.ID] {
250 + merged = append(merged, cfg)
251 + }
252 + }
253 + return merged
254 +}
src/go/pkg/funcapi/params_test.go new
+165
@@ -0,0 +1,165 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package funcapi
4 +
5 +import (
6 + "encoding/json"
7 + "testing"
8 +
9 + "github.com/stretchr/testify/assert"
10 + "github.com/stretchr/testify/require"
11 +)
12 +
13 +func TestParamConfigRequiredParam(t *testing.T) {
14 + sortDir := FieldSortDescending
15 + cfg := ParamConfig{
16 + ID: "__sort",
17 + Name: "Sort",
18 + Help: "Select a sort",
19 + Selection: ParamSelect,
20 + UniqueView: true,
21 + Options: []ParamOption{
22 + {ID: "calls", Name: "Calls"},
23 + {ID: "time", Name: "Time", Default: true, Sort: &sortDir},
24 + {ID: "disabled", Name: "Disabled", Disabled: true},
25 + },
26 + }
27 +
28 + param := cfg.RequiredParam()
29 + assert.Equal(t, "__sort", param["id"])
30 + assert.Equal(t, "Sort", param["name"])
31 + assert.Equal(t, "select", param["type"])
32 + assert.Equal(t, "Select a sort", param["help"])
33 + assert.Equal(t, true, param["unique_view"])
34 +
35 + options, ok := param["options"].([]map[string]any)
36 + require.True(t, ok, "options should be []map[string]any")
37 + require.Len(t, options, 3)
38 +
39 + assert.Equal(t, "calls", options[0]["id"])
40 + assert.Equal(t, "Calls", options[0]["name"])
41 + _, hasDefault0 := options[0]["defaultSelected"]
42 + assert.False(t, hasDefault0)
43 +
44 + assert.Equal(t, "time", options[1]["id"])
45 + assert.Equal(t, "Time", options[1]["name"])
46 + assert.Equal(t, "descending", options[1]["sort"])
47 + assert.Equal(t, true, options[1]["defaultSelected"])
48 +
49 + assert.Equal(t, "disabled", options[2]["id"])
50 + assert.Equal(t, "Disabled", options[2]["name"])
51 + assert.Equal(t, true, options[2]["disabled"])
52 +
53 + data, err := json.Marshal(param)
54 + require.NoError(t, err)
55 + assert.Contains(t, string(data), "\"defaultSelected\"")
56 +}
57 +
58 +func TestBuildParamOptions_DefaultFallback(t *testing.T) {
59 + cfg := ParamConfig{
60 + ID: "db",
61 + Name: "Database",
62 + Selection: ParamSelect,
63 + Options: []ParamOption{
64 + {ID: "a", Name: "A"},
65 + {ID: "b", Name: "B"},
66 + },
67 + }
68 +
69 + param := cfg.RequiredParam()
70 + options := param["options"].([]map[string]any)
71 + require.Len(t, options, 2)
72 + assert.Equal(t, true, options[0]["defaultSelected"], "first option should be default when none explicitly set")
73 + _, hasDefault := options[1]["defaultSelected"]
74 + assert.False(t, hasDefault)
75 +}
76 +
77 +func TestResolveParam_Select(t *testing.T) {
78 + cfg := ParamConfig{
79 + ID: "db",
80 + Name: "Database",
81 + Selection: ParamSelect,
82 + Options: []ParamOption{
83 + {ID: "a", Name: "A"},
84 + {ID: "b", Name: "B", Default: true},
85 + },
86 + }
87 +
88 + resolved := ResolveParam(cfg, []string{"a"})
89 + assert.Equal(t, []string{"a"}, resolved.IDs)
90 +
91 + resolved = ResolveParam(cfg, []string{"missing"})
92 + assert.Equal(t, []string{"b"}, resolved.IDs)
93 +
94 + cfg.Options[1].Default = false
95 + resolved = ResolveParam(cfg, []string{"missing"})
96 + assert.Equal(t, []string{"a"}, resolved.IDs)
97 +}
98 +
99 +func TestResolveParam_MultiSelect(t *testing.T) {
100 + cfg := ParamConfig{
101 + ID: "cols",
102 + Name: "Columns",
103 + Selection: ParamMultiSelect,
104 + Options: []ParamOption{
105 + {ID: "a", Name: "A", Default: true},
106 + {ID: "b", Name: "B"},
107 + {ID: "c", Name: "C", Default: true},
108 + },
109 + }
110 +
111 + resolved := ResolveParam(cfg, []string{"b", "c", "missing"})
112 + assert.Equal(t, []string{"b", "c"}, resolved.IDs)
113 +
114 + resolved = ResolveParam(cfg, []string{"missing"})
115 + assert.Equal(t, []string{"a", "c"}, resolved.IDs)
116 +
117 + cfg.Options[0].Default = false
118 + cfg.Options[2].Default = false
119 + resolved = ResolveParam(cfg, nil)
120 + assert.Equal(t, []string{"a"}, resolved.IDs)
121 +}
122 +
123 +func TestResolvedParamsHelpers(t *testing.T) {
124 + params := ResolvedParams{
125 + "__sort": {
126 + IDs: []string{"calls"},
127 + Options: []ParamOption{
128 + {ID: "calls", Column: "count_calls"},
129 + },
130 + },
131 + "db": {
132 + IDs: []string{"main"},
133 + Options: []ParamOption{
134 + {ID: "main"},
135 + },
136 + },
137 + }
138 +
139 + assert.Equal(t, []string{"calls"}, params.Get("__sort"))
140 + assert.Equal(t, "calls", params.GetOne("__sort"))
141 + opt, ok := params.Option("__sort")
142 + require.True(t, ok)
143 + assert.Equal(t, "calls", opt.ID)
144 + assert.Equal(t, "count_calls", params.Column("__sort"))
145 + assert.Equal(t, "main", params.Column("db"))
146 + assert.Equal(t, "", params.Column("missing"))
147 +}
148 +
149 +func TestMergeParamConfigs(t *testing.T) {
150 + base := []ParamConfig{
151 + {ID: "a"},
152 + {ID: "b"},
153 + }
154 + overrides := []ParamConfig{
155 + {ID: "b", Name: "override"},
156 + {ID: "c"},
157 + }
158 +
159 + merged := MergeParamConfigs(base, overrides)
160 + require.Len(t, merged, 3)
161 + assert.Equal(t, "a", merged[0].ID)
162 + assert.Equal(t, "b", merged[1].ID)
163 + assert.Equal(t, "override", merged[1].Name)
164 + assert.Equal(t, "c", merged[2].ID)
165 +}
src/go/pkg/netdataapi/api.go
+13
@@ -185,3 +185,16 @@ func (a *API) CONFIGDELETE(id string) {
185 func (a *API) CONFIGSTATUS(id, status string) {
186 _, _ = a.Write([]byte("CONFIG " + id + " status " + status + "\n\n"))
187 }
188 +
189 +// FUNCTIONGLOBAL registers a global function with Netdata.
190 +// Format: FUNCTION GLOBAL "<name>" <timeout> "<help>" "<tags>" <access> <priority> <version>
191 +func (a *API) FUNCTIONGLOBAL(opts FunctionGlobalOpts) {
192 + _, _ = a.Write([]byte("FUNCTION GLOBAL \"" +
193 + opts.Name + "\" " +
194 + strconv.Itoa(opts.Timeout) + " \"" +
195 + opts.Help + "\" \"" +
196 + opts.Tags + "\" " +
197 + opts.Access + " " +
198 + strconv.Itoa(opts.Priority) + " " +
199 + strconv.Itoa(opts.Version) + "\n\n"))
200 +}
src/go/pkg/netdataapi/opts.go
+11
@@ -55,3 +55,14 @@ type ConfigOpts struct {
55 Source string
56 SupportedCommands string
57 }
58 +
59 +// FunctionGlobalOpts contains options for registering a global function with Netdata
60 +type FunctionGlobalOpts struct {
61 + Name string // Function name
62 + Timeout int // Timeout in seconds
63 + Help string // Help text
64 + Tags string // Tags (e.g., "top")
65 + Access string // Access permissions in hex format (e.g., "0x0000")
66 + Priority int // Priority (higher = more important)
67 + Version int // Function version
68 +}
src/go/pkg/pluginconfig/pluginconfig.go
+3 -1
@@ -145,6 +145,7 @@ func (d *directories) initUserRoots(opts *cli.Option, env envData, execDir strin
145 }
146
147 relDir := safePathClean(filepath.Join(execDir, "..", "..", "..", "..", "etc", "netdata"))
148 + relDir = handleDirOnWin(env.cygwinBase, relDir, execDir)
149 if isDirExists(relDir) {
150 d.userConfigDirs = multipath.New(relDir)
151 return
@@ -177,6 +178,7 @@ func (d *directories) initStockRoot(env envData, execDir string) {
178 }
179
180 relDir := safePathClean(filepath.Join(execDir, "..", "..", "..", "..", "usr", "lib", "netdata", "conf.d"))
181 + relDir = handleDirOnWin(env.cygwinBase, relDir, execDir)
182 if isDirExists(relDir) {
183 d.stockConfigDir = relDir
184 return
@@ -292,7 +294,7 @@ func handleDirOnWin(base, p string, execDir string) string {
294 if base == "" || !strings.HasPrefix(p, "/") {
295 return p
296 }
295 - return filepath.Join(base, p)
297 + return filepath.Join(base, strings.TrimPrefix(p, "/"))
298 }
299
300 func isDirExists(dir string) bool {
src/go/plugin/go.d/agent/dyncfg/responder.go
+21 -1
@@ -52,11 +52,26 @@ func (r *Responder) SendCodef(fn functions.Function, code int, message string, a
52 })
53 }
54
55 -// SendJSON sends a JSON payload response
55 +// SendJSON sends a JSON payload response with HTTP 200 status
56 func (r *Responder) SendJSON(fn functions.Function, payload string) {
57 r.sendPayload(fn, payload, "application/json")
58 }
59
60 +// SendJSONWithCode sends a JSON payload response with a specific HTTP status code
61 +func (r *Responder) SendJSONWithCode(fn functions.Function, payload string, code int) {
62 + if fn.UID == "" {
63 + return
64 + }
65 +
66 + r.api.FUNCRESULT(netdataapi.FunctionResult{
67 + UID: fn.UID,
68 + ContentType: "application/json",
69 + Payload: payload,
70 + Code: strconv.Itoa(code),
71 + ExpireTimestamp: strconv.FormatInt(time.Now().Unix(), 10),
72 + })
73 +}
74 +
75 // SendYAML sends a YAML payload response
76 func (r *Responder) SendYAML(fn functions.Function, payload string) {
77 r.sendPayload(fn, payload, "application/yaml")
@@ -90,3 +105,8 @@ func (r *Responder) ConfigStatus(id string, status Status) {
105 func (r *Responder) ConfigDelete(id string) {
106 r.api.CONFIGDELETE(id)
107 }
108 +
109 +// FunctionGlobal registers a global function with Netdata
110 +func (r *Responder) FunctionGlobal(opts netdataapi.FunctionGlobalOpts) {
111 + r.api.FUNCTIONGLOBAL(opts)
112 +}
src/go/plugin/go.d/agent/functions/name.go new
+18
@@ -0,0 +1,18 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package functions
4 +
5 +import (
6 + "fmt"
7 + "strings"
8 +)
9 +
10 +// SplitFunctionName splits a function name into module and method parts.
11 +// Expected format: module:method.
12 +func SplitFunctionName(name string) (string, string, error) {
13 + parts := strings.SplitN(name, ":", 2)
14 + if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
15 + return "", "", fmt.Errorf("invalid function name '%s' (expected module:method)", name)
16 + }
17 + return parts[0], parts[1], nil
18 +}
src/go/plugin/go.d/agent/jobmgr/di.go
+2
@@ -12,6 +12,8 @@ type Vnodes interface {
12 }
13
14 type FunctionRegistry interface {
15 + Register(name string, fn func(functions.Function))
16 + Unregister(name string)
17 RegisterPrefix(name, prefix string, fn func(functions.Function))
18 UnregisterPrefix(name string, prefix string)
19 }
src/go/plugin/go.d/agent/jobmgr/funcshandler.go new
+624
@@ -0,0 +1,624 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package jobmgr
4 +
5 +import (
6 + "context"
7 + "encoding/json"
8 + "fmt"
9 + "slices"
10 + "strings"
11 +
12 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
13 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/functions"
14 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
15 +)
16 +
17 +const (
18 + paramJob = "__job"
19 +)
20 +
21 +// makeMethodFuncHandler creates a function handler for a module+method function (module:method).
22 +func (m *Manager) makeMethodFuncHandler(moduleName, methodID string) func(functions.Function) {
23 + return func(fn functions.Function) {
24 + // Check for "info" request
25 + if slices.Contains(fn.Args, "info") {
26 + m.handleMethodFuncInfo(moduleName, methodID, fn)
27 + return
28 + }
29 +
30 + methodCfg, ok := m.moduleFuncs.getMethod(moduleName, methodID)
31 + if !ok {
32 + m.respondError(fn, 404, "unknown method '%s' for module '%s'", methodID, moduleName)
33 + return
34 + }
35 +
36 + payload := parsePayload(fn.Payload)
37 + argValues := parseArgsParams(fn.Args)
38 +
39 + jobs := m.moduleFuncs.getJobNames(moduleName)
40 + if len(jobs) == 0 {
41 + m.respondError(fn, 503, "no %s instances configured", moduleName)
42 + return
43 + }
44 + jobParam := buildJobParamConfig(jobs)
45 + jobValues := paramValues(argValues, payload, paramJob)
46 + resolvedJob := funcapi.ResolveParam(jobParam, jobValues)
47 + jobName := resolvedJob.GetOne()
48 + if jobName == "" {
49 + m.respondError(fn, 404, "no %s instances configured", moduleName)
50 + return
51 + }
52 +
53 + // Get job WITH generation for race condition detection
54 + // The generation increments when a job is replaced (config reload)
55 + job, jobGen := m.moduleFuncs.getJobWithGeneration(moduleName, jobName)
56 + if job == nil {
57 + m.respondError(fn, 404, "unknown job '%s', available: %v", jobName, jobs)
58 + return
59 + }
60 +
61 + // Create context with timeout from function request
62 + // This ensures DB queries are cancelled if the function times out
63 + // NOTE: fn.Timeout is already a time.Duration (set by parser as seconds)
64 + // Do NOT multiply by time.Second again - that would create huge timeouts
65 + ctx, cancel := context.WithTimeout(context.Background(), fn.Timeout)
66 + defer cancel()
67 +
68 + // RACE CONDITION MITIGATION: Verify job is still running before handler
69 + // The job could be stopped between lookup and handler call
70 + if !job.IsRunning() {
71 + m.respondError(fn, 503, "job '%s' is no longer running", jobName)
72 + return
73 + }
74 +
75 + // Get the creator for this module to call HandleMethod
76 + creator, ok := m.moduleFuncs.getCreator(moduleName)
77 + if !ok || creator.HandleMethod == nil {
78 + m.respondError(fn, 500, "module '%s' does not implement HandleMethod", moduleName)
79 + return
80 + }
81 +
82 + // Resolve method-specific required params (job-aware)
83 + methodParams, paramsFromJob, err := m.resolveMethodParamsForJob(ctx, moduleName, methodID, methodCfg, job, creator)
84 + if err != nil {
85 + m.respondError(fn, 503, "job '%s' cannot provide parameters: %v", jobName, err)
86 + return
87 + }
88 +
89 + // Validate provided param values when job-specific options are available
90 + if paramsFromJob {
91 + if err := validateParamValues(methodParams, argValues, payload, jobName); err != nil {
92 + m.respondError(fn, 400, "%v", err)
93 + return
94 + }
95 + }
96 +
97 + methodParamValues := make(map[string][]string, len(methodParams))
98 + for _, paramCfg := range methodParams {
99 + methodParamValues[paramCfg.ID] = paramValues(argValues, payload, paramCfg.ID)
100 + }
101 + resolvedParams := funcapi.ResolveParams(methodParams, methodParamValues)
102 + resolvedParams[paramJob] = resolvedJob
103 +
104 + // Route to the module's handler - get DATA ONLY response
105 + dataResp := creator.HandleMethod(ctx, job, methodID, resolvedParams)
106 +
107 + // RACE CONDITION MITIGATION: Verify job was not replaced during handler execution
108 + // If a config reload replaced this job while we were querying, the response
109 + // might contain stale data or the connection might have been closed
110 + if !m.moduleFuncs.verifyJobGeneration(moduleName, jobName, jobGen) {
111 + // Job was replaced during our request - the response may be unreliable
112 + // Return error to prompt client to retry with new job instance
113 + m.respondError(fn, 503, "job '%s' was replaced during request, please retry", jobName)
114 + return
115 + }
116 +
117 + // Core injects required_params into the response before sending
118 + m.respondWithParams(fn, moduleName, dataResp, methodParams)
119 + }
120 +}
121 +
122 +// handleMethodFuncInfo handles "info" requests for a module:method function
123 +func (m *Manager) handleMethodFuncInfo(moduleName, methodID string, fn functions.Function) {
124 + methodCfg, ok := m.moduleFuncs.getMethod(moduleName, methodID)
125 + if !ok {
126 + m.respondError(fn, 404, "unknown method '%s' for module '%s'", methodID, moduleName)
127 + return
128 + }
129 +
130 + methodParams := m.unionMethodParams(moduleName, methodID, methodCfg, fn)
131 + help := methodCfg.Help
132 + if help == "" {
133 + help = fmt.Sprintf("%s %s data function", moduleName, methodID)
134 + }
135 +
136 + resp := map[string]any{
137 + "v": 3,
138 + "status": 200,
139 + "type": "table",
140 + "has_history": false,
141 + "help": help,
142 + "accepted_params": buildAcceptedParams(methodParams),
143 + "required_params": m.buildRequiredParams(moduleName, methodParams),
144 + }
145 +
146 + m.respondJSON(fn, resp)
147 +}
148 +
149 +// respondWithParams wraps the module's data response with current required_params
150 +func (m *Manager) respondWithParams(fn functions.Function, moduleName string, dataResp *module.FunctionResponse, methodParams []funcapi.ParamConfig) {
151 + // Nil guard: if module returns nil, treat as internal error
152 + if dataResp == nil {
153 + m.respondError(fn, 500, "internal error: module returned nil response")
154 + return
155 + }
156 +
157 + if dataResp.Status >= 400 {
158 + m.respondError(fn, dataResp.Status, "%s", dataResp.Message)
159 + return
160 + }
161 +
162 + paramsForResponse := methodParams
163 + if len(dataResp.RequiredParams) > 0 {
164 + paramsForResponse = funcapi.MergeParamConfigs(paramsForResponse, dataResp.RequiredParams)
165 + }
166 +
167 + // Build the full response with injected required_params
168 + // Use dynamic sort options from response if provided (reflects actual DB capabilities)
169 + resp := map[string]any{
170 + "v": 3,
171 + "status": dataResp.Status,
172 + "type": "table",
173 + "has_history": false,
174 + "help": dataResp.Help,
175 + "accepted_params": buildAcceptedParams(paramsForResponse),
176 + "required_params": m.buildRequiredParams(moduleName, paramsForResponse),
177 + }
178 +
179 + // Only include data fields when present (avoid null values on errors)
180 + if dataResp.Columns != nil {
181 + resp["columns"] = dataResp.Columns
182 + }
183 + if dataResp.Data != nil {
184 + resp["data"] = dataResp.Data
185 + }
186 + if dataResp.DefaultSortColumn != "" {
187 + resp["default_sort_column"] = dataResp.DefaultSortColumn
188 + }
189 +
190 + // Add chart configuration if provided
191 + if len(dataResp.Charts) > 0 {
192 + resp["charts"] = dataResp.Charts
193 + }
194 + if len(dataResp.DefaultCharts) > 0 {
195 + resp["default_charts"] = dataResp.DefaultCharts
196 + }
197 + if len(dataResp.GroupBy) > 0 {
198 + resp["group_by"] = dataResp.GroupBy
199 + }
200 +
201 + m.respondJSON(fn, resp)
202 +}
203 +
204 +// respondError sends a minimal error response (status + errorMessage).
205 +func (m *Manager) respondError(fn functions.Function, status int, format string, args ...any) {
206 + resp := map[string]any{
207 + "status": status,
208 + "errorMessage": fmt.Sprintf(format, args...),
209 + }
210 + m.respondJSON(fn, resp)
211 +}
212 +
213 +// buildRequiredParams creates the required_params array with current job list
214 +func (m *Manager) buildRequiredParams(moduleName string, methodParams []funcapi.ParamConfig) []map[string]any {
215 + jobs := m.moduleFuncs.getJobNames(moduleName)
216 +
217 + paramConfigs := []funcapi.ParamConfig{
218 + buildJobParamConfig(jobs),
219 + }
220 + paramConfigs = append(paramConfigs, methodParams...)
221 +
222 + required := make([]map[string]any, 0, len(paramConfigs))
223 + for _, cfg := range paramConfigs {
224 + required = append(required, cfg.RequiredParam())
225 + }
226 + return required
227 +}
228 +
229 +func (m *Manager) resolveMethodParamsForJob(ctx context.Context, moduleName, methodID string, methodCfg *module.MethodConfig, job *module.Job, creator module.Creator) ([]funcapi.ParamConfig, bool, error) {
230 + methodParams := methodCfg.RequiredParams
231 + if creator.MethodParams == nil {
232 + return methodParams, false, nil
233 + }
234 +
235 + jobParams, err := creator.MethodParams(ctx, job, methodID)
236 + if err != nil {
237 + return nil, false, err
238 + }
239 + if len(jobParams) == 0 {
240 + return methodParams, true, nil
241 + }
242 +
243 + return funcapi.MergeParamConfigs(methodParams, jobParams), true, nil
244 +}
245 +
246 +func (m *Manager) unionMethodParams(moduleName, methodID string, methodCfg *module.MethodConfig, fn functions.Function) []funcapi.ParamConfig {
247 + baseParams := methodCfg.RequiredParams
248 +
249 + creator, ok := m.moduleFuncs.getCreator(moduleName)
250 + if !ok || creator.MethodParams == nil {
251 + return baseParams
252 + }
253 +
254 + jobs := m.moduleFuncs.getJobNames(moduleName)
255 + if len(jobs) == 0 {
256 + return baseParams
257 + }
258 +
259 + ctx, cancel := context.WithTimeout(context.Background(), fn.Timeout)
260 + defer cancel()
261 +
262 + union := []funcapi.ParamConfig{}
263 + for _, jobName := range jobs {
264 + job, ok := m.moduleFuncs.getJob(moduleName, jobName)
265 + if !ok || job == nil {
266 + continue
267 + }
268 + params, err := creator.MethodParams(ctx, job, methodID)
269 + if err != nil {
270 + m.Debugf("method params unavailable for %s:%s job '%s': %v", moduleName, methodID, jobName, err)
271 + continue
272 + }
273 + if len(params) == 0 {
274 + continue
275 + }
276 + union = mergeParamConfigsUnion(union, params)
277 + }
278 + if len(union) == 0 {
279 + return baseParams
280 + }
281 +
282 + out := make([]funcapi.ParamConfig, 0, len(baseParams)+len(union))
283 + baseIndex := make(map[string]bool, len(baseParams))
284 + unionIndex := make(map[string]int, len(union))
285 + for i, cfg := range union {
286 + if cfg.ID != "" {
287 + unionIndex[cfg.ID] = i
288 + }
289 + }
290 +
291 + for _, cfg := range baseParams {
292 + if cfg.ID != "" {
293 + baseIndex[cfg.ID] = true
294 + }
295 + if i, ok := unionIndex[cfg.ID]; ok {
296 + out = append(out, mergeParamConfigMetadata(cfg, union[i]))
297 + continue
298 + }
299 + out = append(out, cfg)
300 + }
301 +
302 + for _, cfg := range union {
303 + if cfg.ID == "" || baseIndex[cfg.ID] {
304 + continue
305 + }
306 + out = append(out, cfg)
307 + }
308 + return out
309 +}
310 +
311 +func mergeParamConfigMetadata(base, add funcapi.ParamConfig) funcapi.ParamConfig {
312 + out := add
313 + if out.Name == "" {
314 + out.Name = base.Name
315 + }
316 + if out.Help == "" {
317 + out.Help = base.Help
318 + }
319 + if base.UniqueView {
320 + out.UniqueView = true
321 + }
322 + return out
323 +}
324 +
325 +func validateParamValues(methodParams []funcapi.ParamConfig, argValues map[string][]string, payload map[string]any, jobName string) error {
326 + for _, cfg := range methodParams {
327 + values := paramValues(argValues, payload, cfg.ID)
328 + if len(values) == 0 {
329 + continue
330 + }
331 + if cfg.Selection == funcapi.ParamSelect && len(values) > 1 {
332 + return fmt.Errorf("parameter '%s' expects a single value for job '%s'", cfg.ID, jobName)
333 + }
334 + allowed := allowedOptions(cfg.Options)
335 + for _, val := range values {
336 + if !allowed[val] {
337 + return fmt.Errorf("parameter '%s' option '%s' is not supported by job '%s'", cfg.ID, val, jobName)
338 + }
339 + }
340 + }
341 + return nil
342 +}
343 +
344 +func allowedOptions(options []funcapi.ParamOption) map[string]bool {
345 + allowed := make(map[string]bool, len(options))
346 + for _, opt := range options {
347 + if opt.ID == "" || opt.Disabled {
348 + continue
349 + }
350 + allowed[opt.ID] = true
351 + }
352 + return allowed
353 +}
354 +
355 +func mergeParamConfigsUnion(base, add []funcapi.ParamConfig) []funcapi.ParamConfig {
356 + if len(add) == 0 {
357 + return base
358 + }
359 +
360 + out := make([]funcapi.ParamConfig, len(base))
361 + copy(out, base)
362 +
363 + index := make(map[string]int, len(out))
364 + for i, cfg := range out {
365 + if cfg.ID != "" {
366 + index[cfg.ID] = i
367 + }
368 + }
369 +
370 + for _, cfg := range add {
371 + if cfg.ID == "" {
372 + continue
373 + }
374 + if i, ok := index[cfg.ID]; ok {
375 + out[i] = mergeParamConfigOptions(out[i], cfg)
376 + continue
377 + }
378 + out = append(out, cfg)
379 + index[cfg.ID] = len(out) - 1
380 + }
381 + return out
382 +}
383 +
384 +func mergeParamConfigOptions(base, add funcapi.ParamConfig) funcapi.ParamConfig {
385 + out := base
386 + if out.Name == "" {
387 + out.Name = add.Name
388 + }
389 + if out.Help == "" {
390 + out.Help = add.Help
391 + }
392 + if out.Selection == funcapi.ParamSelect && add.Selection == funcapi.ParamMultiSelect {
393 + out.Selection = add.Selection
394 + }
395 + if add.UniqueView {
396 + out.UniqueView = true
397 + }
398 +
399 + options := make([]funcapi.ParamOption, len(out.Options))
400 + copy(options, out.Options)
401 +
402 + optIndex := make(map[string]int, len(options))
403 + for i, opt := range options {
404 + if opt.ID != "" {
405 + optIndex[opt.ID] = i
406 + }
407 + }
408 +
409 + hasDefault := false
410 + for _, opt := range options {
411 + if opt.Default {
412 + hasDefault = true
413 + break
414 + }
415 + }
416 +
417 + for _, opt := range add.Options {
418 + if opt.ID == "" {
419 + continue
420 + }
421 + if i, ok := optIndex[opt.ID]; ok {
422 + merged := options[i]
423 + if merged.Name == "" {
424 + merged.Name = opt.Name
425 + }
426 + if merged.Sort == nil && opt.Sort != nil {
427 + merged.Sort = opt.Sort
428 + }
429 + if merged.Column == "" {
430 + merged.Column = opt.Column
431 + }
432 + if opt.Default && !hasDefault {
433 + merged.Default = true
434 + hasDefault = true
435 + }
436 + // Disabled should remain false if any job supports the option.
437 + merged.Disabled = merged.Disabled && opt.Disabled
438 + options[i] = merged
439 + continue
440 + }
441 +
442 + if opt.Default && hasDefault {
443 + opt.Default = false
444 + }
445 + options = append(options, opt)
446 + optIndex[opt.ID] = len(options) - 1
447 + if opt.Default {
448 + hasDefault = true
449 + }
450 + }
451 +
452 + out.Options = options
453 + return out
454 +}
455 +
456 +// respondJSON sends a JSON response to the function request
457 +// The HTTP status code is extracted from the "status" field in the response
458 +func (m *Manager) respondJSON(fn functions.Function, resp map[string]any) {
459 + data, err := json.Marshal(resp)
460 + if err != nil {
461 + m.Errorf("failed to marshal function response: %v", err)
462 + return
463 + }
464 +
465 + // Extract status code from response for pluginsd protocol
466 + // Default to 200 if not present or not an int
467 + code := 200
468 + if status, ok := resp["status"]; ok {
469 + switch v := status.(type) {
470 + case int:
471 + code = v
472 + case int64:
473 + code = int(v)
474 + case float64:
475 + code = int(v)
476 + }
477 + }
478 +
479 + if m.FunctionJSONWriter != nil {
480 + m.FunctionJSONWriter(data, code)
481 + return
482 + }
483 +
484 + m.dyncfgApi.SendJSONWithCode(fn, string(data), code)
485 +}
486 +
487 +func parsePayload(raw []byte) map[string]any {
488 + if len(raw) == 0 {
489 + return nil
490 + }
491 + var payload map[string]any
492 + if err := json.Unmarshal(raw, &payload); err != nil {
493 + return nil
494 + }
495 + return payload
496 +}
497 +
498 +func parseArgsParams(args []string) map[string][]string {
499 + if len(args) == 0 {
500 + return nil
501 + }
502 + params := make(map[string][]string)
503 + for _, arg := range args {
504 + if arg == "info" {
505 + continue
506 + }
507 + parts := strings.SplitN(arg, ":", 2)
508 + if len(parts) != 2 {
509 + continue
510 + }
511 + key := parts[0]
512 + value := parts[1]
513 + if key == "" || value == "" {
514 + continue
515 + }
516 + params[key] = splitCSV(value)
517 + }
518 + return params
519 +}
520 +
521 +func paramValues(args map[string][]string, payload map[string]any, key string) []string {
522 + if args != nil {
523 + if vals := args[key]; len(vals) > 0 {
524 + return vals
525 + }
526 + }
527 + return extractParamValues(payload, key)
528 +}
529 +
530 +// extractParamValues extracts parameter values from payload, checking selections first.
531 +func extractParamValues(payload map[string]any, key string) []string {
532 + if payload == nil {
533 + return nil
534 + }
535 + if selections, ok := payload["selections"].(map[string]any); ok {
536 + if vals := extractValues(selections[key]); len(vals) > 0 {
537 + return vals
538 + }
539 + }
540 + return extractValues(payload[key])
541 +}
542 +
543 +func extractValues(val any) []string {
544 + switch v := val.(type) {
545 + case string:
546 + if v == "" {
547 + return nil
548 + }
549 + return []string{v}
550 + case []any:
551 + var out []string
552 + for _, item := range v {
553 + if s, ok := item.(string); ok && s != "" {
554 + out = append(out, s)
555 + }
556 + }
557 + return out
558 + case []string:
559 + var out []string
560 + for _, s := range v {
561 + if s != "" {
562 + out = append(out, s)
563 + }
564 + }
565 + return out
566 + default:
567 + return nil
568 + }
569 +}
570 +
571 +func splitCSV(value string) []string {
572 + if !strings.Contains(value, ",") {
573 + return []string{value}
574 + }
575 + parts := strings.Split(value, ",")
576 + out := make([]string, 0, len(parts))
577 + for _, p := range parts {
578 + if p == "" {
579 + continue
580 + }
581 + out = append(out, p)
582 + }
583 + return out
584 +}
585 +
586 +func buildJobParamConfig(jobs []string) funcapi.ParamConfig {
587 + options := make([]funcapi.ParamOption, 0, len(jobs))
588 + if len(jobs) == 0 {
589 + options = append(options, funcapi.ParamOption{
590 + ID: "",
591 + Name: "(No instances configured)",
592 + Disabled: true,
593 + })
594 + } else {
595 + for i, j := range jobs {
596 + opt := funcapi.ParamOption{
597 + ID: j,
598 + Name: j,
599 + }
600 + if i == 0 {
601 + opt.Default = true
602 + }
603 + options = append(options, opt)
604 + }
605 + }
606 + return funcapi.ParamConfig{
607 + ID: paramJob,
608 + Name: "Instance",
609 + Help: "Select which database instance to query",
610 + Selection: funcapi.ParamSelect,
611 + Options: options,
612 + UniqueView: true,
613 + }
614 +}
615 +
616 +func buildAcceptedParams(methodParams []funcapi.ParamConfig) []string {
617 + accepted := []string{paramJob}
618 + for _, p := range methodParams {
619 + if !slices.Contains(accepted, p.ID) {
620 + accepted = append(accepted, p.ID)
621 + }
622 + }
623 + return accepted
624 +}
src/go/plugin/go.d/agent/jobmgr/funcshandler_test.go new
+267
@@ -0,0 +1,267 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package jobmgr
4 +
5 +import (
6 + "context"
7 + "errors"
8 + "testing"
9 + "time"
10 +
11 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
12 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/functions"
13 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
14 + "github.com/stretchr/testify/assert"
15 +)
16 +
17 +func TestExtractParamValues(t *testing.T) {
18 + tests := map[string]struct {
19 + payload map[string]any
20 + key string
21 + expected []string
22 + }{
23 + "string value": {
24 + payload: map[string]any{"__job": "local"},
25 + key: "__job",
26 + expected: []string{"local"},
27 + },
28 + "array value (single element)": {
29 + payload: map[string]any{"__job": []any{"local"}},
30 + key: "__job",
31 + expected: []string{"local"},
32 + },
33 + "array value (multiple elements)": {
34 + payload: map[string]any{"__sort": []any{"calls", "total_time"}},
35 + key: "__sort",
36 + expected: []string{"calls", "total_time"},
37 + },
38 + "string array value": {
39 + payload: map[string]any{"__job": []string{"local"}},
40 + key: "__job",
41 + expected: []string{"local"},
42 + },
43 + "missing key": {
44 + payload: map[string]any{"__job": "local"},
45 + key: "__sort",
46 + expected: nil,
47 + },
48 + "empty payload": {
49 + payload: map[string]any{},
50 + key: "__job",
51 + expected: nil,
52 + },
53 + "nil in array": {
54 + payload: map[string]any{"__job": []any{nil, "test"}},
55 + key: "__job",
56 + expected: []string{"test"},
57 + },
58 + "empty array": {
59 + payload: map[string]any{"__job": []any{}},
60 + key: "__job",
61 + expected: nil,
62 + },
63 + "non-string value": {
64 + payload: map[string]any{"__job": 123},
65 + key: "__job",
66 + expected: nil,
67 + },
68 + "prefers selections": {
69 + payload: map[string]any{
70 + "__job": "root",
71 + "selections": map[string]any{
72 + "__job": []any{"selected"},
73 + },
74 + },
75 + key: "__job",
76 + expected: []string{"selected"},
77 + },
78 + }
79 +
80 + for name, tc := range tests {
81 + t.Run(name, func(t *testing.T) {
82 + result := extractParamValues(tc.payload, tc.key)
83 + assert.Equal(t, tc.expected, result)
84 + })
85 + }
86 +}
87 +
88 +func TestBuildAcceptedParams(t *testing.T) {
89 + sortDir := funcapi.FieldSortDescending
90 + methodParams := []funcapi.ParamConfig{
91 + {ID: "__sort", Selection: funcapi.ParamSelect, Options: []funcapi.ParamOption{{ID: "calls", Name: "Calls", Sort: &sortDir}}},
92 + {ID: "db"},
93 + {ID: "extra"},
94 + }
95 +
96 + result := buildAcceptedParams(methodParams)
97 + assert.Equal(t, []string{"__job", "__sort", "db", "extra"}, result)
98 +}
99 +
100 +// TestBuildRequiredParams_TypeSelect verifies that all selectors use type "select" (single-select)
101 +// This is critical because type "multiselect" would show checkboxes instead of dropdowns
102 +func TestBuildRequiredParams_TypeSelect(t *testing.T) {
103 + // Setup a minimal manager with test data
104 + r := newModuleFuncRegistry()
105 + r.registerModule("postgres", module.Creator{
106 + Methods: func() []module.MethodConfig {
107 + return []module.MethodConfig{{
108 + ID: "top-queries",
109 + Name: "Top Queries",
110 + }}
111 + },
112 + })
113 + r.addJob("postgres", "master-db", newTestModuleFuncsJob("master-db"))
114 +
115 + // Create a manager with the registry
116 + mgr := &Manager{moduleFuncs: r}
117 +
118 + // Get required_params through the public method
119 + methodParams := []funcapi.ParamConfig{
120 + {
121 + ID: "__sort",
122 + Name: "Filter By",
123 + Selection: funcapi.ParamSelect,
124 + UniqueView: true,
125 + Options: []funcapi.ParamOption{
126 + {ID: "total_time", Name: "By Total Time", Default: true},
127 + },
128 + },
129 + }
130 + params := mgr.buildRequiredParams("postgres", methodParams)
131 +
132 + // Verify structure
133 + assert.Len(t, params, 2, "should have 2 required params: __job, __sort")
134 +
135 + // All params should have type: "select" (NOT "multiselect")
136 + for _, param := range params {
137 + paramType, ok := param["type"]
138 + assert.True(t, ok, "param should have type field")
139 + assert.Equal(t, "select", paramType, "param type must be 'select' for single-select, not 'multiselect'")
140 +
141 + // Verify required fields exist
142 + assert.Contains(t, param, "id", "param should have id")
143 + assert.Contains(t, param, "name", "param should have name")
144 + assert.Contains(t, param, "options", "param should have options")
145 + assert.Contains(t, param, "unique_view", "param should have unique_view")
146 +
147 + // Verify unique_view is true
148 + uniqueView, _ := param["unique_view"].(bool)
149 + assert.True(t, uniqueView, "unique_view should be true")
150 + }
151 +
152 + // Verify specific param IDs
153 + assert.Equal(t, "__job", params[0]["id"])
154 + assert.Equal(t, "__sort", params[1]["id"])
155 +}
156 +
157 +func TestUnionMethodParams_JobOptionsOverrideStatic(t *testing.T) {
158 + sortDir := funcapi.FieldSortDescending
159 + baseParams := []funcapi.ParamConfig{
160 + {
161 + ID: "__sort",
162 + Name: "Filter By",
163 + Selection: funcapi.ParamSelect,
164 + UniqueView: true,
165 + Options: []funcapi.ParamOption{
166 + {ID: "a", Name: "A", Sort: &sortDir},
167 + {ID: "b", Name: "B", Sort: &sortDir},
168 + },
169 + },
170 + {
171 + ID: "mode",
172 + Name: "Mode",
173 + Selection: funcapi.ParamSelect,
174 + Options: []funcapi.ParamOption{{ID: "x", Name: "X"}},
175 + },
176 + }
177 +
178 + r := newModuleFuncRegistry()
179 + r.registerModule("postgres", module.Creator{
180 + Methods: func() []module.MethodConfig {
181 + return []module.MethodConfig{{ID: "top-queries", RequiredParams: baseParams}}
182 + },
183 + MethodParams: func(ctx context.Context, job *module.Job, method string) ([]funcapi.ParamConfig, error) {
184 + switch job.Name() {
185 + case "job1":
186 + return []funcapi.ParamConfig{{
187 + ID: "__sort",
188 + Name: "Filter By",
189 + Selection: funcapi.ParamSelect,
190 + UniqueView: true,
191 + Options: []funcapi.ParamOption{
192 + {ID: "b", Name: "B", Sort: &sortDir},
193 + {ID: "c", Name: "C", Sort: &sortDir},
194 + },
195 + }}, nil
196 + case "job2":
197 + return []funcapi.ParamConfig{{
198 + ID: "__sort",
199 + Name: "Filter By",
200 + Selection: funcapi.ParamSelect,
201 + UniqueView: true,
202 + Options: []funcapi.ParamOption{
203 + {ID: "c", Name: "C", Sort: &sortDir},
204 + {ID: "d", Name: "D", Sort: &sortDir},
205 + },
206 + }}, nil
207 + default:
208 + return nil, nil
209 + }
210 + },
211 + })
212 + r.addJob("postgres", "job1", newTestModuleFuncsJob("job1"))
213 + r.addJob("postgres", "job2", newTestModuleFuncsJob("job2"))
214 +
215 + mgr := &Manager{moduleFuncs: r}
216 + fn := functions.Function{Timeout: time.Second}
217 +
218 + got := mgr.unionMethodParams("postgres", "top-queries", &module.MethodConfig{
219 + ID: "top-queries",
220 + RequiredParams: baseParams,
221 + }, fn)
222 +
223 + assert.Len(t, got, 2)
224 + assert.Equal(t, "__sort", got[0].ID)
225 + assert.Equal(t, "mode", got[1].ID)
226 +
227 + sortOpts := make(map[string]bool)
228 + for _, opt := range got[0].Options {
229 + sortOpts[opt.ID] = true
230 + }
231 + assert.False(t, sortOpts["a"], "static-only option should not be included when jobs provide options")
232 + assert.True(t, sortOpts["b"])
233 + assert.True(t, sortOpts["c"])
234 + assert.True(t, sortOpts["d"])
235 +}
236 +
237 +func TestUnionMethodParams_FallbackToStaticOnAllErrors(t *testing.T) {
238 + baseParams := []funcapi.ParamConfig{
239 + {
240 + ID: "__sort",
241 + Name: "Sort",
242 + Selection: funcapi.ParamSelect,
243 + Options: []funcapi.ParamOption{{ID: "a", Name: "A"}},
244 + },
245 + }
246 +
247 + r := newModuleFuncRegistry()
248 + r.registerModule("postgres", module.Creator{
249 + Methods: func() []module.MethodConfig {
250 + return []module.MethodConfig{{ID: "top-queries", RequiredParams: baseParams}}
251 + },
252 + MethodParams: func(ctx context.Context, job *module.Job, method string) ([]funcapi.ParamConfig, error) {
253 + return nil, errors.New("backend unavailable")
254 + },
255 + })
256 + r.addJob("postgres", "job1", newTestModuleFuncsJob("job1"))
257 +
258 + mgr := &Manager{moduleFuncs: r}
259 + fn := functions.Function{Timeout: time.Second}
260 +
261 + got := mgr.unionMethodParams("postgres", "top-queries", &module.MethodConfig{
262 + ID: "top-queries",
263 + RequiredParams: baseParams,
264 + }, fn)
265 +
266 + assert.Equal(t, baseParams, got)
267 +}
src/go/plugin/go.d/agent/jobmgr/manager.go
+87 -2
@@ -40,6 +40,7 @@ func New() *Manager {
40
41 Vnodes: make(map[string]*vnodes.VirtualNode),
42
43 + moduleFuncs: newModuleFuncRegistry(),
44 discoveredConfigs: newDiscoveredConfigsCache(),
45 seenConfigs: newSeenConfigCache(),
46 exposedConfigs: newExposedConfigCache(),
@@ -56,6 +57,13 @@ func New() *Manager {
57 return mgr
58 }
59
60 +// SetDyncfgResponder allows overriding the default responder (e.g., to silence output in CLI mode).
61 +func (m *Manager) SetDyncfgResponder(responder *dyncfg.Responder) {
62 + if responder != nil {
63 + m.dyncfgApi = responder
64 + }
65 +}
66 +
67 type Manager struct {
68 *logger.Logger
69
@@ -73,7 +81,8 @@ type Manager struct {
81 DumpAnalyzer interface{} // Will be *agent.DumpAnalyzer but avoid circular dependency
82 DumpDataDir string
83
76 - fileStatus *fileStatus
84 + fileStatus *fileStatus
85 + moduleFuncs *moduleFuncRegistry
86
87 discoveredConfigs *discoveredConfigs
88 seenConfigs *seenConfigs
@@ -91,6 +100,9 @@ type Manager struct {
100 waitCfgOnOff string // block processing of discovered configs until "enable"/"disable" is received from Netdata
101
102 dyncfgApi *dyncfg.Responder
103 +
104 + // FunctionJSONWriter, when set, bypasses Netdata protocol output and writes raw JSON.
105 + FunctionJSONWriter func(payload []byte, code int)
106 }
107
108 func (m *Manager) Run(ctx context.Context, in chan []*confgroup.Group) {
@@ -107,8 +119,37 @@ func (m *Manager) Run(ctx context.Context, in chan []*confgroup.Group) {
119 m.dyncfgVnodeJobCreate(cfg, dyncfg.StatusRunning)
120 }
121
110 - for name := range m.Modules {
122 + for name, creator := range m.Modules {
123 m.dyncfgCollectorModuleCreate(name)
124 +
125 + // Register module-level function if this module provides methods
126 + if creator.Methods != nil {
127 + m.moduleFuncs.registerModule(name, creator)
128 + methods := creator.Methods()
129 + for _, method := range methods {
130 + if method.ID == "" {
131 + m.Warningf("skipping function registration for module '%s': empty method ID", name)
132 + continue
133 + }
134 + funcName := fmt.Sprintf("%s:%s", name, method.ID)
135 + m.FnReg.Register(funcName, m.makeMethodFuncHandler(name, method.ID))
136 +
137 + // Notify Netdata about this function so it appears in the functions API
138 + help := method.Help
139 + if help == "" {
140 + help = fmt.Sprintf("%s %s data function", name, method.ID)
141 + }
142 + m.dyncfgApi.FunctionGlobal(netdataapi.FunctionGlobalOpts{
143 + Name: funcName,
144 + Timeout: 60,
145 + Help: help,
146 + Tags: "top",
147 + Access: "0x0000",
148 + Priority: 100,
149 + Version: 3,
150 + })
151 + }
152 + }
153 }
154
155 m.loadFileStatus()
@@ -133,6 +174,32 @@ func (m *Manager) Run(ctx context.Context, in chan []*confgroup.Group) {
174 <-m.ctx.Done()
175 }
176
177 +// WaitStarted blocks until Run has completed initialization or the context is canceled.
178 +func (m *Manager) WaitStarted(ctx context.Context) bool {
179 + select {
180 + case <-m.started:
181 + return true
182 + case <-ctx.Done():
183 + return false
184 + }
185 +}
186 +
187 +// GetJobNames returns the currently running job names for a module.
188 +func (m *Manager) GetJobNames(moduleName string) []string {
189 + return m.moduleFuncs.getJobNames(moduleName)
190 +}
191 +
192 +// ExecuteFunction executes a function handler directly (function name must be module:method).
193 +func (m *Manager) ExecuteFunction(functionName string, fn functions.Function) {
194 + moduleName, methodID, err := functions.SplitFunctionName(functionName)
195 + if err != nil {
196 + m.respondError(fn, 400, "%v", err)
197 + return
198 + }
199 + handler := m.makeMethodFuncHandler(moduleName, methodID)
200 + handler(fn)
201 +}
202 +
203 func (m *Manager) runProcessConfGroups(in chan []*confgroup.Group) {
204 for {
205 select {
@@ -266,6 +333,9 @@ func (m *Manager) startRunningJob(job *module.Job) {
333
334 go job.Start()
335 m.runningJobs.add(job.FullName(), job)
336 +
337 + // Track job for module function routing
338 + m.moduleFuncs.addJob(job.ModuleName(), job.Name(), job)
339 }
340
341 func (m *Manager) stopRunningJob(name string) {
@@ -277,6 +347,8 @@ func (m *Manager) stopRunningJob(name string) {
347 m.runningJobs.unlock()
348
349 if ok {
350 + // Remove job from module function registry
351 + m.moduleFuncs.removeJob(job.ModuleName(), job.Name())
352 job.Stop()
353 }
354 }
@@ -285,6 +357,19 @@ func (m *Manager) cleanup() {
357 m.FnReg.UnregisterPrefix("config", m.dyncfgCollectorPrefixValue())
358 m.FnReg.UnregisterPrefix("config", m.dyncfgVnodePrefixValue())
359
360 + // Unregister module functions
361 + for name, creator := range m.Modules {
362 + if creator.Methods != nil {
363 + for _, method := range creator.Methods() {
364 + if method.ID == "" {
365 + continue
366 + }
367 + funcName := fmt.Sprintf("%s:%s", name, method.ID)
368 + m.FnReg.Unregister(funcName)
369 + }
370 + }
371 + }
372 +
373 m.runningJobs.lock()
374 defer m.runningJobs.unlock()
375
src/go/plugin/go.d/agent/jobmgr/modulefuncs.go new
+232
@@ -0,0 +1,232 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package jobmgr
4 +
5 +import (
6 + "sort"
7 + "sync"
8 +
9 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
10 +)
11 +
12 +// moduleFuncRegistry tracks module-level functions and their running jobs
13 +type moduleFuncRegistry struct {
14 + mu sync.RWMutex
15 +
16 + // moduleName → moduleFunc
17 + // e.g., "postgres" → {methods: [...], jobs: {"master-db": job1, "replica-db": job2}}
18 + modules map[string]*moduleFunc
19 +}
20 +
21 +type moduleFunc struct {
22 + creator module.Creator // The module creator (has Methods())
23 + methods []module.MethodConfig // Static methods from creator (ordered)
24 + methodsByID map[string]module.MethodConfig
25 + jobs map[string]*jobEntry // jobName → job entry with generation
26 + lastGeneration map[string]uint64 // jobName → last known generation (persists across removals)
27 +}
28 +
29 +// jobEntry wraps a job with a generation number for race detection
30 +type jobEntry struct {
31 + job *module.Job
32 + generation uint64 // Incremented each time this job name is replaced
33 +}
34 +
35 +func newModuleFuncRegistry() *moduleFuncRegistry {
36 + return &moduleFuncRegistry{
37 + modules: make(map[string]*moduleFunc),
38 + }
39 +}
40 +
41 +// registerModule is called at startup for each module that implements FunctionProvider
42 +func (r *moduleFuncRegistry) registerModule(name string, creator module.Creator) {
43 + r.mu.Lock()
44 + defer r.mu.Unlock()
45 +
46 + var methods []module.MethodConfig
47 + if creator.Methods != nil {
48 + methods = creator.Methods()
49 + }
50 +
51 + r.modules[name] = &moduleFunc{
52 + creator: creator,
53 + methods: methods,
54 + methodsByID: indexMethods(methods),
55 + jobs: make(map[string]*jobEntry),
56 + lastGeneration: make(map[string]uint64),
57 + }
58 +}
59 +
60 +func indexMethods(methods []module.MethodConfig) map[string]module.MethodConfig {
61 + if len(methods) == 0 {
62 + return nil
63 + }
64 + idx := make(map[string]module.MethodConfig, len(methods))
65 + for _, m := range methods {
66 + if m.ID == "" {
67 + continue
68 + }
69 + idx[m.ID] = m
70 + }
71 + return idx
72 +}
73 +
74 +// addJob is called when jobs start.
75 +// NOTE: Job names MUST be unique per module. If a job with the same name
76 +// already exists, it is replaced (this handles config reload scenarios)
77 +func (r *moduleFuncRegistry) addJob(moduleName, jobName string, job *module.Job) {
78 + r.mu.Lock()
79 + defer r.mu.Unlock()
80 +
81 + mf, ok := r.modules[moduleName]
82 + if !ok {
83 + return // Module not registered (doesn't implement FunctionProvider)
84 + }
85 +
86 + // Replace any existing job with the same name
87 + // Increment generation to invalidate any in-flight requests to old job
88 + // Use lastGeneration to ensure generation monotonically increases even across removals
89 + lastGen := mf.lastGeneration[jobName]
90 + newGen := lastGen + 1
91 + mf.jobs[jobName] = &jobEntry{
92 + job: job,
93 + generation: newGen,
94 + }
95 + mf.lastGeneration[jobName] = newGen
96 +}
97 +
98 +// removeJob is called when jobs stop
99 +func (r *moduleFuncRegistry) removeJob(moduleName, jobName string) {
100 + r.mu.Lock()
101 + defer r.mu.Unlock()
102 +
103 + if mf, ok := r.modules[moduleName]; ok {
104 + delete(mf.jobs, jobName)
105 + }
106 +}
107 +
108 +// getJobWithGeneration returns the job and its generation for race detection
109 +func (r *moduleFuncRegistry) getJobWithGeneration(moduleName, jobName string) (*module.Job, uint64) {
110 + r.mu.RLock()
111 + defer r.mu.RUnlock()
112 +
113 + mf, ok := r.modules[moduleName]
114 + if !ok {
115 + return nil, 0
116 + }
117 + entry, ok := mf.jobs[jobName]
118 + if !ok {
119 + return nil, 0
120 + }
121 + return entry.job, entry.generation
122 +}
123 +
124 +// verifyJobGeneration checks if the job still has the expected generation
125 +// Returns false if the job was replaced OR stopped during request processing
126 +func (r *moduleFuncRegistry) verifyJobGeneration(moduleName, jobName string, expectedGen uint64) bool {
127 + r.mu.RLock()
128 + defer r.mu.RUnlock()
129 +
130 + mf, ok := r.modules[moduleName]
131 + if !ok {
132 + return false
133 + }
134 + entry, ok := mf.jobs[jobName]
135 + if !ok {
136 + return false // Job was removed
137 + }
138 +
139 + // CRITICAL: Also check if job is still running
140 + // The job could be stopped (but not yet removed) during our request
141 + // This catches the race where generation matches but job.Stop() was called
142 + if !entry.job.IsRunning() {
143 + return false // Job was stopped
144 + }
145 +
146 + return entry.generation == expectedGen
147 +}
148 +
149 +// getMethods returns the method configurations for a module
150 +func (r *moduleFuncRegistry) getMethods(moduleName string) []module.MethodConfig {
151 + r.mu.RLock()
152 + defer r.mu.RUnlock()
153 +
154 + mf, ok := r.modules[moduleName]
155 + if !ok {
156 + return nil
157 + }
158 + return mf.methods
159 +}
160 +
161 +// getMethod returns a method config by ID for a module.
162 +func (r *moduleFuncRegistry) getMethod(moduleName, methodID string) (*module.MethodConfig, bool) {
163 + r.mu.RLock()
164 + defer r.mu.RUnlock()
165 +
166 + mf, ok := r.modules[moduleName]
167 + if !ok || mf.methodsByID == nil {
168 + return nil, false
169 + }
170 + cfg, ok := mf.methodsByID[methodID]
171 + if !ok {
172 + return nil, false
173 + }
174 + return &cfg, true
175 +}
176 +
177 +// getJobNames returns job names in STABLE alphabetical order
178 +// This ensures consistent UI presentation across refreshes
179 +func (r *moduleFuncRegistry) getJobNames(moduleName string) []string {
180 + r.mu.RLock()
181 + defer r.mu.RUnlock()
182 +
183 + mf, ok := r.modules[moduleName]
184 + if !ok {
185 + return nil
186 + }
187 +
188 + // Extract job names and sort for stable ordering
189 + names := make([]string, 0, len(mf.jobs))
190 + for name := range mf.jobs {
191 + names = append(names, name)
192 + }
193 + sort.Strings(names) // Alphabetical order for consistent UI
194 + return names
195 +}
196 +
197 +// getJob returns the job by name for routing requests
198 +func (r *moduleFuncRegistry) getJob(moduleName, jobName string) (*module.Job, bool) {
199 + r.mu.RLock()
200 + defer r.mu.RUnlock()
201 +
202 + mf, ok := r.modules[moduleName]
203 + if !ok {
204 + return nil, false
205 + }
206 + entry, ok := mf.jobs[jobName]
207 + if !ok {
208 + return nil, false
209 + }
210 + return entry.job, true
211 +}
212 +
213 +// getCreator returns the module creator for a registered module
214 +func (r *moduleFuncRegistry) getCreator(moduleName string) (module.Creator, bool) {
215 + r.mu.RLock()
216 + defer r.mu.RUnlock()
217 +
218 + mf, ok := r.modules[moduleName]
219 + if !ok {
220 + return module.Creator{}, false
221 + }
222 + return mf.creator, true
223 +}
224 +
225 +// isModuleRegistered checks if a module is registered (implements FunctionProvider)
226 +func (r *moduleFuncRegistry) isModuleRegistered(moduleName string) bool {
227 + r.mu.RLock()
228 + defer r.mu.RUnlock()
229 +
230 + _, ok := r.modules[moduleName]
231 + return ok
232 +}
src/go/plugin/go.d/agent/jobmgr/modulefuncs_test.go new
+269
@@ -0,0 +1,269 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package jobmgr
4 +
5 +import (
6 + "context"
7 + "io"
8 + "testing"
9 +
10 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
11 + "github.com/stretchr/testify/assert"
12 + "github.com/stretchr/testify/require"
13 +)
14 +
15 +func TestModuleFuncRegistry_RegisterModule(t *testing.T) {
16 + tests := map[string]struct {
17 + modules []string
18 + expected []string
19 + }{
20 + "single module": {
21 + modules: []string{"postgres"},
22 + expected: []string{"postgres"},
23 + },
24 + "multiple modules": {
25 + modules: []string{"postgres", "mysql", "mssql"},
26 + expected: []string{"mysql", "mssql", "postgres"}, // sorted
27 + },
28 + "duplicate registration overwrites": {
29 + modules: []string{"postgres", "postgres"},
30 + expected: []string{"postgres"},
31 + },
32 + }
33 +
34 + for name, tc := range tests {
35 + t.Run(name, func(t *testing.T) {
36 + r := newModuleFuncRegistry()
37 +
38 + for _, m := range tc.modules {
39 + r.registerModule(m, module.Creator{
40 + Methods: func() []module.MethodConfig {
41 + return []module.MethodConfig{{ID: "test"}}
42 + },
43 + })
44 + }
45 +
46 + assert.Equal(t, len(tc.expected), len(r.modules))
47 + for _, m := range tc.expected {
48 + assert.True(t, r.isModuleRegistered(m))
49 + }
50 + })
51 + }
52 +}
53 +
54 +func TestModuleFuncRegistry_AddRemoveJob(t *testing.T) {
55 + r := newModuleFuncRegistry()
56 + r.registerModule("postgres", module.Creator{})
57 +
58 + // Create test jobs
59 + job1 := newTestModuleFuncsJob("job1")
60 + job2 := newTestModuleFuncsJob("job2")
61 +
62 + // Add jobs
63 + r.addJob("postgres", "job1", job1)
64 + r.addJob("postgres", "job2", job2)
65 +
66 + // Verify jobs are retrievable
67 + names := r.getJobNames("postgres")
68 + assert.ElementsMatch(t, []string{"job1", "job2"}, names)
69 +
70 + got1, ok := r.getJob("postgres", "job1")
71 + assert.True(t, ok)
72 + assert.Equal(t, job1, got1)
73 +
74 + // Remove job
75 + r.removeJob("postgres", "job1")
76 +
77 + names = r.getJobNames("postgres")
78 + assert.ElementsMatch(t, []string{"job2"}, names)
79 +
80 + _, ok = r.getJob("postgres", "job1")
81 + assert.False(t, ok)
82 +}
83 +
84 +func TestModuleFuncRegistry_JobReplacement(t *testing.T) {
85 + r := newModuleFuncRegistry()
86 + r.registerModule("postgres", module.Creator{})
87 +
88 + job1 := newTestModuleFuncsJob("master")
89 + job2 := newTestModuleFuncsJob("master") // Same name, different instance
90 +
91 + // Add first job
92 + r.addJob("postgres", "master", job1)
93 + _, gen1 := r.getJobWithGeneration("postgres", "master")
94 + assert.Equal(t, uint64(1), gen1)
95 +
96 + // Replace with second job
97 + r.addJob("postgres", "master", job2)
98 + got, gen2 := r.getJobWithGeneration("postgres", "master")
99 + assert.Equal(t, uint64(2), gen2) // Generation incremented
100 + assert.Equal(t, job2, got) // New job returned
101 +}
102 +
103 +func TestModuleFuncRegistry_GenerationVerification(t *testing.T) {
104 + r := newModuleFuncRegistry()
105 + r.registerModule("postgres", module.Creator{})
106 +
107 + job := newTestModuleFuncsJob("master")
108 +
109 + r.addJob("postgres", "master", job)
110 + _, gen := r.getJobWithGeneration("postgres", "master")
111 +
112 + // Note: verifyJobGeneration checks BOTH generation AND IsRunning()
113 + // Since our test job isn't running, verification should fail
114 + // This is actually correct behavior - it catches stopped jobs
115 +
116 + // Verify with wrong generation - should fail
117 + assert.False(t, r.verifyJobGeneration("postgres", "master", gen+1))
118 +
119 + // Remove job and verify - should fail
120 + r.removeJob("postgres", "master")
121 + assert.False(t, r.verifyJobGeneration("postgres", "master", gen))
122 +}
123 +
124 +func TestModuleFuncRegistry_GetMethods(t *testing.T) {
125 + r := newModuleFuncRegistry()
126 +
127 + expectedMethods := []module.MethodConfig{
128 + {ID: "top-queries", Name: "Top Queries"},
129 + }
130 +
131 + r.registerModule("postgres", module.Creator{
132 + Methods: func() []module.MethodConfig {
133 + return expectedMethods
134 + },
135 + })
136 +
137 + methods := r.getMethods("postgres")
138 + assert.Equal(t, expectedMethods, methods)
139 +
140 + // Non-existent module
141 + assert.Nil(t, r.getMethods("nonexistent"))
142 +}
143 +
144 +func TestModuleFuncRegistry_GetJobNames_Sorted(t *testing.T) {
145 + r := newModuleFuncRegistry()
146 + r.registerModule("postgres", module.Creator{})
147 +
148 + // Add jobs in random order
149 + r.addJob("postgres", "zebra-db", newTestModuleFuncsJob("zebra"))
150 + r.addJob("postgres", "alpha-db", newTestModuleFuncsJob("alpha"))
151 + r.addJob("postgres", "middle-db", newTestModuleFuncsJob("middle"))
152 +
153 + names := r.getJobNames("postgres")
154 +
155 + // Should be sorted alphabetically
156 + assert.Equal(t, []string{"alpha-db", "middle-db", "zebra-db"}, names)
157 +}
158 +
159 +func TestModuleFuncRegistry_UnregisteredModule(t *testing.T) {
160 + r := newModuleFuncRegistry()
161 +
162 + // Operations on unregistered module should be no-ops
163 + r.addJob("nonexistent", "job1", newTestModuleFuncsJob("job1"))
164 + r.removeJob("nonexistent", "job1")
165 +
166 + assert.False(t, r.isModuleRegistered("nonexistent"))
167 + assert.Nil(t, r.getJobNames("nonexistent"))
168 + assert.Nil(t, r.getMethods("nonexistent"))
169 +
170 + _, ok := r.getJob("nonexistent", "job1")
171 + assert.False(t, ok)
172 +}
173 +
174 +func TestModuleFuncRegistry_GetCreator(t *testing.T) {
175 + r := newModuleFuncRegistry()
176 +
177 + creator := module.Creator{
178 + JobConfigSchema: "test-schema",
179 + }
180 + r.registerModule("postgres", creator)
181 +
182 + got, ok := r.getCreator("postgres")
183 + require.True(t, ok)
184 + assert.Equal(t, "test-schema", got.JobConfigSchema)
185 +
186 + // Non-existent module
187 + _, ok = r.getCreator("nonexistent")
188 + assert.False(t, ok)
189 +}
190 +
191 +// newTestModuleFuncsJob creates a minimal job for testing modulefuncs
192 +func newTestModuleFuncsJob(name string) *module.Job {
193 + return module.NewJob(module.JobConfig{
194 + PluginName: "test",
195 + Name: name,
196 + ModuleName: "test",
197 + FullName: "test_" + name,
198 + Module: &module.MockModule{},
199 + Out: io.Discard,
200 + UpdateEvery: 1,
201 + AutoDetectEvery: 0,
202 + Priority: 1000,
203 + })
204 +}
205 +
206 +// TestModuleFuncRegistry_Concurrency tests thread safety
207 +func TestModuleFuncRegistry_Concurrency(t *testing.T) {
208 + r := newModuleFuncRegistry()
209 + r.registerModule("postgres", module.Creator{
210 + Methods: func() []module.MethodConfig {
211 + return []module.MethodConfig{{ID: "test"}}
212 + },
213 + })
214 +
215 + done := make(chan bool)
216 +
217 + // Writer goroutine
218 + go func() {
219 + for i := 0; i < 100; i++ {
220 + job := newTestModuleFuncsJob("job")
221 + r.addJob("postgres", "job", job)
222 + r.removeJob("postgres", "job")
223 + }
224 + done <- true
225 + }()
226 +
227 + // Reader goroutine
228 + go func() {
229 + for i := 0; i < 100; i++ {
230 + _ = r.getJobNames("postgres")
231 + _ = r.getMethods("postgres")
232 + _, _ = r.getJob("postgres", "job")
233 + }
234 + done <- true
235 + }()
236 +
237 + <-done
238 + <-done
239 +}
240 +
241 +// TestModuleFuncRegistry_VerifyJobGeneration_JobStopped tests race detection when job stops
242 +func TestModuleFuncRegistry_VerifyJobGeneration_JobStopped(t *testing.T) {
243 + r := newModuleFuncRegistry()
244 + r.registerModule("postgres", module.Creator{})
245 +
246 + job := module.NewJob(module.JobConfig{
247 + PluginName: "test",
248 + Name: "master",
249 + ModuleName: "postgres",
250 + FullName: "postgres_master",
251 + Module: &module.MockModule{
252 + InitFunc: func(context.Context) error { return nil },
253 + CheckFunc: func(context.Context) error { return nil },
254 + ChartsFunc: func() *module.Charts { return &module.Charts{} },
255 + CollectFunc: func(context.Context) map[string]int64 { return nil },
256 + },
257 + Out: io.Discard,
258 + UpdateEvery: 1,
259 + AutoDetectEvery: 0,
260 + Priority: 1000,
261 + })
262 +
263 + r.addJob("postgres", "master", job)
264 + _, gen := r.getJobWithGeneration("postgres", "master")
265 +
266 + // Job exists but is not running - verification should fail
267 + // (job.IsRunning() returns false because job hasn't started)
268 + assert.False(t, r.verifyJobGeneration("postgres", "master", gen))
269 +}
src/go/plugin/go.d/agent/jobmgr/noop.go
+2
@@ -14,5 +14,7 @@ func (n noop) Lock(string) (bool, error)
14 func (n noop) Unlock(string) {}
15 func (n noop) UnlockAll() {}
16 func (n noop) Lookup(string) (*vnodes.VirtualNode, bool) { return nil, false }
17 +func (n noop) Register(name string, fn func(functions.Function)) {}
18 +func (n noop) Unregister(name string) {}
19 func (n noop) RegisterPrefix(name, prefix string, reg func(functions.Function)) {}
20 func (n noop) UnregisterPrefix(name, prefix string) {}
src/go/plugin/go.d/agent/module/job.go
+21 -1
@@ -12,6 +12,7 @@ import (
12 "runtime/debug"
13 "strings"
14 "sync"
15 + "sync/atomic"
16 "time"
17
18 "github.com/netdata/netdata/go/plugins/logger"
@@ -152,6 +153,9 @@ type Job struct {
153
154 module Module
155
156 + // running tracks whether the job's main loop is active (set in Start, cleared in Start's defer)
157 + running atomic.Bool
158 +
159 initialized bool
160 panicked bool
161
@@ -323,10 +327,26 @@ func (j *Job) Tick(clock int) {
327 }
328 }
329
330 +// IsRunning returns true if the job's main loop is currently running.
331 +// This is safe to call from any goroutine.
332 +func (j *Job) IsRunning() bool {
333 + return j.running.Load()
334 +}
335 +
336 +// Module returns the underlying module instance.
337 +// This allows function handlers to access the collector for querying data.
338 +func (j *Job) Module() Module {
339 + return j.module
340 +}
341 +
342 // Start starts job main loop.
343 func (j *Job) Start() {
344 + j.running.Store(true)
345 j.Infof("started, data collection interval %ds", j.updateEvery)
329 - defer func() { j.Info("stopped") }()
346 + defer func() {
347 + j.running.Store(false)
348 + j.Info("stopped")
349 + }()
350
351 LOOP:
352 for {
src/go/plugin/go.d/agent/module/registry.go
+64 -1
@@ -2,7 +2,12 @@
2
3 package module
4
5 -import "fmt"
5 +import (
6 + "context"
7 + "fmt"
8 +
9 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
10 +)
11
12 const (
13 UpdateEvery = 1
@@ -18,13 +23,71 @@ type Defaults struct {
23 Disabled bool
24 }
25
26 +// MethodConfig describes a function method provided by a module.
27 +type MethodConfig struct {
28 + ID string // Method ID (e.g., "top-queries")
29 + Name string // Display name (e.g., "Top Queries")
30 + Help string // Description for UI
31 + RequiredParams []funcapi.ParamConfig // Required parameters for this method (including __sort if used)
32 +}
33 +
34 +// FunctionResponse is the response from a module's HandleMethod.
35 +type FunctionResponse struct {
36 + Status int // HTTP-like status code (200, 400, 403, 500, 503)
37 + Message string // Error message (if Status != 200)
38 + Help string // Help text for this response
39 + Columns map[string]any // Column definitions for the table
40 + Data any // Row data: [][]any (array of arrays, ordered by column index)
41 + DefaultSortColumn string // Default sort column ID
42 +
43 + // Optional dynamic required params (override MethodConfig.RequiredParams)
44 + RequiredParams []funcapi.ParamConfig
45 +
46 + // Chart configuration for visualization
47 + Charts map[string]ChartConfig // Chart definitions (chartID -> config)
48 + DefaultCharts [][]string // Default charts: [[chartID, groupByID], ...]
49 + GroupBy map[string]GroupByConfig // Group-by options (groupByID -> config)
50 +}
51 +
52 +// ChartConfig defines a chart for visualization.
53 +type ChartConfig struct {
54 + Name string `json:"name"`
55 + Type string `json:"type"` // "stacked-bar", "line", etc.
56 + Columns []string `json:"columns"` // Column IDs to include in chart
57 +}
58 +
59 +// GroupByConfig defines a grouping option for function responses.
60 +type GroupByConfig struct {
61 + Name string `json:"name"`
62 + Columns []string `json:"columns"` // Columns to group by
63 +}
64 +
65 type (
66 // Creator is a Job builder.
67 + // Optional function fields (Methods/HandleMethod) enable the FunctionProvider pattern:
68 + // modules that set these fields can expose data functions to the UI.
69 Creator struct {
70 Defaults
71 Create func() Module
72 JobConfigSchema string
73 Config func() any
74 +
75 + // Optional: FunctionProvider fields for exposing data functions
76 + // If Methods is non-nil, this module provides functions
77 + Methods func() []MethodConfig
78 +
79 + // Optional: MethodParams returns dynamic required params for a job+method.
80 + // Use this to provide job-specific options (e.g., based on DB capabilities).
81 + // When nil, MethodConfig.RequiredParams is used as-is.
82 + MethodParams func(ctx context.Context, job *Job, method string) ([]funcapi.ParamConfig, error)
83 +
84 + // HandleMethod handles a function request for a specific job
85 + // ctx: context with timeout from function request
86 + // job: the job instance to query
87 + // method: the method name (e.g., "top-queries")
88 + // params: resolved required params (includes __sort)
89 + // Returns: FunctionResponse with data or error
90 + HandleMethod func(ctx context.Context, job *Job, method string, params funcapi.ResolvedParams) *FunctionResponse
91 }
92 // Registry is a collection of Creators.
93 Registry map[string]Creator
src/go/plugin/go.d/collector/mongodb/collector.go
+24 -6
@@ -23,6 +23,9 @@ func init() {
23 JobConfigSchema: configSchema,
24 Create: func() module.Module { return New() },
25 Config: func() any { return &Config{} },
26 + Methods: mongoMethods,
27 + MethodParams: mongoMethodParams,
28 + HandleMethod: mongoHandleMethod,
29 })
30 }
31
@@ -50,12 +53,23 @@ func New() *Collector {
53 }
54
55 type Config struct {
53 - Vnode string `yaml:"vnode,omitempty" json:"vnode"`
54 - UpdateEvery int `yaml:"update_every,omitempty" json:"update_every"`
55 - AutoDetectionRetry int `yaml:"autodetection_retry,omitempty" json:"autodetection_retry"`
56 - URI string `yaml:"uri" json:"uri"`
57 - Timeout confopt.Duration `yaml:"timeout,omitempty" json:"timeout"`
58 - Databases matcher.SimpleExpr `yaml:"databases,omitempty" json:"databases"`
56 + Vnode string `yaml:"vnode,omitempty" json:"vnode"`
57 + UpdateEvery int `yaml:"update_every,omitempty" json:"update_every"`
58 + AutoDetectionRetry int `yaml:"autodetection_retry,omitempty" json:"autodetection_retry"`
59 + URI string `yaml:"uri" json:"uri"`
60 + Timeout confopt.Duration `yaml:"timeout,omitempty" json:"timeout"`
61 + Databases matcher.SimpleExpr `yaml:"databases,omitempty" json:"databases"`
62 + TopQueriesFunctionEnabled *bool `yaml:"top_queries_function_enabled,omitempty" json:"top_queries_function_enabled,omitempty"`
63 + TopQueriesLimit int `yaml:"top_queries_limit,omitempty" json:"top_queries_limit,omitempty"`
64 +}
65 +
66 +// GetTopQueriesFunctionEnabled returns whether the top queries function is enabled.
67 +// Defaults to true if not explicitly configured.
68 +func (c *Config) GetTopQueriesFunctionEnabled() bool {
69 + if c.TopQueriesFunctionEnabled == nil {
70 + return true
71 + }
72 + return *c.TopQueriesFunctionEnabled
73 }
74
75 type Collector struct {
@@ -72,6 +86,10 @@ type Collector struct {
86 databases map[string]bool
87 replSetMembers map[string]bool
88 shards map[string]bool
89 +
90 + // Top queries column cache with double-checked locking
91 + topQueriesColsMu sync.RWMutex
92 + topQueriesCols map[string]bool
93 }
94
95 func (c *Collector) Configuration() any {
src/go/plugin/go.d/collector/mongodb/config_schema.json
+27
@@ -67,6 +67,20 @@
67 }
68 }
69 },
70 + "top_queries_function_enabled": {
71 + "title": "Enable Top Queries function",
72 + "description": "Enables or disables the Top Queries function that exposes slow queries from MongoDB Profiler (system.profile). **WARNING**: Query text may contain unmasked literals (potential PII). Requires profiling to be enabled on target databases.",
73 + "type": "boolean",
74 + "default": true
75 + },
76 + "top_queries_limit": {
77 + "title": "Top Queries limit",
78 + "description": "Maximum number of queries to return from the Top Queries function. Set to 0 to use the default limit (500).",
79 + "type": "integer",
80 + "minimum": 0,
81 + "maximum": 10000,
82 + "default": 500
83 + },
84 "vnode": {
85 "title": "Vnode",
86 "description": "Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes).",
@@ -96,6 +110,12 @@
110 "databases": {
111 "ui:help": "The logic for inclusion and exclusion is as follows: `(include1 OR include2) AND !(exclude1 OR exclude2)`."
112 },
113 + "top_queries_function_enabled": {
114 + "ui:help": "When enabled, the Top Queries function allows you to view slow queries from MongoDB's system.profile collection. Note: Query text may contain unmasked literals which could include sensitive information (PII)."
115 + },
116 + "top_queries_limit": {
117 + "ui:help": "Controls how many queries are returned. Higher values provide more data but may impact performance."
118 + },
119 "ui:flavour": "tabs",
120 "ui:options": {
121 "tabs": [
@@ -114,6 +134,13 @@
134 "fields": [
135 "databases"
136 ]
137 + },
138 + {
139 + "title": "Top Queries",
140 + "fields": [
141 + "top_queries_function_enabled",
142 + "top_queries_limit"
143 + ]
144 }
145 ]
146 }
src/go/plugin/go.d/collector/mongodb/functions.go new
+781
@@ -0,0 +1,781 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package mongo
4 +
5 +import (
6 + "context"
7 + "encoding/json"
8 + "errors"
9 + "fmt"
10 + "sort"
11 + "time"
12 +
13 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
14 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
15 + "github.com/netdata/netdata/go/plugins/plugin/go.d/pkg/strmutil"
16 +
17 + "go.mongodb.org/mongo-driver/bson"
18 + "go.mongodb.org/mongo-driver/mongo/options"
19 +)
20 +
21 +const (
22 + maxQueryTextLength = 4096
23 + defaultTopQueriesLimit = 500
24 + topQueriesHelpText = "Top queries from MongoDB Profiler (system.profile). " +
25 + "WARNING: Query text may contain unmasked literals (potential PII). " +
26 + "Requires profiling enabled on target databases (db.setProfilingLevel)."
27 +)
28 +
29 +const (
30 + paramSort = "__sort"
31 +
32 + ftString = funcapi.FieldTypeString
33 + ftInteger = funcapi.FieldTypeInteger
34 + ftDuration = funcapi.FieldTypeDuration
35 + ftTimestamp = funcapi.FieldTypeTimestamp
36 +
37 + trNumber = funcapi.FieldTransformNumber
38 + trDuration = funcapi.FieldTransformDuration
39 + trDatetime = funcapi.FieldTransformDatetime
40 + trText = funcapi.FieldTransformText
41 +
42 + visValue = funcapi.FieldVisualValue
43 + visBar = funcapi.FieldVisualBar
44 +
45 + summarySum = funcapi.FieldSummarySum
46 + summaryMax = funcapi.FieldSummaryMax
47 +
48 + filterRange = funcapi.FieldFilterRange
49 + filterMulti = funcapi.FieldFilterMultiselect
50 +)
51 +
52 +// mongoColumnMeta defines metadata for a column in the response
53 +type mongoColumnMeta struct {
54 + id string // column ID in response (e.g., "execution_time")
55 + dbField string // MongoDB document field name (e.g., "millis")
56 + name string // display name (e.g., "Execution Time")
57 + colType funcapi.FieldType // column type: integer, duration, timestamp, string, bool
58 + visible bool // default visibility
59 + sortable bool // can be used for sorting
60 + fullWidth bool // for query text columns
61 + wrap bool // wrap text
62 + sticky bool // sticky column
63 + filter funcapi.FieldFilter // filter type: range, multiselect, text
64 + visualization funcapi.FieldVisual // visualization type: value, bar
65 + summary funcapi.FieldSummary // summary type: sum, max, or empty
66 + transform funcapi.FieldTransform // value transform: number, duration, datetime, text
67 + units string // display units (e.g., "seconds")
68 + decimalPoints int // decimal points for numeric display
69 + uniqueKey bool // unique key column
70 + expandFilter bool // default expanded filter
71 +}
72 +
73 +// mongoAllColumns defines all available columns from system.profile
74 +// Ordered by display priority (index)
75 +var mongoAllColumns = []mongoColumnMeta{
76 + // Core fields (visible by default)
77 + {id: "timestamp", dbField: "ts", name: "Timestamp", colType: ftTimestamp, visible: true, sortable: true, filter: filterRange, visualization: visValue, summary: summaryMax, transform: trDatetime, uniqueKey: true},
78 + {id: "namespace", dbField: "ns", name: "Namespace", colType: ftString, visible: true, sortable: false, sticky: true, filter: filterMulti, visualization: visValue, transform: trText, expandFilter: true},
79 + {id: "operation", dbField: "op", name: "Operation", colType: ftString, visible: true, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
80 + {id: "query", dbField: "command", name: "Query", colType: ftString, visible: true, sortable: false, fullWidth: true, wrap: true, filter: filterMulti, visualization: visValue, transform: trText},
81 + {id: "execution_time", dbField: "millis", name: "Execution Time", colType: ftDuration, visible: true, sortable: true, filter: filterRange, visualization: visBar, summary: summarySum, transform: trDuration, units: "seconds", decimalPoints: 3},
82 + {id: "docs_examined", dbField: "docsExamined", name: "Docs Examined", colType: ftInteger, visible: true, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
83 + {id: "keys_examined", dbField: "keysExamined", name: "Keys Examined", colType: ftInteger, visible: true, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
84 + {id: "docs_returned", dbField: "nreturned", name: "Docs Returned", colType: ftInteger, visible: true, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
85 + {id: "plan_summary", dbField: "planSummary", name: "Plan Summary", colType: ftString, visible: true, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
86 +
87 + // Secondary fields
88 + {id: "client", dbField: "client", name: "Client", colType: ftString, visible: true, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
89 + {id: "user", dbField: "user", name: "User", colType: ftString, visible: true, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
90 + {id: "docs_deleted", dbField: "ndeleted", name: "Docs Deleted", colType: ftInteger, visible: false, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
91 + {id: "docs_inserted", dbField: "ninserted", name: "Docs Inserted", colType: ftInteger, visible: false, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
92 + {id: "docs_modified", dbField: "nModified", name: "Docs Modified", colType: ftInteger, visible: false, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
93 + {id: "response_length", dbField: "responseLength", name: "Response Length", colType: ftInteger, visible: false, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
94 + {id: "num_yield", dbField: "numYield", name: "Num Yield", colType: ftInteger, visible: false, sortable: true, filter: filterRange, visualization: visValue, summary: summarySum, transform: trNumber},
95 + {id: "app_name", dbField: "appName", name: "App Name", colType: ftString, visible: true, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
96 + {id: "cursor_exhausted", dbField: "cursorExhausted", name: "Cursor Exhausted", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
97 + {id: "has_sort_stage", dbField: "hasSortStage", name: "Has Sort Stage", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
98 + {id: "uses_disk", dbField: "usedDisk", name: "Uses Disk", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
99 + {id: "from_multi_planner", dbField: "fromMultiPlanner", name: "From Multi Planner", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
100 + {id: "replanned", dbField: "replanned", name: "Replanned", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText},
101 +
102 + // Version-specific fields (hidden by default)
103 + {id: "query_hash", dbField: "queryHash", name: "Query Hash", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText}, // 4.2+
104 + {id: "plan_cache_key", dbField: "planCacheKey", name: "Plan Cache Key", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText}, // 4.2+
105 + {id: "planning_time", dbField: "planningTimeMicros", name: "Planning Time", colType: ftDuration, visible: false, sortable: true, filter: filterRange, visualization: visBar, summary: summarySum, transform: trDuration, units: "seconds", decimalPoints: 3},
106 + {id: "cpu_time", dbField: "cpuNanos", name: "CPU Time", colType: ftDuration, visible: false, sortable: true, filter: filterRange, visualization: visBar, summary: summarySum, transform: trDuration, units: "seconds", decimalPoints: 3},
107 + {id: "query_framework", dbField: "queryFramework", name: "Query Framework", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText}, // 7.0+
108 + {id: "query_shape_hash", dbField: "queryShapeHash", name: "Query Shape Hash", colType: ftString, visible: false, sortable: false, filter: filterMulti, visualization: visValue, transform: trText}, // 8.0+
109 +}
110 +
111 +// optionalDuration converts an optional int64 pointer to float64 seconds, returning nil if nil
112 +func optionalDuration(v *int64, divisor float64) any {
113 + if v == nil {
114 + return nil
115 + }
116 + return float64(*v) / divisor
117 +}
118 +
119 +// optionalBool converts a bool pointer to a display value
120 +func optionalBool(v *bool) any {
121 + if v == nil {
122 + return nil
123 + }
124 + if *v {
125 + return "Yes"
126 + }
127 + return "No"
128 +}
129 +
130 +// topQueriesCharts returns the chart configuration for top queries responses
131 +func topQueriesCharts() map[string]module.ChartConfig {
132 + return map[string]module.ChartConfig{
133 + "Time": {
134 + Name: "Execution Time",
135 + Type: "stacked-bar",
136 + Columns: []string{"execution_time"},
137 + },
138 + "DocsExamined": {
139 + Name: "Documents & Keys Examined",
140 + Type: "stacked-bar",
141 + Columns: []string{"docs_examined", "keys_examined"},
142 + },
143 + "DocsReturned": {
144 + Name: "Documents Returned",
145 + Type: "stacked-bar",
146 + Columns: []string{"docs_returned"},
147 + },
148 + }
149 +}
150 +
151 +// topQueriesDefaultCharts returns the default chart configuration
152 +func topQueriesDefaultCharts() [][]string {
153 + return [][]string{
154 + {"Time", "namespace"},
155 + {"DocsExamined", "namespace"},
156 + }
157 +}
158 +
159 +// topQueriesGroupBy returns the group by configuration for top queries responses
160 +func topQueriesGroupBy() map[string]module.GroupByConfig {
161 + return map[string]module.GroupByConfig{
162 + "namespace": {
163 + Name: "Group by Namespace",
164 + Columns: []string{"namespace"},
165 + },
166 + "operation": {
167 + Name: "Group by Operation Type",
168 + Columns: []string{"operation"},
169 + },
170 + }
171 +}
172 +
173 +func buildMongoSortParam(cols []mongoColumnMeta) funcapi.ParamConfig {
174 + var sortOptions []funcapi.ParamOption
175 + sortDir := funcapi.FieldSortDescending
176 + for _, col := range cols {
177 + if !col.sortable {
178 + continue
179 + }
180 + opt := funcapi.ParamOption{
181 + ID: col.id,
182 + Column: col.dbField,
183 + Name: fmt.Sprintf("Top queries by %s", col.name),
184 + Sort: &sortDir,
185 + }
186 + if col.id == "execution_time" {
187 + opt.Default = true
188 + }
189 + sortOptions = append(sortOptions, opt)
190 + }
191 +
192 + return funcapi.ParamConfig{
193 + ID: paramSort,
194 + Name: "Filter By",
195 + Help: "Select the primary sort column",
196 + Selection: funcapi.ParamSelect,
197 + Options: sortOptions,
198 + UniqueView: true,
199 + }
200 +}
201 +
202 +// mongoMethods returns the available function methods for MongoDB
203 +func mongoMethods() []module.MethodConfig {
204 + // Build sort options from column metadata
205 + var sortOptions []funcapi.ParamOption
206 + sortDir := funcapi.FieldSortDescending
207 + for _, col := range mongoAllColumns {
208 + if !col.sortable {
209 + continue
210 + }
211 + opt := funcapi.ParamOption{
212 + ID: col.id,
213 + Column: col.dbField,
214 + Name: fmt.Sprintf("Top queries by %s", col.name),
215 + Sort: &sortDir,
216 + }
217 + if col.id == "execution_time" {
218 + opt.Default = true
219 + }
220 + sortOptions = append(sortOptions, opt)
221 + }
222 +
223 + return []module.MethodConfig{{
224 + ID: "top-queries",
225 + Name: "Top Queries",
226 + Help: topQueriesHelpText,
227 + RequiredParams: []funcapi.ParamConfig{
228 + {
229 + ID: paramSort,
230 + Name: "Filter By",
231 + Help: "Select the primary sort column",
232 + Selection: funcapi.ParamSelect,
233 + Options: sortOptions,
234 + UniqueView: true,
235 + },
236 + },
237 + }}
238 +}
239 +
240 +func mongoMethodParams(ctx context.Context, job *module.Job, method string) ([]funcapi.ParamConfig, error) {
241 + collector, ok := job.Module().(*Collector)
242 + if !ok {
243 + return nil, fmt.Errorf("invalid module type")
244 + }
245 + if collector.conn == nil {
246 + return nil, fmt.Errorf("collector is still initializing")
247 + }
248 + switch method {
249 + case "top-queries":
250 + return collector.topQueriesParams(ctx)
251 + default:
252 + return nil, fmt.Errorf("unknown method: %s", method)
253 + }
254 +}
255 +
256 +// mongoHandleMethod handles function requests for MongoDB
257 +func mongoHandleMethod(ctx context.Context, job *module.Job, method string, params funcapi.ResolvedParams) *module.FunctionResponse {
258 + collector, ok := job.Module().(*Collector)
259 + if !ok {
260 + return &module.FunctionResponse{Status: 500, Message: "internal error: invalid module type"}
261 + }
262 +
263 + // Check if collector is initialized
264 + if collector.conn == nil {
265 + return &module.FunctionResponse{
266 + Status: 503,
267 + Message: "collector is still initializing, please retry in a few seconds",
268 + }
269 + }
270 +
271 + switch method {
272 + case "top-queries":
273 + // Check if function is enabled
274 + if !collector.Config.GetTopQueriesFunctionEnabled() {
275 + return &module.FunctionResponse{
276 + Status: 403,
277 + Message: "Top Queries function has been disabled in configuration. Set 'top_queries_function_enabled: true' to enable.",
278 + }
279 + }
280 + return collector.collectTopQueries(ctx, params.Column(paramSort))
281 + default:
282 + return &module.FunctionResponse{Status: 404, Message: fmt.Sprintf("unknown method: %s", method)}
283 + }
284 +}
285 +
286 +// profileDocument represents a document from system.profile
287 +type profileDocument struct {
288 + // Core fields (always present)
289 + Timestamp time.Time `bson:"ts"`
290 + Op string `bson:"op"`
291 + Ns string `bson:"ns"`
292 + Command bson.M `bson:"command"`
293 + Millis int64 `bson:"millis"`
294 + PlanSummary string `bson:"planSummary"`
295 +
296 + // Common fields
297 + DocsExamined int64 `bson:"docsExamined"`
298 + KeysExamined int64 `bson:"keysExamined"`
299 + Nreturned int64 `bson:"nreturned"`
300 + Client string `bson:"client"`
301 + User string `bson:"user"`
302 + Ndeleted int64 `bson:"ndeleted"`
303 + Ninserted int64 `bson:"ninserted"`
304 + NModified int64 `bson:"nModified"`
305 + ResponseLength int64 `bson:"responseLength"`
306 + NumYield int64 `bson:"numYield"`
307 + AppName string `bson:"appName"`
308 +
309 + // Boolean fields (pointers for nil detection)
310 + CursorExhausted *bool `bson:"cursorExhausted"`
311 + HasSortStage *bool `bson:"hasSortStage"`
312 + UsedDisk *bool `bson:"usedDisk"`
313 + FromMultiPlanner *bool `bson:"fromMultiPlanner"`
314 + Replanned *bool `bson:"replanned"`
315 +
316 + // Version-specific fields
317 + QueryHash string `bson:"queryHash"` // 4.2+
318 + PlanCacheKey string `bson:"planCacheKey"` // 4.2+
319 + PlanningTimeMicros *int64 `bson:"planningTimeMicros"` // 6.2+
320 + CpuNanos *int64 `bson:"cpuNanos"` // 6.3+ Linux only
321 + QueryFramework string `bson:"queryFramework"` // 7.0+
322 + QueryShapeHash string `bson:"queryShapeHash"` // 8.0+
323 +}
324 +
325 +// detectMongoProfileFields detects available fields in system.profile using double-checked locking
326 +func (c *Collector) detectMongoProfileFields(ctx context.Context, databases []string) (map[string]bool, error) {
327 + // Fast path: return cached
328 + c.topQueriesColsMu.RLock()
329 + if c.topQueriesCols != nil {
330 + cols := c.topQueriesCols
331 + c.topQueriesColsMu.RUnlock()
332 + return cols, nil
333 + }
334 + c.topQueriesColsMu.RUnlock()
335 +
336 + // Slow path: detect and cache
337 + c.topQueriesColsMu.Lock()
338 + defer c.topQueriesColsMu.Unlock()
339 +
340 + // Double-check after acquiring write lock
341 + if c.topQueriesCols != nil {
342 + return c.topQueriesCols, nil
343 + }
344 +
345 + client, ok := c.conn.(*mongoClient)
346 + if !ok || client == nil || client.client == nil {
347 + return nil, fmt.Errorf("client not initialized")
348 + }
349 +
350 + available := make(map[string]bool)
351 +
352 + // Always include core fields that are guaranteed to exist
353 + coreFields := []string{"ts", "op", "ns", "command", "millis"}
354 + for _, f := range coreFields {
355 + available[f] = true
356 + }
357 +
358 + // Sample documents from system.profile to detect available fields
359 + for _, dbName := range databases {
360 + queryCtx, cancel := context.WithTimeout(ctx, client.timeout)
361 +
362 + collection := client.client.Database(dbName).Collection("system.profile")
363 + opts := options.FindOne().SetSort(bson.D{{Key: "$natural", Value: -1}})
364 +
365 + var doc bson.M
366 + err := collection.FindOne(queryCtx, bson.M{}, opts).Decode(&doc)
367 + cancel()
368 +
369 + if err != nil {
370 + continue // No documents or profiling disabled
371 + }
372 +
373 + // Add all fields found in this document
374 + for field := range doc {
375 + available[field] = true
376 + }
377 + }
378 +
379 + c.topQueriesCols = available
380 + return available, nil
381 +}
382 +
383 +// buildAvailableMongoColumns returns columns that are available based on detected fields
384 +func buildAvailableMongoColumns(available map[string]bool) []mongoColumnMeta {
385 + var result []mongoColumnMeta
386 + for _, col := range mongoAllColumns {
387 + // command field maps to query column, always include
388 + if col.dbField == "command" || available[col.dbField] {
389 + result = append(result, col)
390 + }
391 + }
392 + return result
393 +}
394 +
395 +// buildMongoColumnsFromMeta builds the Columns map for FunctionResponse
396 +func buildMongoColumnsFromMeta(cols []mongoColumnMeta) map[string]any {
397 + result := make(map[string]any)
398 + for i, col := range cols {
399 + sortDir := funcapi.FieldSortDescending
400 + if col.colType == ftString {
401 + sortDir = funcapi.FieldSortAscending
402 + }
403 + colDef := funcapi.Column{
404 + Index: i,
405 + Name: col.name,
406 + Type: col.colType,
407 + Units: col.units,
408 + Visualization: col.visualization,
409 + Sort: sortDir,
410 + Sortable: col.sortable,
411 + Sticky: col.sticky,
412 + Summary: col.summary,
413 + Filter: col.filter,
414 + FullWidth: col.fullWidth,
415 + Wrap: col.wrap,
416 + DefaultExpandedFilter: col.expandFilter,
417 + UniqueKey: col.uniqueKey,
418 + Visible: col.visible,
419 + ValueOptions: funcapi.ValueOptions{
420 + Transform: col.transform,
421 + DecimalPoints: col.decimalPoints,
422 + DefaultValue: nil,
423 + },
424 + }
425 +
426 + result[col.id] = colDef.BuildColumn()
427 + }
428 + return result
429 +}
430 +
431 +// collectTopQueries queries system.profile for top queries across all databases
432 +func (c *Collector) collectTopQueries(ctx context.Context, sortColumn string) *module.FunctionResponse {
433 + // Get limit from config
434 + limit := c.Config.TopQueriesLimit
435 + if limit <= 0 {
436 + limit = defaultTopQueriesLimit
437 + }
438 +
439 + // Build valid sort columns map from metadata
440 + validSortCols := make(map[string]bool)
441 + for _, col := range mongoAllColumns {
442 + if col.sortable {
443 + validSortCols[col.dbField] = true
444 + }
445 + }
446 + if !validSortCols[sortColumn] {
447 + sortColumn = "millis" // safe default
448 + }
449 +
450 + // Get list of databases to query
451 + databases, err := c.topQueriesDatabases()
452 + if err != nil {
453 + return &module.FunctionResponse{
454 + Status: 500,
455 + Message: fmt.Sprintf("failed to list databases: %v", err),
456 + }
457 + }
458 + filteredDBs := databases
459 +
460 + // Detect available fields (with caching)
461 + availableFields, err := c.detectMongoProfileFields(ctx, filteredDBs)
462 + if err != nil {
463 + c.Debugf("failed to detect profile fields: %v", err)
464 + }
465 + availableCols := buildAvailableMongoColumns(availableFields)
466 + columns := buildMongoColumnsFromMeta(availableCols)
467 + sortParam := buildMongoSortParam(availableCols)
468 +
469 + // Query system.profile from each database
470 + var allDocs []profileDocument
471 + var profilingDisabledDBs []string
472 + var failedDBs []string
473 + var successfulDBs int
474 +
475 + for _, dbName := range filteredDBs {
476 + docs, enabled, err := c.querySystemProfile(ctx, dbName, sortColumn, limit)
477 + if err != nil {
478 + // Check for timeout (parent or child context)
479 + if ctx.Err() == context.DeadlineExceeded || errors.Is(err, context.DeadlineExceeded) {
480 + return &module.FunctionResponse{Status: 504, Message: "query timed out"}
481 + }
482 + c.Debugf("failed to query system.profile in %s: %v", dbName, err)
483 + failedDBs = append(failedDBs, dbName)
484 + continue
485 + }
486 +
487 + if !enabled {
488 + profilingDisabledDBs = append(profilingDisabledDBs, dbName)
489 + continue
490 + }
491 +
492 + successfulDBs++
493 + allDocs = append(allDocs, docs...)
494 + }
495 +
496 + // Check if all databases failed with errors (not just profiling disabled)
497 + if successfulDBs == 0 && len(failedDBs) > 0 && len(profilingDisabledDBs) == 0 {
498 + return &module.FunctionResponse{
499 + Status: 500,
500 + Message: fmt.Sprintf("failed to query all databases: %v", failedDBs),
501 + }
502 + }
503 +
504 + // Check if profiling is disabled everywhere (no successful queries, no errors, only disabled)
505 + if len(allDocs) == 0 && len(profilingDisabledDBs) > 0 && len(failedDBs) == 0 {
506 + return &module.FunctionResponse{
507 + Status: 503,
508 + Message: fmt.Sprintf(
509 + "Database profiling is disabled. Enable it with: db.setProfilingLevel(1, {slowms: 100}). "+
510 + "Disabled databases: %v", profilingDisabledDBs),
511 + }
512 + }
513 +
514 + // Check if we have a mix of failures and disabled profiling (no successful queries at all)
515 + if successfulDBs == 0 && (len(failedDBs) > 0 || len(profilingDisabledDBs) > 0) {
516 + msg := "No databases could be queried successfully."
517 + if len(failedDBs) > 0 {
518 + msg += fmt.Sprintf(" Failed: %v.", failedDBs)
519 + }
520 + if len(profilingDisabledDBs) > 0 {
521 + msg += fmt.Sprintf(" Profiling disabled: %v.", profilingDisabledDBs)
522 + }
523 + return &module.FunctionResponse{
524 + Status: 503,
525 + Message: msg,
526 + }
527 + }
528 +
529 + // Build empty response structure
530 + emptyResponse := &module.FunctionResponse{
531 + Status: 200,
532 + Message: "No slow queries found. Profiling may be disabled or no queries exceeded the slowms threshold.",
533 + Help: topQueriesHelpText,
534 + Columns: columns,
535 + Data: [][]any{},
536 + DefaultSortColumn: "execution_time",
537 + RequiredParams: []funcapi.ParamConfig{sortParam},
538 + Charts: topQueriesCharts(),
539 + DefaultCharts: topQueriesDefaultCharts(),
540 + GroupBy: topQueriesGroupBy(),
541 + }
542 +
543 + if len(allDocs) == 0 {
544 + return emptyResponse
545 + }
546 +
547 + // Sort all documents by the requested column
548 + sortProfileDocuments(allDocs, sortColumn)
549 +
550 + // Apply limit
551 + if len(allDocs) > limit {
552 + allDocs = allDocs[:limit]
553 + }
554 +
555 + // Convert to response format: [][]any (array of arrays, ordered by column index)
556 + data := make([][]any, 0, len(allDocs))
557 + for _, doc := range allDocs {
558 + row := make([]any, len(availableCols))
559 +
560 + // Fill each column based on available columns
561 + for i, col := range availableCols {
562 + switch col.id {
563 + case "timestamp":
564 + row[i] = doc.Timestamp.Format(time.RFC3339Nano)
565 + case "namespace":
566 + row[i] = doc.Ns
567 + case "operation":
568 + row[i] = doc.Op
569 + case "query":
570 + cmdJSON, err := json.Marshal(doc.Command)
571 + if err != nil {
572 + cmdJSON = []byte("{}")
573 + }
574 + row[i] = strmutil.TruncateText(string(cmdJSON), maxQueryTextLength)
575 + case "execution_time":
576 + row[i] = float64(doc.Millis) / 1000.0 // ms to seconds
577 + case "docs_examined":
578 + row[i] = doc.DocsExamined
579 + case "keys_examined":
580 + row[i] = doc.KeysExamined
581 + case "docs_returned":
582 + row[i] = doc.Nreturned
583 + case "plan_summary":
584 + row[i] = doc.PlanSummary
585 + case "client":
586 + row[i] = doc.Client
587 + case "user":
588 + row[i] = doc.User
589 + case "docs_deleted":
590 + row[i] = doc.Ndeleted
591 + case "docs_inserted":
592 + row[i] = doc.Ninserted
593 + case "docs_modified":
594 + row[i] = doc.NModified
595 + case "response_length":
596 + row[i] = doc.ResponseLength
597 + case "num_yield":
598 + row[i] = doc.NumYield
599 + case "app_name":
600 + row[i] = doc.AppName
601 + case "cursor_exhausted":
602 + row[i] = optionalBool(doc.CursorExhausted)
603 + case "has_sort_stage":
604 + row[i] = optionalBool(doc.HasSortStage)
605 + case "uses_disk":
606 + row[i] = optionalBool(doc.UsedDisk)
607 + case "from_multi_planner":
608 + row[i] = optionalBool(doc.FromMultiPlanner)
609 + case "replanned":
610 + row[i] = optionalBool(doc.Replanned)
611 + case "query_hash":
612 + row[i] = doc.QueryHash
613 + case "plan_cache_key":
614 + row[i] = doc.PlanCacheKey
615 + case "planning_time":
616 + row[i] = optionalDuration(doc.PlanningTimeMicros, 1000000.0) // us to seconds
617 + case "cpu_time":
618 + row[i] = optionalDuration(doc.CpuNanos, 1000000000.0) // ns to seconds
619 + case "query_framework":
620 + row[i] = doc.QueryFramework
621 + case "query_shape_hash":
622 + row[i] = doc.QueryShapeHash
623 + default:
624 + row[i] = nil
625 + }
626 + }
627 +
628 + data = append(data, row)
629 + }
630 +
631 + return &module.FunctionResponse{
632 + Status: 200,
633 + Help: topQueriesHelpText,
634 + Columns: columns,
635 + Data: data,
636 + DefaultSortColumn: "execution_time",
637 + RequiredParams: []funcapi.ParamConfig{sortParam},
638 + Charts: topQueriesCharts(),
639 + DefaultCharts: topQueriesDefaultCharts(),
640 + GroupBy: topQueriesGroupBy(),
641 + }
642 +}
643 +
644 +func (c *Collector) topQueriesDatabases() ([]string, error) {
645 + databases, err := c.conn.listDatabaseNames()
646 + if err != nil {
647 + return nil, err
648 + }
649 +
650 + var filteredDBs []string
651 + for _, dbName := range databases {
652 + if dbName == "admin" || dbName == "local" || dbName == "config" {
653 + continue
654 + }
655 + if c.dbSelector != nil && !c.dbSelector.MatchString(dbName) {
656 + continue
657 + }
658 + filteredDBs = append(filteredDBs, dbName)
659 + }
660 +
661 + return filteredDBs, nil
662 +}
663 +
664 +func (c *Collector) topQueriesParams(ctx context.Context) ([]funcapi.ParamConfig, error) {
665 + if !c.Config.GetTopQueriesFunctionEnabled() {
666 + return nil, fmt.Errorf("top queries function disabled")
667 + }
668 +
669 + databases, err := c.topQueriesDatabases()
670 + if err != nil {
671 + return nil, err
672 + }
673 +
674 + availableFields, err := c.detectMongoProfileFields(ctx, databases)
675 + if err != nil {
676 + return nil, err
677 + }
678 +
679 + availableCols := buildAvailableMongoColumns(availableFields)
680 + sortParam := buildMongoSortParam(availableCols)
681 + return []funcapi.ParamConfig{sortParam}, nil
682 +}
683 +
684 +// querySystemProfile queries the system.profile collection for a specific database
685 +func (c *Collector) querySystemProfile(ctx context.Context, dbName, sortColumn string, limit int) ([]profileDocument, bool, error) {
686 + client, ok := c.conn.(*mongoClient)
687 + if !ok || client == nil || client.client == nil {
688 + return nil, false, fmt.Errorf("client not initialized")
689 + }
690 +
691 + // Set timeout for the query
692 + queryCtx, cancel := context.WithTimeout(ctx, client.timeout)
693 + defer cancel()
694 +
695 + // Check if profiling is enabled for this database
696 + var profilingStatus struct {
697 + Was int `bson:"was"`
698 + }
699 + err := client.client.Database(dbName).RunCommand(queryCtx, bson.D{{Key: "profile", Value: -1}}).Decode(&profilingStatus)
700 + if err != nil {
701 + return nil, false, fmt.Errorf("failed to check profiling status: %w", err)
702 + }
703 +
704 + if profilingStatus.Was == 0 {
705 + return nil, false, nil // Profiling disabled
706 + }
707 +
708 + // Query system.profile
709 + collection := client.client.Database(dbName).Collection("system.profile")
710 +
711 + // Build sort order (descending for all except timestamp which can be either)
712 + sortOrder := -1 // descending by default
713 + sortField := sortColumn
714 +
715 + findOpts := options.Find().
716 + SetSort(bson.D{{Key: sortField, Value: sortOrder}}).
717 + SetLimit(int64(limit))
718 +
719 + cursor, err := collection.Find(queryCtx, bson.M{}, findOpts)
720 + if err != nil {
721 + return nil, true, fmt.Errorf("find failed: %w", err)
722 + }
723 + defer cursor.Close(queryCtx)
724 +
725 + var docs []profileDocument
726 + if err := cursor.All(queryCtx, &docs); err != nil {
727 + return nil, true, fmt.Errorf("cursor.All failed: %w", err)
728 + }
729 +
730 + return docs, true, nil
731 +}
732 +
733 +// sortProfileDocuments sorts documents in place by the specified column (descending)
734 +func sortProfileDocuments(docs []profileDocument, sortColumn string) {
735 + sort.Slice(docs, func(i, j int) bool {
736 + switch sortColumn {
737 + case "millis":
738 + return docs[i].Millis > docs[j].Millis
739 + case "docsExamined":
740 + return docs[i].DocsExamined > docs[j].DocsExamined
741 + case "keysExamined":
742 + return docs[i].KeysExamined > docs[j].KeysExamined
743 + case "nreturned":
744 + return docs[i].Nreturned > docs[j].Nreturned
745 + case "ts":
746 + return docs[i].Timestamp.After(docs[j].Timestamp)
747 + case "ndeleted":
748 + return docs[i].Ndeleted > docs[j].Ndeleted
749 + case "ninserted":
750 + return docs[i].Ninserted > docs[j].Ninserted
751 + case "nModified":
752 + return docs[i].NModified > docs[j].NModified
753 + case "responseLength":
754 + return docs[i].ResponseLength > docs[j].ResponseLength
755 + case "numYield":
756 + return docs[i].NumYield > docs[j].NumYield
757 + case "planningTimeMicros":
758 + vi := int64(0)
759 + vj := int64(0)
760 + if docs[i].PlanningTimeMicros != nil {
761 + vi = *docs[i].PlanningTimeMicros
762 + }
763 + if docs[j].PlanningTimeMicros != nil {
764 + vj = *docs[j].PlanningTimeMicros
765 + }
766 + return vi > vj
767 + case "cpuNanos":
768 + vi := int64(0)
769 + vj := int64(0)
770 + if docs[i].CpuNanos != nil {
771 + vi = *docs[i].CpuNanos
772 + }
773 + if docs[j].CpuNanos != nil {
774 + vj = *docs[j].CpuNanos
775 + }
776 + return vi > vj
777 + default:
778 + return docs[i].Millis > docs[j].Millis
779 + }
780 + })
781 +}
src/go/plugin/go.d/collector/mongodb/functions_test.go new
+390
@@ -0,0 +1,390 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package mongo
4 +
5 +import (
6 + "testing"
7 + "time"
8 +
9 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
10 + "github.com/stretchr/testify/assert"
11 + "github.com/stretchr/testify/require"
12 +)
13 +
14 +func TestMongoMethods(t *testing.T) {
15 + methods := mongoMethods()
16 +
17 + assert.Len(t, methods, 1)
18 + assert.Equal(t, "top-queries", methods[0].ID)
19 + assert.Equal(t, "Top Queries", methods[0].Name)
20 + assert.Contains(t, methods[0].Help, "WARNING")
21 + assert.Contains(t, methods[0].Help, "PII")
22 +
23 + var sortParam *funcapi.ParamConfig
24 + for i := range methods[0].RequiredParams {
25 + if methods[0].RequiredParams[i].ID == "__sort" {
26 + sortParam = &methods[0].RequiredParams[i]
27 + break
28 + }
29 + }
30 + require.NotNil(t, sortParam, "should have __sort param")
31 + require.NotEmpty(t, sortParam.Options, "should have sort options")
32 +
33 + // Check default sort option
34 + defaultFound := false
35 + for _, opt := range sortParam.Options {
36 + if opt.Default {
37 + defaultFound = true
38 + assert.Equal(t, "execution_time", opt.ID)
39 + assert.Equal(t, "millis", opt.Column)
40 + }
41 + }
42 + assert.True(t, defaultFound, "should have a default sort option")
43 +}
44 +
45 +func TestMongoColumnMeta(t *testing.T) {
46 + // Check that mongoAllColumns has all expected columns
47 + assert.NotEmpty(t, mongoAllColumns, "should have column definitions")
48 +
49 + // Check required core columns exist
50 + coreColumns := []string{"timestamp", "namespace", "operation", "query", "execution_time", "docs_examined", "keys_examined", "docs_returned", "plan_summary"}
51 + for _, colID := range coreColumns {
52 + found := false
53 + for _, col := range mongoAllColumns {
54 + if col.id == colID {
55 + found = true
56 + break
57 + }
58 + }
59 + assert.True(t, found, "core column %s should exist", colID)
60 + }
61 +
62 + // Check that core columns are visible by default
63 + for _, col := range mongoAllColumns {
64 + switch col.id {
65 + case "timestamp", "namespace", "operation", "query", "execution_time", "docs_examined", "keys_examined", "docs_returned", "plan_summary":
66 + assert.True(t, col.visible, "column %s should be visible by default", col.id)
67 + }
68 + }
69 +}
70 +
71 +func TestBuildAvailableMongoColumns(t *testing.T) {
72 + tests := []struct {
73 + name string
74 + available map[string]bool
75 + expectLen int
76 + }{
77 + {
78 + name: "core fields only",
79 + available: map[string]bool{
80 + "ts": true, "ns": true, "op": true, "command": true, "millis": true,
81 + },
82 + expectLen: 5, // timestamp, namespace, operation, query, execution_time
83 + },
84 + {
85 + name: "with extra fields",
86 + available: map[string]bool{
87 + "ts": true, "ns": true, "op": true, "command": true, "millis": true,
88 + "docsExamined": true, "keysExamined": true, "planSummary": true,
89 + },
90 + expectLen: 8,
91 + },
92 + {
93 + name: "all fields",
94 + available: func() map[string]bool {
95 + m := make(map[string]bool)
96 + for _, col := range mongoAllColumns {
97 + m[col.dbField] = true
98 + }
99 + return m
100 + }(),
101 + expectLen: len(mongoAllColumns),
102 + },
103 + }
104 +
105 + for _, tc := range tests {
106 + t.Run(tc.name, func(t *testing.T) {
107 + cols := buildAvailableMongoColumns(tc.available)
108 + assert.Len(t, cols, tc.expectLen)
109 + })
110 + }
111 +}
112 +
113 +func TestBuildMongoColumnsFromMeta(t *testing.T) {
114 + // Use a subset of columns for testing
115 + testCols := []mongoColumnMeta{
116 + {id: "timestamp", dbField: "ts", name: "Timestamp", colType: ftTimestamp, visible: true, sortable: true, filter: filterRange, visualization: visValue, summary: summaryMax, transform: trDatetime, uniqueKey: true},
117 + {id: "execution_time", dbField: "millis", name: "Execution Time", colType: ftDuration, visible: true, sortable: true, filter: filterRange, visualization: visBar, summary: summarySum, transform: trDuration, units: "seconds", decimalPoints: 3},
118 + {id: "query", dbField: "command", name: "Query", colType: ftString, visible: true, sortable: false, fullWidth: true, wrap: true, filter: filterMulti, visualization: visValue, transform: trText},
119 + }
120 +
121 + cols := buildMongoColumnsFromMeta(testCols)
122 +
123 + // Check required columns exist
124 + assert.Contains(t, cols, "timestamp")
125 + assert.Contains(t, cols, "execution_time")
126 + assert.Contains(t, cols, "query")
127 +
128 + // Check execution_time column properties
129 + execTimeVal, ok := cols["execution_time"]
130 + require.True(t, ok, "execution_time column must exist")
131 + execTime, ok := execTimeVal.(map[string]any)
132 + require.True(t, ok, "execution_time must be map[string]any")
133 + assert.Equal(t, "duration", execTime["type"])
134 + assert.Equal(t, true, execTime["visible"])
135 + assert.Equal(t, "seconds", execTime["units"])
136 +
137 + // Check query column properties
138 + queryVal, ok := cols["query"]
139 + require.True(t, ok, "query column must exist")
140 + query, ok := queryVal.(map[string]any)
141 + require.True(t, ok, "query must be map[string]any")
142 + assert.Equal(t, true, query["full_width"])
143 + assert.Equal(t, false, query["sortable"])
144 +}
145 +
146 +func TestSortProfileDocuments(t *testing.T) {
147 + docs := []profileDocument{
148 + {Millis: 100, DocsExamined: 50, KeysExamined: 10, Nreturned: 5},
149 + {Millis: 500, DocsExamined: 200, KeysExamined: 30, Nreturned: 20},
150 + {Millis: 200, DocsExamined: 100, KeysExamined: 20, Nreturned: 10},
151 + }
152 +
153 + tests := []struct {
154 + name string
155 + sortColumn string
156 + expected []int64 // expected order of millis values after sort
157 + }{
158 + {
159 + name: "sort by millis",
160 + sortColumn: "millis",
161 + expected: []int64{500, 200, 100},
162 + },
163 + {
164 + name: "sort by docsExamined",
165 + sortColumn: "docsExamined",
166 + expected: []int64{500, 200, 100}, // 200, 100, 50 -> millis 500, 200, 100
167 + },
168 + }
169 +
170 + for _, tc := range tests {
171 + t.Run(tc.name, func(t *testing.T) {
172 + // Make a copy to avoid modifying original
173 + docsCopy := make([]profileDocument, len(docs))
174 + copy(docsCopy, docs)
175 +
176 + sortProfileDocuments(docsCopy, tc.sortColumn)
177 +
178 + for i, expectedMillis := range tc.expected {
179 + assert.Equal(t, expectedMillis, docsCopy[i].Millis,
180 + "position %d should have millis=%d", i, expectedMillis)
181 + }
182 + })
183 + }
184 +}
185 +
186 +func TestSortProfileDocumentsByTimestamp(t *testing.T) {
187 + now := time.Now()
188 + docs := []profileDocument{
189 + {Timestamp: now.Add(-time.Hour), Millis: 100},
190 + {Timestamp: now, Millis: 200},
191 + {Timestamp: now.Add(-30 * time.Minute), Millis: 300},
192 + }
193 +
194 + sortProfileDocuments(docs, "ts")
195 +
196 + // Should be sorted by timestamp descending (most recent first)
197 + assert.Equal(t, int64(200), docs[0].Millis) // now
198 + assert.Equal(t, int64(300), docs[1].Millis) // -30min
199 + assert.Equal(t, int64(100), docs[2].Millis) // -1hr
200 +}
201 +
202 +func TestSortProfileDocumentsByNewColumns(t *testing.T) {
203 + docs := []profileDocument{
204 + {Millis: 100, Ndeleted: 5, Ninserted: 10, NModified: 15, ResponseLength: 100, NumYield: 1},
205 + {Millis: 200, Ndeleted: 15, Ninserted: 5, NModified: 10, ResponseLength: 300, NumYield: 3},
206 + {Millis: 300, Ndeleted: 10, Ninserted: 15, NModified: 5, ResponseLength: 200, NumYield: 2},
207 + }
208 +
209 + tests := []struct {
210 + name string
211 + sortColumn string
212 + expected []int64 // expected order of millis values after sort
213 + }{
214 + {
215 + name: "sort by ndeleted",
216 + sortColumn: "ndeleted",
217 + expected: []int64{200, 300, 100}, // 15, 10, 5
218 + },
219 + {
220 + name: "sort by ninserted",
221 + sortColumn: "ninserted",
222 + expected: []int64{300, 100, 200}, // 15, 10, 5
223 + },
224 + {
225 + name: "sort by nModified",
226 + sortColumn: "nModified",
227 + expected: []int64{100, 200, 300}, // 15, 10, 5
228 + },
229 + {
230 + name: "sort by responseLength",
231 + sortColumn: "responseLength",
232 + expected: []int64{200, 300, 100}, // 300, 200, 100
233 + },
234 + {
235 + name: "sort by numYield",
236 + sortColumn: "numYield",
237 + expected: []int64{200, 300, 100}, // 3, 2, 1
238 + },
239 + }
240 +
241 + for _, tc := range tests {
242 + t.Run(tc.name, func(t *testing.T) {
243 + docsCopy := make([]profileDocument, len(docs))
244 + copy(docsCopy, docs)
245 +
246 + sortProfileDocuments(docsCopy, tc.sortColumn)
247 +
248 + for i, expectedMillis := range tc.expected {
249 + assert.Equal(t, expectedMillis, docsCopy[i].Millis,
250 + "position %d should have millis=%d", i, expectedMillis)
251 + }
252 + })
253 + }
254 +}
255 +
256 +func TestSortProfileDocumentsByOptionalColumns(t *testing.T) {
257 + pt1 := int64(100)
258 + pt2 := int64(300)
259 + pt3 := int64(200)
260 + cpu1 := int64(1000)
261 + cpu2 := int64(3000)
262 + cpu3 := int64(2000)
263 +
264 + docs := []profileDocument{
265 + {Millis: 100, PlanningTimeMicros: &pt1, CpuNanos: &cpu1},
266 + {Millis: 200, PlanningTimeMicros: &pt2, CpuNanos: &cpu2},
267 + {Millis: 300, PlanningTimeMicros: &pt3, CpuNanos: &cpu3},
268 + }
269 +
270 + t.Run("sort by planningTimeMicros", func(t *testing.T) {
271 + docsCopy := make([]profileDocument, len(docs))
272 + copy(docsCopy, docs)
273 +
274 + sortProfileDocuments(docsCopy, "planningTimeMicros")
275 +
276 + // Should be sorted by planningTimeMicros descending: 300, 200, 100
277 + assert.Equal(t, int64(200), docsCopy[0].Millis) // pt2=300
278 + assert.Equal(t, int64(300), docsCopy[1].Millis) // pt3=200
279 + assert.Equal(t, int64(100), docsCopy[2].Millis) // pt1=100
280 + })
281 +
282 + t.Run("sort by cpuNanos", func(t *testing.T) {
283 + docsCopy := make([]profileDocument, len(docs))
284 + copy(docsCopy, docs)
285 +
286 + sortProfileDocuments(docsCopy, "cpuNanos")
287 +
288 + // Should be sorted by cpuNanos descending: 3000, 2000, 1000
289 + assert.Equal(t, int64(200), docsCopy[0].Millis) // cpu2=3000
290 + assert.Equal(t, int64(300), docsCopy[1].Millis) // cpu3=2000
291 + assert.Equal(t, int64(100), docsCopy[2].Millis) // cpu1=1000
292 + })
293 +
294 + t.Run("sort by planningTimeMicros with nil values", func(t *testing.T) {
295 + pt := int64(100)
296 + docsWithNil := []profileDocument{
297 + {Millis: 100, PlanningTimeMicros: nil},
298 + {Millis: 200, PlanningTimeMicros: &pt},
299 + {Millis: 300, PlanningTimeMicros: nil},
300 + }
301 +
302 + sortProfileDocuments(docsWithNil, "planningTimeMicros")
303 +
304 + // Non-nil values should come first when sorted descending
305 + assert.Equal(t, int64(200), docsWithNil[0].Millis) // has pt=100
306 + })
307 +}
308 +
309 +func TestConfigGetTopQueriesFunctionEnabled(t *testing.T) {
310 + t.Run("nil returns true (default)", func(t *testing.T) {
311 + cfg := Config{TopQueriesFunctionEnabled: nil}
312 + assert.True(t, cfg.GetTopQueriesFunctionEnabled())
313 + })
314 +
315 + t.Run("explicit true returns true", func(t *testing.T) {
316 + enabled := true
317 + cfg := Config{TopQueriesFunctionEnabled: &enabled}
318 + assert.True(t, cfg.GetTopQueriesFunctionEnabled())
319 + })
320 +
321 + t.Run("explicit false returns false", func(t *testing.T) {
322 + disabled := false
323 + cfg := Config{TopQueriesFunctionEnabled: &disabled}
324 + assert.False(t, cfg.GetTopQueriesFunctionEnabled())
325 + })
326 +}
327 +
328 +func TestTopQueriesLimitDefault(t *testing.T) {
329 + // When TopQueriesLimit is 0 or negative, should use default
330 + assert.Equal(t, 500, defaultTopQueriesLimit)
331 +
332 + cfg := Config{TopQueriesLimit: 0}
333 + // collectTopQueries would use defaultTopQueriesLimit when config is 0
334 + limit := cfg.TopQueriesLimit
335 + if limit <= 0 {
336 + limit = defaultTopQueriesLimit
337 + }
338 + assert.Equal(t, 500, limit)
339 +
340 + cfg2 := Config{TopQueriesLimit: 100}
341 + limit2 := cfg2.TopQueriesLimit
342 + if limit2 <= 0 {
343 + limit2 = defaultTopQueriesLimit
344 + }
345 + assert.Equal(t, 100, limit2)
346 +}
347 +
348 +func TestOptionalBool(t *testing.T) {
349 + t.Run("nil returns nil", func(t *testing.T) {
350 + result := optionalBool(nil)
351 + assert.Nil(t, result)
352 + })
353 +
354 + t.Run("true returns Yes", func(t *testing.T) {
355 + v := true
356 + result := optionalBool(&v)
357 + assert.Equal(t, "Yes", result)
358 + })
359 +
360 + t.Run("false returns No", func(t *testing.T) {
361 + v := false
362 + result := optionalBool(&v)
363 + assert.Equal(t, "No", result)
364 + })
365 +}
366 +
367 +func TestOptionalDuration(t *testing.T) {
368 + t.Run("nil returns nil", func(t *testing.T) {
369 + result := optionalDuration(nil, 1000.0)
370 + assert.Nil(t, result)
371 + })
372 +
373 + t.Run("value converts correctly", func(t *testing.T) {
374 + v := int64(1000000)
375 + result := optionalDuration(&v, 1000000.0)
376 + assert.Equal(t, 1.0, result)
377 + })
378 +
379 + t.Run("microseconds to seconds", func(t *testing.T) {
380 + v := int64(500000) // 500,000 microseconds = 0.5 seconds
381 + result := optionalDuration(&v, 1000000.0)
382 + assert.Equal(t, 0.5, result)
383 + })
384 +
385 + t.Run("nanoseconds to seconds", func(t *testing.T) {
386 + v := int64(1500000000) // 1.5 billion nanoseconds = 1.5 seconds
387 + result := optionalDuration(&v, 1000000000.0)
388 + assert.Equal(t, 1.5, result)
389 + })
390 +}
src/go/plugin/go.d/collector/mssql/collector.go
+44 -2
@@ -7,6 +7,7 @@ import (
7 "database/sql"
8 _ "embed"
9 "errors"
10 + "sync"
11 "time"
12
13 "github.com/netdata/netdata/go/plugins/pkg/confopt"
@@ -24,8 +25,11 @@ func init() {
25 Defaults: module.Defaults{
26 UpdateEvery: 10,
27 },
27 - Create: func() module.Module { return New() },
28 - Config: func() any { return &Config{} },
28 + Create: func() module.Module { return New() },
29 + Config: func() any { return &Config{} },
30 + Methods: mssqlMethods,
31 + MethodParams: mssqlMethodParams,
32 + HandleMethod: mssqlHandleMethod,
33 })
34 }
35
@@ -52,6 +56,40 @@ type Config struct {
56 UpdateEvery int `yaml:"update_every,omitempty" json:"update_every"`
57 DSN string `yaml:"dsn" json:"dsn"`
58 Timeout confopt.Duration `yaml:"timeout,omitempty" json:"timeout"`
59 +
60 + // QueryStoreTimeWindowDays controls how far back to look in Query Store
61 + // Uses pointer to distinguish "unset" from explicit "0":
62 + // - nil (unset): Apply default of 7 days
63 + // - 0: Query ALL available data (not recommended for busy servers)
64 + // - N > 0: Query last N days
65 + QueryStoreTimeWindowDays *int `yaml:"query_store_time_window_days,omitempty" json:"query_store_time_window_days"`
66 +
67 + // QueryStoreFunctionEnabled controls whether the top-queries function is available
68 + // Uses pointer to distinguish "unset" from explicit "false":
69 + // - nil (unset): Apply default of true (enabled)
70 + // - false: Explicitly disabled
71 + // - true: Explicitly enabled
72 + // Default: true - MSSQL Query Store may contain unmasked PII in query text
73 + QueryStoreFunctionEnabled *bool `yaml:"query_store_function_enabled,omitempty" json:"query_store_function_enabled"`
74 +
75 + // TopQueriesLimit is the maximum number of queries to return
76 + TopQueriesLimit int `yaml:"top_queries_limit,omitempty" json:"top_queries_limit,omitempty"`
77 +}
78 +
79 +// GetQueryStoreTimeWindowDays returns the time window for Query Store queries (default: 7)
80 +func (c *Config) GetQueryStoreTimeWindowDays() int {
81 + if c.QueryStoreTimeWindowDays == nil {
82 + return 7
83 + }
84 + return *c.QueryStoreTimeWindowDays
85 +}
86 +
87 +// GetQueryStoreFunctionEnabled returns whether the Query Store function is enabled (default: true)
88 +func (c *Config) GetQueryStoreFunctionEnabled() bool {
89 + if c.QueryStoreFunctionEnabled == nil {
90 + return true
91 + }
92 + return *c.QueryStoreFunctionEnabled
93 }
94
95 type Collector struct {
@@ -70,6 +108,10 @@ type Collector struct {
108 seenLockStatsTypes map[string]bool
109 seenJobs map[string]bool
110 seenReplications map[string]bool
111 +
112 + // Query Store column cache (per-instance to handle different SQL Server versions)
113 + queryStoreColsMu sync.RWMutex // protects queryStoreCols for concurrent access
114 + queryStoreCols map[string]bool
115 }
116
117 func (c *Collector) Configuration() any {
src/go/plugin/go.d/collector/mssql/config_schema.json
+34
@@ -29,6 +29,27 @@
29 "type": "number",
30 "minimum": 0.5,
31 "default": 5
32 + },
33 + "query_store_function_enabled": {
34 + "title": "Enable Query Store Function",
35 + "description": "Enable the top-queries function using SQL Server Query Store. WARNING: Query Store may contain unmasked PII (customer names, emails, IDs) in query text. Only enable after ensuring proper access controls to the Netdata dashboard.",
36 + "type": "boolean",
37 + "default": true
38 + },
39 + "query_store_time_window_days": {
40 + "title": "Query Store Time Window (Days)",
41 + "description": "How many days of Query Store data to query for the top-queries function. Set to 0 to query all available data (not recommended for busy servers).",
42 + "type": "integer",
43 + "minimum": 0,
44 + "default": 7
45 + },
46 + "top_queries_limit": {
47 + "title": "Top Queries Limit",
48 + "description": "Maximum number of queries to return in the top-queries function response.",
49 + "type": "integer",
50 + "minimum": 1,
51 + "maximum": 5000,
52 + "default": 500
53 }
54 },
55 "required": [
@@ -52,6 +73,12 @@
73 "timeout": {
74 "ui:help": "Accepts decimals for sub-second granularity (e.g., 0.5 for 500ms)."
75 },
76 + "query_store_function_enabled": {
77 + "ui:help": "When enabled, the 'top-queries' function becomes available in the Netdata dashboard. Review the PII warning before enabling."
78 + },
79 + "query_store_time_window_days": {
80 + "ui:help": "Limits Query Store data to recent days. Lower values improve performance on busy servers."
81 + },
82 "ui:flavour": "tabs",
83 "ui:options": {
84 "tabs": [
@@ -63,6 +90,13 @@
90 "timeout",
91 "vnode"
92 ]
93 + },
94 + {
95 + "title": "Query Store",
96 + "fields": [
97 + "query_store_function_enabled",
98 + "query_store_time_window_days"
99 + ]
100 }
101 ]
102 }
src/go/plugin/go.d/collector/mssql/functions.go new
+762
@@ -0,0 +1,762 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package mssql
4 +
5 +import (
6 + "context"
7 + "database/sql"
8 + "fmt"
9 + "strings"
10 +
11 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
12 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
13 + "github.com/netdata/netdata/go/plugins/plugin/go.d/pkg/strmutil"
14 +)
15 +
16 +const maxQueryTextLength = 4096
17 +
18 +const (
19 + paramSort = "__sort"
20 +
21 + ftString = funcapi.FieldTypeString
22 + ftInteger = funcapi.FieldTypeInteger
23 + ftFloat = funcapi.FieldTypeFloat
24 + ftDuration = funcapi.FieldTypeDuration
25 +
26 + trNone = funcapi.FieldTransformNone
27 + trNumber = funcapi.FieldTransformNumber
28 + trDuration = funcapi.FieldTransformDuration
29 +
30 + sortAsc = funcapi.FieldSortAscending
31 + sortDesc = funcapi.FieldSortDescending
32 +
33 + summaryCount = funcapi.FieldSummaryCount
34 + summarySum = funcapi.FieldSummarySum
35 + summaryMin = funcapi.FieldSummaryMin
36 + summaryMax = funcapi.FieldSummaryMax
37 + summaryMean = funcapi.FieldSummaryMean
38 +
39 + filterMulti = funcapi.FieldFilterMultiselect
40 + filterRange = funcapi.FieldFilterRange
41 +)
42 +
43 +// mssqlColumnMeta defines metadata for a single column
44 +type mssqlColumnMeta struct {
45 + dbColumn string // Column name in sys.query_store_runtime_stats
46 + uiKey string // Canonical name used everywhere: SQL alias, UI key, sort key
47 + displayName string // Display name in UI
48 + dataType funcapi.FieldType // "string", "integer", "float", "duration"
49 + units string // Unit for duration types
50 + visible bool // Default visibility
51 + transform funcapi.FieldTransform // Transform for value_options
52 + decimalPoints int // Decimal points for display
53 + sortDir funcapi.FieldSort // Sort direction: "ascending" or "descending"
54 + summary funcapi.FieldSummary // Summary function
55 + filter funcapi.FieldFilter // Filter type
56 + isMicroseconds bool // Needs μs to milliseconds conversion
57 + isSortOption bool // Show in sort dropdown
58 + sortLabel string // Label for sort option
59 + isDefaultSort bool // Is this the default sort option
60 + isUniqueKey bool // Is this column a unique key
61 + isSticky bool // Is this column sticky in UI
62 + fullWidth bool // Should column take full width
63 + isIdentity bool // Is this an identity column (query_hash, query_text, etc.)
64 + needsAvg bool // Needs weighted average calculation (avg_* columns)
65 +}
66 +
67 +// mssqlAllColumns defines ALL possible columns from Query Store
68 +// Columns that don't exist in certain SQL Server versions will be filtered at runtime
69 +var mssqlAllColumns = []mssqlColumnMeta{
70 + // Identity columns - always available
71 + {dbColumn: "query_hash", uiKey: "queryHash", displayName: "Query Hash", dataType: ftString, visible: false, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, isUniqueKey: true, isIdentity: true},
72 + {dbColumn: "query_sql_text", uiKey: "query", displayName: "Query", dataType: ftString, visible: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, isSticky: true, fullWidth: true, isIdentity: true},
73 + {dbColumn: "database_name", uiKey: "database", displayName: "Database", dataType: ftString, visible: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, isIdentity: true},
74 +
75 + // Execution count - always available
76 + {dbColumn: "count_executions", uiKey: "calls", displayName: "Calls", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Number of Calls"},
77 +
78 + // Duration metrics (microseconds -> milliseconds) - SQL 2016+
79 + {dbColumn: "avg_duration", uiKey: "totalTime", displayName: "Total Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange, isMicroseconds: true, isSortOption: true, sortLabel: "Top queries by Total Execution Time", isDefaultSort: true},
80 + {dbColumn: "avg_duration", uiKey: "avgTime", displayName: "Avg Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMean, filter: filterRange, isMicroseconds: true, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Execution Time"},
81 + {dbColumn: "last_duration", uiKey: "lastTime", displayName: "Last Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
82 + {dbColumn: "min_duration", uiKey: "minTime", displayName: "Min Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMin, filter: filterRange, isMicroseconds: true},
83 + {dbColumn: "max_duration", uiKey: "maxTime", displayName: "Max Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
84 + {dbColumn: "stdev_duration", uiKey: "stdevTime", displayName: "StdDev Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
85 +
86 + // CPU time metrics (microseconds -> milliseconds)
87 + {dbColumn: "avg_cpu_time", uiKey: "avgCpu", displayName: "Avg CPU", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMean, filter: filterRange, isMicroseconds: true, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average CPU Time"},
88 + {dbColumn: "last_cpu_time", uiKey: "lastCpu", displayName: "Last CPU", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
89 + {dbColumn: "min_cpu_time", uiKey: "minCpu", displayName: "Min CPU", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMin, filter: filterRange, isMicroseconds: true},
90 + {dbColumn: "max_cpu_time", uiKey: "maxCpu", displayName: "Max CPU", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
91 + {dbColumn: "stdev_cpu_time", uiKey: "stdevCpu", displayName: "StdDev CPU", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
92 +
93 + // Logical I/O reads
94 + {dbColumn: "avg_logical_io_reads", uiKey: "avgReads", displayName: "Avg Logical Reads", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 0, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Logical Reads"},
95 + {dbColumn: "last_logical_io_reads", uiKey: "lastReads", displayName: "Last Logical Reads", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
96 + {dbColumn: "min_logical_io_reads", uiKey: "minReads", displayName: "Min Logical Reads", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
97 + {dbColumn: "max_logical_io_reads", uiKey: "maxReads", displayName: "Max Logical Reads", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
98 + {dbColumn: "stdev_logical_io_reads", uiKey: "stdevReads", displayName: "StdDev Logical Reads", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
99 +
100 + // Logical I/O writes
101 + {dbColumn: "avg_logical_io_writes", uiKey: "avgWrites", displayName: "Avg Logical Writes", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 0, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Logical Writes"},
102 + {dbColumn: "last_logical_io_writes", uiKey: "lastWrites", displayName: "Last Logical Writes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
103 + {dbColumn: "min_logical_io_writes", uiKey: "minWrites", displayName: "Min Logical Writes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
104 + {dbColumn: "max_logical_io_writes", uiKey: "maxWrites", displayName: "Max Logical Writes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
105 + {dbColumn: "stdev_logical_io_writes", uiKey: "stdevWrites", displayName: "StdDev Logical Writes", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
106 +
107 + // Physical I/O reads
108 + {dbColumn: "avg_physical_io_reads", uiKey: "avgPhysReads", displayName: "Avg Physical Reads", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 0, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Physical Reads"},
109 + {dbColumn: "last_physical_io_reads", uiKey: "lastPhysReads", displayName: "Last Physical Reads", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
110 + {dbColumn: "min_physical_io_reads", uiKey: "minPhysReads", displayName: "Min Physical Reads", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
111 + {dbColumn: "max_physical_io_reads", uiKey: "maxPhysReads", displayName: "Max Physical Reads", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
112 + {dbColumn: "stdev_physical_io_reads", uiKey: "stdevPhysReads", displayName: "StdDev Physical Reads", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
113 +
114 + // CLR time (microseconds -> milliseconds)
115 + {dbColumn: "avg_clr_time", uiKey: "avgClr", displayName: "Avg CLR Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMean, filter: filterRange, isMicroseconds: true, needsAvg: true},
116 + {dbColumn: "last_clr_time", uiKey: "lastClr", displayName: "Last CLR Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
117 + {dbColumn: "min_clr_time", uiKey: "minClr", displayName: "Min CLR Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMin, filter: filterRange, isMicroseconds: true},
118 + {dbColumn: "max_clr_time", uiKey: "maxClr", displayName: "Max CLR Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
119 + {dbColumn: "stdev_clr_time", uiKey: "stdevClr", displayName: "StdDev CLR Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isMicroseconds: true},
120 +
121 + // DOP (degree of parallelism)
122 + {dbColumn: "avg_dop", uiKey: "avgDop", displayName: "Avg DOP", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 1, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Parallelism"},
123 + {dbColumn: "last_dop", uiKey: "lastDop", displayName: "Last DOP", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
124 + {dbColumn: "min_dop", uiKey: "minDop", displayName: "Min DOP", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
125 + {dbColumn: "max_dop", uiKey: "maxDop", displayName: "Max DOP", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
126 + {dbColumn: "stdev_dop", uiKey: "stdevDop", displayName: "StdDev DOP", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
127 +
128 + // Memory grant (8KB pages)
129 + {dbColumn: "avg_query_max_used_memory", uiKey: "avgMemory", displayName: "Avg Memory (8KB pages)", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 0, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Memory Grant"},
130 + {dbColumn: "last_query_max_used_memory", uiKey: "lastMemory", displayName: "Last Memory (8KB pages)", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
131 + {dbColumn: "min_query_max_used_memory", uiKey: "minMemory", displayName: "Min Memory (8KB pages)", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
132 + {dbColumn: "max_query_max_used_memory", uiKey: "maxMemory", displayName: "Max Memory (8KB pages)", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
133 + {dbColumn: "stdev_query_max_used_memory", uiKey: "stdevMemory", displayName: "StdDev Memory", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
134 +
135 + // Row count
136 + {dbColumn: "avg_rowcount", uiKey: "avgRows", displayName: "Avg Rows", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 0, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Row Count"},
137 + {dbColumn: "last_rowcount", uiKey: "lastRows", displayName: "Last Rows", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
138 + {dbColumn: "min_rowcount", uiKey: "minRows", displayName: "Min Rows", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
139 + {dbColumn: "max_rowcount", uiKey: "maxRows", displayName: "Max Rows", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
140 + {dbColumn: "stdev_rowcount", uiKey: "stdevRows", displayName: "StdDev Rows", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
141 +
142 + // SQL Server 2017+ log bytes
143 + {dbColumn: "avg_log_bytes_used", uiKey: "avgLogBytes", displayName: "Avg Log Bytes", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 0, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average Log Bytes"},
144 + {dbColumn: "last_log_bytes_used", uiKey: "lastLogBytes", displayName: "Last Log Bytes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
145 + {dbColumn: "min_log_bytes_used", uiKey: "minLogBytes", displayName: "Min Log Bytes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
146 + {dbColumn: "max_log_bytes_used", uiKey: "maxLogBytes", displayName: "Max Log Bytes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
147 + {dbColumn: "stdev_log_bytes_used", uiKey: "stdevLogBytes", displayName: "StdDev Log Bytes", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
148 +
149 + // SQL Server 2017+ tempdb space
150 + {dbColumn: "avg_tempdb_space_used", uiKey: "avgTempdb", displayName: "Avg TempDB (8KB pages)", dataType: ftFloat, visible: true, transform: trNumber, decimalPoints: 0, sortDir: sortDesc, summary: summaryMean, filter: filterRange, needsAvg: true, isSortOption: true, sortLabel: "Top queries by Average TempDB Usage"},
151 + {dbColumn: "last_tempdb_space_used", uiKey: "lastTempdb", displayName: "Last TempDB (8KB pages)", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
152 + {dbColumn: "min_tempdb_space_used", uiKey: "minTempdb", displayName: "Min TempDB (8KB pages)", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
153 + {dbColumn: "max_tempdb_space_used", uiKey: "maxTempdb", displayName: "Max TempDB (8KB pages)", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
154 + {dbColumn: "stdev_tempdb_space_used", uiKey: "stdevTempdb", displayName: "StdDev TempDB", dataType: ftFloat, visible: false, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
155 +}
156 +
157 +// mssqlMethods returns the available function methods for MSSQL
158 +func mssqlMethods() []module.MethodConfig {
159 + // Build sort options from column metadata
160 + var sortOptions []funcapi.ParamOption
161 + sortDir := funcapi.FieldSortDescending
162 + seen := make(map[string]bool) // Avoid duplicates from totalTime/avgTime using same dbColumn
163 + for _, col := range mssqlAllColumns {
164 + if col.isSortOption && !seen[col.uiKey] {
165 + seen[col.uiKey] = true
166 + sortOptions = append(sortOptions, funcapi.ParamOption{
167 + ID: col.uiKey,
168 + Column: col.uiKey, // Use UI key for sort, we'll map internally
169 + Name: col.sortLabel,
170 + Default: col.isDefaultSort,
171 + Sort: &sortDir,
172 + })
173 + }
174 + }
175 +
176 + return []module.MethodConfig{{
177 + ID: "top-queries",
178 + Name: "Top Queries",
179 + Help: "Top SQL queries from Query Store",
180 + RequiredParams: []funcapi.ParamConfig{
181 + {
182 + ID: paramSort,
183 + Name: "Filter By",
184 + Help: "Select the primary sort column",
185 + Selection: funcapi.ParamSelect,
186 + Options: sortOptions,
187 + UniqueView: true,
188 + },
189 + },
190 + }}
191 +}
192 +
193 +func mssqlMethodParams(ctx context.Context, job *module.Job, method string) ([]funcapi.ParamConfig, error) {
194 + collector, ok := job.Module().(*Collector)
195 + if !ok {
196 + return nil, fmt.Errorf("invalid module type")
197 + }
198 + if collector.db == nil {
199 + return nil, fmt.Errorf("collector is still initializing")
200 + }
201 + switch method {
202 + case "top-queries":
203 + return collector.topQueriesParams(ctx)
204 + default:
205 + return nil, fmt.Errorf("unknown method: %s", method)
206 + }
207 +}
208 +
209 +// mssqlHandleMethod handles function requests for MSSQL
210 +func mssqlHandleMethod(ctx context.Context, job *module.Job, method string, params funcapi.ResolvedParams) *module.FunctionResponse {
211 + collector, ok := job.Module().(*Collector)
212 + if !ok {
213 + return &module.FunctionResponse{Status: 500, Message: "internal error: invalid module type"}
214 + }
215 +
216 + // Check if collector is initialized (first collect() may not have run yet)
217 + if collector.db == nil {
218 + return &module.FunctionResponse{
219 + Status: 503,
220 + Message: "collector is still initializing, please retry in a few seconds",
221 + }
222 + }
223 +
224 + switch method {
225 + case "top-queries":
226 + return collector.collectTopQueries(ctx, params.Column(paramSort))
227 + default:
228 + return &module.FunctionResponse{Status: 404, Message: fmt.Sprintf("unknown method: %s", method)}
229 + }
230 +}
231 +
232 +// detectMSSQLQueryStoreColumns queries any database with Query Store enabled to discover available columns
233 +func (c *Collector) detectMSSQLQueryStoreColumns(ctx context.Context) (map[string]bool, error) {
234 + // Fast path: return cached result
235 + c.queryStoreColsMu.RLock()
236 + if c.queryStoreCols != nil {
237 + cols := c.queryStoreCols
238 + c.queryStoreColsMu.RUnlock()
239 + return cols, nil
240 + }
241 + c.queryStoreColsMu.RUnlock()
242 +
243 + // Slow path: query and cache
244 + c.queryStoreColsMu.Lock()
245 + defer c.queryStoreColsMu.Unlock()
246 +
247 + // Double-check after acquiring write lock
248 + if c.queryStoreCols != nil {
249 + return c.queryStoreCols, nil
250 + }
251 +
252 + // Find any database with Query Store enabled (excluding system databases)
253 + var sampleDB string
254 + err := c.db.QueryRowContext(ctx, `
255 + SELECT TOP 1 name
256 + FROM sys.databases
257 + WHERE is_query_store_on = 1
258 + AND name NOT IN ('master', 'tempdb', 'model', 'msdb')
259 + `).Scan(&sampleDB)
260 + if err != nil {
261 + if err == sql.ErrNoRows {
262 + return nil, fmt.Errorf("no databases have Query Store enabled")
263 + }
264 + return nil, fmt.Errorf("failed to find database with Query Store: %w", err)
265 + }
266 +
267 + // Use dynamic SQL to get column metadata from that database's Query Store view
268 + // Three-part naming works: [DatabaseName].sys.query_store_runtime_stats
269 + query := fmt.Sprintf(`SELECT TOP 0 * FROM [%s].sys.query_store_runtime_stats`, sampleDB)
270 + rows, err := c.db.QueryContext(ctx, query)
271 + if err != nil {
272 + return nil, fmt.Errorf("failed to query Query Store columns from %s: %w", sampleDB, err)
273 + }
274 + defer rows.Close()
275 +
276 + // Get column names from result set metadata
277 + columnNames, err := rows.Columns()
278 + if err != nil {
279 + return nil, fmt.Errorf("failed to get column names: %w", err)
280 + }
281 +
282 + cols := make(map[string]bool)
283 + for _, colName := range columnNames {
284 + // Normalize to lowercase for case-insensitive comparison
285 + cols[strings.ToLower(colName)] = true
286 + }
287 +
288 + // If we found no columns, something is wrong
289 + if len(cols) == 0 {
290 + return nil, fmt.Errorf("no columns found in sys.query_store_runtime_stats")
291 + }
292 +
293 + // Add identity columns that are always available (from other Query Store views)
294 + cols["query_hash"] = true
295 + cols["query_sql_text"] = true
296 + cols["database_name"] = true
297 +
298 + // Cache the result
299 + c.queryStoreCols = cols
300 +
301 + return cols, nil
302 +}
303 +
304 +// buildAvailableMSSQLColumns filters columns based on what's available in the database
305 +func (c *Collector) buildAvailableMSSQLColumns(availableCols map[string]bool) []mssqlColumnMeta {
306 + var cols []mssqlColumnMeta
307 + seen := make(map[string]bool)
308 +
309 + for _, col := range mssqlAllColumns {
310 + // Skip duplicates (e.g., totalTime and avgTime both use avg_duration)
311 + if seen[col.uiKey] {
312 + continue
313 + }
314 + // Identity columns are always available
315 + if col.isIdentity {
316 + cols = append(cols, col)
317 + seen[col.uiKey] = true
318 + continue
319 + }
320 + // Check if the dbColumn exists
321 + if availableCols[col.dbColumn] {
322 + cols = append(cols, col)
323 + seen[col.uiKey] = true
324 + }
325 + }
326 + return cols
327 +}
328 +
329 +// mapAndValidateMSSQLSortColumn maps UI sort key to the appropriate sort expression
330 +// Uses the filtered cols list to ensure the sort column is actually in the SELECT
331 +func (c *Collector) mapAndValidateMSSQLSortColumn(sortKey string, cols []mssqlColumnMeta) string {
332 + // First, check if the requested sort key is in the available columns
333 + for _, col := range cols {
334 + if col.uiKey == sortKey && col.isSortOption {
335 + return col.uiKey
336 + }
337 + }
338 +
339 + // Fall back to the first available sort column
340 + for _, col := range cols {
341 + if col.isSortOption {
342 + return col.uiKey
343 + }
344 + }
345 +
346 + // Last resort: use first non-identity column
347 + for _, col := range cols {
348 + if !col.isIdentity {
349 + return col.uiKey
350 + }
351 + }
352 +
353 + // Absolute fallback: use first column in the list (must exist in SELECT)
354 + if len(cols) > 0 {
355 + return cols[0].uiKey
356 + }
357 +
358 + return "" // empty - will be handled by caller
359 +}
360 +
361 +// buildMSSQLSelectExpressions builds the SELECT expressions for a single database query
362 +func (c *Collector) buildMSSQLSelectExpressions(cols []mssqlColumnMeta, dbNameExpr string) []string {
363 + var selectParts []string
364 +
365 + for _, col := range cols {
366 + var expr string
367 + switch {
368 + case col.isIdentity:
369 + switch col.uiKey {
370 + case "queryHash":
371 + expr = fmt.Sprintf("CONVERT(VARCHAR(64), q.query_hash, 1) AS [%s]", col.uiKey)
372 + case "query":
373 + expr = fmt.Sprintf("qt.query_sql_text AS [%s]", col.uiKey)
374 + case "database":
375 + expr = fmt.Sprintf("%s AS [%s]", dbNameExpr, col.uiKey)
376 + }
377 + case col.uiKey == "calls":
378 + expr = fmt.Sprintf("SUM(rs.count_executions) AS [%s]", col.uiKey)
379 + case col.uiKey == "totalTime":
380 + // Total time = sum of (avg_duration * executions) converted to milliseconds
381 + expr = fmt.Sprintf("SUM(rs.avg_duration * rs.count_executions) / 1000.0 AS [%s]", col.uiKey)
382 + case col.needsAvg && col.isMicroseconds:
383 + // Weighted average with μs to milliseconds conversion
384 + expr = fmt.Sprintf("CASE WHEN SUM(rs.count_executions) > 0 THEN SUM(rs.%s * rs.count_executions) / SUM(rs.count_executions) / 1000.0 ELSE 0 END AS [%s]", col.dbColumn, col.uiKey)
385 + case col.needsAvg:
386 + // Weighted average without time conversion
387 + expr = fmt.Sprintf("CASE WHEN SUM(rs.count_executions) > 0 THEN SUM(rs.%s * rs.count_executions) / SUM(rs.count_executions) ELSE 0 END AS [%s]", col.dbColumn, col.uiKey)
388 + case col.isMicroseconds:
389 + // Aggregate with μs to milliseconds conversion
390 + aggFunc := "MAX"
391 + if strings.HasPrefix(col.dbColumn, "min_") {
392 + aggFunc = "MIN"
393 + }
394 + expr = fmt.Sprintf("%s(rs.%s) / 1000.0 AS [%s]", aggFunc, col.dbColumn, col.uiKey)
395 + default:
396 + // Simple aggregate
397 + aggFunc := "MAX"
398 + if strings.HasPrefix(col.dbColumn, "min_") {
399 + aggFunc = "MIN"
400 + }
401 + if strings.HasPrefix(col.dbColumn, "stdev_") {
402 + aggFunc = "MAX" // Use MAX for stddev aggregation
403 + }
404 + expr = fmt.Sprintf("%s(rs.%s) AS [%s]", aggFunc, col.dbColumn, col.uiKey)
405 + }
406 + if expr != "" {
407 + selectParts = append(selectParts, expr)
408 + }
409 + }
410 + return selectParts
411 +}
412 +
413 +// buildMSSQLDynamicSQL builds dynamic SQL that aggregates across all databases with Query Store enabled
414 +// Uses sp_executesql to execute the built query
415 +func (c *Collector) buildMSSQLDynamicSQL(cols []mssqlColumnMeta, sortColumn string, timeWindowDays int, limit int) string {
416 + // Build the SELECT expressions template (with placeholder for database name)
417 + // We use ''' + name + N''' to close the outer string, concatenate the db name, and reopen
418 + // This produces a properly quoted string literal like 'DatabaseName' in the final SQL
419 + selectParts := c.buildMSSQLSelectExpressions(cols, "''' + name + N'''")
420 + selectExpr := strings.Join(selectParts, ",\n ")
421 +
422 + // Time window filter
423 + timeFilter := ""
424 + if timeWindowDays > 0 {
425 + timeFilter = fmt.Sprintf("WHERE rsi.start_time >= DATEADD(day, -%d, GETUTCDATE())", timeWindowDays)
426 + }
427 +
428 + // Validate sort column
429 + orderByExpr := sortColumn
430 + if orderByExpr == "" {
431 + for _, col := range cols {
432 + if !col.isIdentity {
433 + orderByExpr = col.uiKey
434 + break
435 + }
436 + }
437 + if orderByExpr == "" && len(cols) > 0 {
438 + orderByExpr = cols[0].uiKey
439 + }
440 + }
441 +
442 + // Build the dynamic SQL that creates UNION ALL across all databases
443 + // The database names come from sys.databases, ensuring safety (no user input)
444 + return fmt.Sprintf(`
445 +DECLARE @sql NVARCHAR(MAX) = N'';
446 +
447 +SELECT @sql = @sql +
448 + CASE WHEN @sql = N'' THEN N'' ELSE N' UNION ALL ' END +
449 + N'SELECT
450 + %s
451 + FROM ' + QUOTENAME(name) + N'.sys.query_store_query q
452 + INNER JOIN ' + QUOTENAME(name) + N'.sys.query_store_query_text qt ON q.query_text_id = qt.query_text_id
453 + INNER JOIN ' + QUOTENAME(name) + N'.sys.query_store_plan p ON q.query_id = p.query_id
454 + INNER JOIN ' + QUOTENAME(name) + N'.sys.query_store_runtime_stats rs ON p.plan_id = rs.plan_id
455 + INNER JOIN ' + QUOTENAME(name) + N'.sys.query_store_runtime_stats_interval rsi ON rs.runtime_stats_interval_id = rsi.runtime_stats_interval_id
456 + %s
457 + GROUP BY q.query_hash, qt.query_sql_text'
458 +FROM sys.databases
459 +WHERE is_query_store_on = 1
460 + AND name NOT IN ('master', 'tempdb', 'model', 'msdb');
461 +
462 +IF @sql = N''
463 +BEGIN
464 + RAISERROR('No databases have Query Store enabled', 16, 1);
465 + RETURN;
466 +END
467 +
468 +SET @sql = N'SELECT TOP %d * FROM (' + @sql + N') AS combined ORDER BY [%s] DESC';
469 +EXEC sp_executesql @sql;
470 +`, selectExpr, timeFilter, limit, orderByExpr)
471 +}
472 +
473 +// mssqlRowScanner interface for testing
474 +type mssqlRowScanner interface {
475 + Next() bool
476 + Scan(dest ...any) error
477 + Err() error
478 +}
479 +
480 +// scanMSSQLDynamicRows scans rows dynamically based on column types
481 +func (c *Collector) scanMSSQLDynamicRows(rows mssqlRowScanner, cols []mssqlColumnMeta) ([][]any, error) {
482 + data := make([][]any, 0, 500)
483 +
484 + // Create value holders for scanning
485 + for rows.Next() {
486 + values := make([]any, len(cols))
487 + valuePtrs := make([]any, len(cols))
488 +
489 + for i, col := range cols {
490 + switch col.dataType {
491 + case ftString:
492 + var v sql.NullString
493 + values[i] = &v
494 + case ftInteger:
495 + var v sql.NullInt64
496 + values[i] = &v
497 + case ftFloat, ftDuration:
498 + var v sql.NullFloat64
499 + values[i] = &v
500 + default:
501 + var v any
502 + values[i] = &v
503 + }
504 + valuePtrs[i] = values[i]
505 + }
506 +
507 + if err := rows.Scan(valuePtrs...); err != nil {
508 + return nil, fmt.Errorf("row scan failed: %w", err)
509 + }
510 +
511 + // Convert scanned values to output format
512 + row := make([]any, len(cols))
513 + for i, col := range cols {
514 + switch v := values[i].(type) {
515 + case *sql.NullString:
516 + if v.Valid {
517 + s := v.String
518 + // Truncate query text
519 + if col.uiKey == "query" {
520 + s = strmutil.TruncateText(s, maxQueryTextLength)
521 + }
522 + row[i] = s
523 + } else {
524 + row[i] = ""
525 + }
526 + case *sql.NullInt64:
527 + if v.Valid {
528 + row[i] = v.Int64
529 + } else {
530 + row[i] = int64(0)
531 + }
532 + case *sql.NullFloat64:
533 + if v.Valid {
534 + row[i] = v.Float64
535 + } else {
536 + row[i] = float64(0)
537 + }
538 + default:
539 + row[i] = nil
540 + }
541 + }
542 + data = append(data, row)
543 + }
544 +
545 + if err := rows.Err(); err != nil {
546 + return nil, fmt.Errorf("rows iteration error: %w", err)
547 + }
548 +
549 + return data, nil
550 +}
551 +
552 +// buildMSSQLDynamicSortOptions builds sort options from available columns
553 +// Returns only sort options for columns that actually exist in the database
554 +func (c *Collector) buildMSSQLDynamicSortOptions(cols []mssqlColumnMeta) []funcapi.ParamOption {
555 + var sortOpts []funcapi.ParamOption
556 + seen := make(map[string]bool)
557 + sortDir := funcapi.FieldSortDescending
558 +
559 + for _, col := range cols {
560 + if col.isSortOption && !seen[col.uiKey] {
561 + seen[col.uiKey] = true
562 + sortOpts = append(sortOpts, funcapi.ParamOption{
563 + ID: col.uiKey,
564 + Column: col.uiKey,
565 + Name: col.sortLabel,
566 + Default: col.isDefaultSort,
567 + Sort: &sortDir,
568 + })
569 + }
570 + }
571 + return sortOpts
572 +}
573 +
574 +func (c *Collector) topQueriesSortParam(cols []mssqlColumnMeta) (funcapi.ParamConfig, []funcapi.ParamOption) {
575 + sortOptions := c.buildMSSQLDynamicSortOptions(cols)
576 + sortParam := funcapi.ParamConfig{
577 + ID: paramSort,
578 + Name: "Filter By",
579 + Help: "Select the primary sort column",
580 + Selection: funcapi.ParamSelect,
581 + Options: sortOptions,
582 + UniqueView: true,
583 + }
584 + return sortParam, sortOptions
585 +}
586 +
587 +func (c *Collector) topQueriesParams(ctx context.Context) ([]funcapi.ParamConfig, error) {
588 + if !c.Config.GetQueryStoreFunctionEnabled() {
589 + return nil, fmt.Errorf("query store function disabled")
590 + }
591 +
592 + availableCols, err := c.detectMSSQLQueryStoreColumns(ctx)
593 + if err != nil {
594 + return nil, err
595 + }
596 +
597 + cols := c.buildAvailableMSSQLColumns(availableCols)
598 + if len(cols) == 0 {
599 + return nil, fmt.Errorf("no columns available in Query Store")
600 + }
601 +
602 + sortParam, _ := c.topQueriesSortParam(cols)
603 + return []funcapi.ParamConfig{sortParam}, nil
604 +}
605 +
606 +// buildMSSQLDynamicColumns builds column definitions for the response
607 +func (c *Collector) buildMSSQLDynamicColumns(cols []mssqlColumnMeta) map[string]any {
608 + columns := make(map[string]any)
609 + for i, col := range cols {
610 + visual := funcapi.FieldVisualValue
611 + if col.dataType == ftDuration {
612 + visual = funcapi.FieldVisualBar
613 + }
614 + colDef := funcapi.Column{
615 + Index: i,
616 + Name: col.displayName,
617 + Type: col.dataType,
618 + Units: col.units,
619 + Visualization: visual,
620 + Sort: col.sortDir,
621 + Sortable: true,
622 + Sticky: col.isSticky,
623 + Summary: col.summary,
624 + Filter: col.filter,
625 + FullWidth: col.fullWidth,
626 + Wrap: false,
627 + DefaultExpandedFilter: false,
628 + UniqueKey: col.isUniqueKey,
629 + Visible: col.visible,
630 + ValueOptions: funcapi.ValueOptions{
631 + Transform: col.transform,
632 + DecimalPoints: col.decimalPoints,
633 + DefaultValue: nil,
634 + },
635 + }
636 + columns[col.uiKey] = colDef.BuildColumn()
637 + }
638 + return columns
639 +}
640 +
641 +// collectTopQueries queries Query Store for top queries using dynamic columns
642 +func (c *Collector) collectTopQueries(ctx context.Context, sortColumn string) *module.FunctionResponse {
643 + // Check if function is enabled
644 + if !c.Config.GetQueryStoreFunctionEnabled() {
645 + return &module.FunctionResponse{
646 + Status: 403,
647 + Message: "Query Store function has been disabled in configuration. " +
648 + "To enable, set query_store_function_enabled: true in the MSSQL collector config.",
649 + }
650 + }
651 +
652 + // Detect available columns
653 + availableCols, err := c.detectMSSQLQueryStoreColumns(ctx)
654 + if err != nil {
655 + return &module.FunctionResponse{
656 + Status: 500,
657 + Message: fmt.Sprintf("failed to detect available columns: %v", err),
658 + }
659 + }
660 +
661 + // Build list of available columns
662 + cols := c.buildAvailableMSSQLColumns(availableCols)
663 + if len(cols) == 0 {
664 + return &module.FunctionResponse{
665 + Status: 500,
666 + Message: "no columns available in Query Store",
667 + }
668 + }
669 +
670 + // Validate and map sort column (use filtered cols to ensure sort column is in SELECT)
671 + validatedSortColumn := c.mapAndValidateMSSQLSortColumn(sortColumn, cols)
672 +
673 + // Build and execute query
674 + timeWindowDays := c.Config.GetQueryStoreTimeWindowDays()
675 + limit := c.TopQueriesLimit
676 + if limit <= 0 {
677 + limit = 500
678 + }
679 + query := c.buildMSSQLDynamicSQL(cols, validatedSortColumn, timeWindowDays, limit)
680 +
681 + rows, err := c.db.QueryContext(ctx, query)
682 + if err != nil {
683 + if ctx.Err() == context.DeadlineExceeded {
684 + return &module.FunctionResponse{Status: 504, Message: "query timed out"}
685 + }
686 + // Include diagnostic info: which columns were detected and used
687 + colUIKeys := make([]string, len(cols))
688 + for i, col := range cols {
689 + colUIKeys[i] = col.uiKey
690 + }
691 + return &module.FunctionResponse{
692 + Status: 500,
693 + Message: fmt.Sprintf("query failed: %v (sort: %s, detected cols: %v)", err, validatedSortColumn, colUIKeys),
694 + }
695 + }
696 + defer rows.Close()
697 +
698 + // Scan rows dynamically
699 + data, err := c.scanMSSQLDynamicRows(rows, cols)
700 + if err != nil {
701 + return &module.FunctionResponse{Status: 500, Message: err.Error()}
702 + }
703 +
704 + // Build dynamic sort options from available columns (only those actually detected)
705 + sortParam, sortOptions := c.topQueriesSortParam(cols)
706 +
707 + // Find default sort column UI key
708 + defaultSort := ""
709 + for _, col := range cols {
710 + if col.isDefaultSort && col.isSortOption {
711 + defaultSort = col.uiKey
712 + break
713 + }
714 + }
715 + // Fallback to first sort option if no default
716 + if defaultSort == "" && len(sortOptions) > 0 {
717 + defaultSort = sortOptions[0].ID
718 + }
719 +
720 + return &module.FunctionResponse{
721 + Status: 200,
722 + Help: "Top SQL queries from Query Store. WARNING: Query text may contain unmasked literals (potential PII).",
723 + Columns: c.buildMSSQLDynamicColumns(cols),
724 + Data: data,
725 + DefaultSortColumn: defaultSort,
726 + RequiredParams: []funcapi.ParamConfig{sortParam},
727 +
728 + // Charts for aggregated visualization
729 + Charts: map[string]module.ChartConfig{
730 + "Calls": {
731 + Name: "Number of Calls",
732 + Type: "stacked-bar",
733 + Columns: []string{"calls"},
734 + },
735 + "Time": {
736 + Name: "Execution Time",
737 + Type: "stacked-bar",
738 + Columns: []string{"totalTime", "avgTime"},
739 + },
740 + "CPU": {
741 + Name: "CPU Time",
742 + Type: "stacked-bar",
743 + Columns: []string{"avgCpu"},
744 + },
745 + "IO": {
746 + Name: "Logical I/O",
747 + Type: "stacked-bar",
748 + Columns: []string{"avgReads", "avgWrites"},
749 + },
750 + },
751 + DefaultCharts: [][]string{
752 + {"Time", "database"},
753 + {"Calls", "database"},
754 + },
755 + GroupBy: map[string]module.GroupByConfig{
756 + "database": {
757 + Name: "Group by Database",
758 + Columns: []string{"database"},
759 + },
760 + },
761 + }
762 +}
src/go/plugin/go.d/collector/mssql/functions_test.go new
+321
@@ -0,0 +1,321 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package mssql
4 +
5 +import (
6 + "testing"
7 +
8 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
9 + "github.com/stretchr/testify/assert"
10 +)
11 +
12 +func TestMssqlMethods(t *testing.T) {
13 + methods := mssqlMethods()
14 +
15 + require := assert.New(t)
16 + require.Len(methods, 1)
17 + require.Equal("top-queries", methods[0].ID)
18 + require.Equal("Top Queries", methods[0].Name)
19 + require.NotEmpty(methods[0].RequiredParams)
20 +
21 + // Verify at least one default sort option exists
22 + var sortParam *funcapi.ParamConfig
23 + for i := range methods[0].RequiredParams {
24 + if methods[0].RequiredParams[i].ID == "__sort" {
25 + sortParam = &methods[0].RequiredParams[i]
26 + break
27 + }
28 + }
29 + require.NotNil(sortParam, "expected __sort required param")
30 + require.NotEmpty(sortParam.Options)
31 +
32 + hasDefault := false
33 + for _, opt := range sortParam.Options {
34 + if opt.Default {
35 + hasDefault = true
36 + require.Equal("totalTime", opt.ID) // camelCase for UI
37 + break
38 + }
39 + }
40 + require.True(hasDefault, "should have a default sort option")
41 +}
42 +
43 +func TestMssqlAllColumns_HasRequiredColumns(t *testing.T) {
44 + // Verify all required base columns are defined
45 + requiredUIKeys := []string{
46 + "queryHash", "query", "database", "calls",
47 + "totalTime", "avgTime", "avgCpu",
48 + "avgReads", "avgWrites",
49 + }
50 +
51 + uiKeys := make(map[string]bool)
52 + for _, col := range mssqlAllColumns {
53 + uiKeys[col.uiKey] = true
54 + }
55 +
56 + for _, key := range requiredUIKeys {
57 + assert.True(t, uiKeys[key], "column %s should be defined in mssqlAllColumns", key)
58 + }
59 +}
60 +
61 +func TestMssqlAllColumns_HasValidMetadata(t *testing.T) {
62 + for _, col := range mssqlAllColumns {
63 + // Every column must have a UI key
64 + assert.NotEmpty(t, col.uiKey, "column %s must have uiKey", col.dbColumn)
65 +
66 + // Every column must have a display name
67 + assert.NotEmpty(t, col.displayName, "column %s must have displayName", col.uiKey)
68 +
69 + // Every column must have a data type
70 + assert.NotEqual(t, funcapi.FieldTypeNone, col.dataType, "column %s must have dataType", col.uiKey)
71 +
72 + // Duration columns must have units
73 + if col.dataType == ftDuration {
74 + assert.NotEmpty(t, col.units, "duration column %s must have units", col.uiKey)
75 + }
76 +
77 + // Sort options must have labels
78 + if col.isSortOption {
79 + assert.NotEmpty(t, col.sortLabel, "sort option column %s must have sortLabel", col.uiKey)
80 + }
81 + }
82 +}
83 +
84 +func TestCollector_mapAndValidateMSSQLSortColumn(t *testing.T) {
85 + // Build a filtered column list with common available columns
86 + availableCols := map[string]bool{"avg_duration": true, "count_executions": true}
87 + c := &Collector{}
88 + cols := c.buildAvailableMSSQLColumns(availableCols)
89 +
90 + tests := map[string]struct {
91 + cols []mssqlColumnMeta
92 + input string
93 + expected string
94 + }{
95 + "totalTime maps correctly": {
96 + cols: cols,
97 + input: "totalTime",
98 + expected: "totalTime",
99 + },
100 + "calls maps correctly": {
101 + cols: cols,
102 + input: "calls",
103 + expected: "calls",
104 + },
105 + "invalid column falls back to first sort option": {
106 + cols: cols,
107 + input: "invalid_column",
108 + expected: "calls", // first sort option in filtered cols
109 + },
110 + "SQL injection attempt falls back to first sort option": {
111 + cols: cols,
112 + input: "'; DROP TABLE users;--",
113 + expected: "calls", // first sort option in filtered cols
114 + },
115 + }
116 +
117 + for name, tc := range tests {
118 + t.Run(name, func(t *testing.T) {
119 + result := c.mapAndValidateMSSQLSortColumn(tc.input, tc.cols)
120 + assert.Equal(t, tc.expected, result)
121 + })
122 + }
123 +}
124 +
125 +func TestCollector_buildAvailableMSSQLColumns(t *testing.T) {
126 + tests := map[string]struct {
127 + availableCols map[string]bool
128 + expectCols []string // UI keys we expect to see
129 + notExpectCols []string // UI keys we don't expect
130 + }{
131 + "SQL Server 2016 columns": {
132 + availableCols: map[string]bool{
133 + "count_executions": true,
134 + "avg_duration": true, "last_duration": true, "min_duration": true, "max_duration": true,
135 + "avg_cpu_time": true, "last_cpu_time": true, "min_cpu_time": true, "max_cpu_time": true,
136 + "avg_logical_io_reads": true, "avg_logical_io_writes": true,
137 + },
138 + expectCols: []string{"queryHash", "query", "database", "calls", "totalTime", "avgTime", "avgCpu", "avgReads"},
139 + notExpectCols: []string{"avgLogBytes", "avgTempdb"}, // SQL Server 2017+ only
140 + },
141 + "SQL Server 2017 with log bytes and tempdb": {
142 + availableCols: map[string]bool{
143 + "count_executions": true,
144 + "avg_duration": true,
145 + "avg_cpu_time": true,
146 + "avg_log_bytes_used": true,
147 + "avg_tempdb_space_used": true,
148 + },
149 + expectCols: []string{"queryHash", "query", "calls", "avgLogBytes", "avgTempdb"},
150 + },
151 + }
152 +
153 + for name, tc := range tests {
154 + t.Run(name, func(t *testing.T) {
155 + c := &Collector{}
156 + cols := c.buildAvailableMSSQLColumns(tc.availableCols)
157 +
158 + // Build map of UI keys for easy lookup
159 + uiKeys := make(map[string]bool)
160 + for _, col := range cols {
161 + uiKeys[col.uiKey] = true
162 + }
163 +
164 + for _, key := range tc.expectCols {
165 + assert.True(t, uiKeys[key], "expected column %s to be present", key)
166 + }
167 + for _, key := range tc.notExpectCols {
168 + assert.False(t, uiKeys[key], "did not expect column %s to be present", key)
169 + }
170 + })
171 + }
172 +}
173 +
174 +func TestCollector_buildMSSQLDynamicSQL(t *testing.T) {
175 + c := &Collector{}
176 +
177 + cols := []mssqlColumnMeta{
178 + {dbColumn: "query_hash", uiKey: "queryHash", dataType: ftString, isIdentity: true},
179 + {dbColumn: "query_sql_text", uiKey: "query", dataType: ftString, isIdentity: true},
180 + {dbColumn: "database_name", uiKey: "database", dataType: ftString, isIdentity: true},
181 + {dbColumn: "count_executions", uiKey: "calls", dataType: ftInteger},
182 + {dbColumn: "avg_duration", uiKey: "totalTime", dataType: ftDuration, isMicroseconds: true},
183 + }
184 +
185 + // sortColumn is the uiKey which is also used as the SQL alias
186 + sql := c.buildMSSQLDynamicSQL(cols, "totalTime", 7, 500)
187 +
188 + // Basic query structure
189 + assert.Contains(t, sql, "sys.query_store_query")
190 + assert.Contains(t, sql, "AS [totalTime]")
191 + assert.Contains(t, sql, "ORDER BY [totalTime] DESC")
192 + assert.Contains(t, sql, "TOP 500")
193 + assert.Contains(t, sql, "DATEADD")
194 + assert.Contains(t, sql, "q.query_hash")
195 +
196 + // Cross-database aggregation features
197 + assert.Contains(t, sql, "QUOTENAME(name)") // Safe database name escaping
198 + assert.Contains(t, sql, "UNION ALL") // Combining results from multiple databases
199 + assert.Contains(t, sql, "sp_executesql") // Executing dynamic SQL
200 + assert.Contains(t, sql, "sys.databases") // Finding databases with Query Store
201 + assert.Contains(t, sql, "is_query_store_on = 1") // Condition for Query Store enabled
202 + assert.Contains(t, sql, "NOT IN ('master', 'tempdb', 'model', 'msdb')") // Excluding system databases
203 +}
204 +
205 +func TestCollector_buildMSSQLDynamicSQL_NoTimeFilter(t *testing.T) {
206 + c := &Collector{}
207 +
208 + cols := []mssqlColumnMeta{
209 + {dbColumn: "query_hash", uiKey: "queryHash", dataType: ftString, isIdentity: true},
210 + {dbColumn: "count_executions", uiKey: "calls", dataType: ftInteger},
211 + }
212 +
213 + sql := c.buildMSSQLDynamicSQL(cols, "calls", 0, 500)
214 +
215 + assert.Contains(t, sql, "sys.query_store_query")
216 + assert.Contains(t, sql, "TOP 500")
217 + assert.Contains(t, sql, "ORDER BY [calls] DESC")
218 + assert.NotContains(t, sql, "DATEADD")
219 +}
220 +
221 +func TestCollector_buildMSSQLDynamicColumns(t *testing.T) {
222 + c := &Collector{}
223 +
224 + cols := []mssqlColumnMeta{
225 + {uiKey: "queryHash", displayName: "Query Hash", dataType: ftString, visible: false, isUniqueKey: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
226 + {uiKey: "query", displayName: "Query", dataType: ftString, visible: true, isSticky: true, fullWidth: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
227 + {uiKey: "totalTime", displayName: "Total Time", dataType: ftDuration, units: "seconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
228 + }
229 +
230 + columns := c.buildMSSQLDynamicColumns(cols)
231 +
232 + // Verify column count
233 + assert.Len(t, columns, 3)
234 +
235 + // Verify queryHash column
236 + queryHashCol := columns["queryHash"].(map[string]any)
237 + assert.Equal(t, "Query Hash", queryHashCol["name"])
238 + assert.Equal(t, "string", queryHashCol["type"])
239 + assert.True(t, queryHashCol["unique_key"].(bool))
240 + assert.False(t, queryHashCol["visible"].(bool))
241 + assert.Equal(t, 0, queryHashCol["index"])
242 +
243 + // Verify query column
244 + queryCol := columns["query"].(map[string]any)
245 + assert.Equal(t, "Query", queryCol["name"])
246 + assert.True(t, queryCol["sticky"].(bool))
247 + assert.True(t, queryCol["full_width"].(bool))
248 + assert.Equal(t, 1, queryCol["index"])
249 +
250 + // Verify totalTime column
251 + totalTimeCol := columns["totalTime"].(map[string]any)
252 + assert.Equal(t, "Total Time", totalTimeCol["name"])
253 + assert.Equal(t, "duration", totalTimeCol["type"])
254 + assert.Equal(t, "seconds", totalTimeCol["units"])
255 + assert.Equal(t, "bar", totalTimeCol["visualization"]) // duration uses bar
256 + assert.Equal(t, 2, totalTimeCol["index"])
257 +}
258 +
259 +// Test that method config sort options have valid column references
260 +func TestMssqlMethods_SortOptionsHaveLabels(t *testing.T) {
261 + methods := mssqlMethods()
262 +
263 + for _, method := range methods {
264 + var sortParam *funcapi.ParamConfig
265 + for i := range method.RequiredParams {
266 + if method.RequiredParams[i].ID == "__sort" {
267 + sortParam = &method.RequiredParams[i]
268 + break
269 + }
270 + }
271 + assert.NotNil(t, sortParam)
272 + for _, opt := range sortParam.Options {
273 + assert.NotEmpty(t, opt.ID, "sort option must have ID")
274 + assert.NotEmpty(t, opt.Name, "sort option %s must have Name", opt.ID)
275 + assert.Contains(t, opt.Name, "Top queries by", "label should have standard prefix")
276 + }
277 + }
278 +}
279 +
280 +// TestMapAndValidateMSSQLSortColumn_NoSortOptions verifies fallback when no sort columns exist
281 +func TestMapAndValidateMSSQLSortColumn_NoSortOptions(t *testing.T) {
282 + c := &Collector{}
283 +
284 + // Only identity columns available (no sort options)
285 + identityOnlyCols := []mssqlColumnMeta{
286 + {uiKey: "queryHash", isIdentity: true},
287 + {uiKey: "query", isIdentity: true},
288 + {uiKey: "database", isIdentity: true},
289 + }
290 +
291 + result := c.mapAndValidateMSSQLSortColumn("totalTime", identityOnlyCols)
292 + // Should fall back to first column since no sort options exist
293 + assert.Equal(t, "queryHash", result, "should fall back to first column when no sort options")
294 +
295 + // Empty columns list
296 + result = c.mapAndValidateMSSQLSortColumn("totalTime", []mssqlColumnMeta{})
297 + assert.Equal(t, "", result, "should return empty string when no columns available")
298 +}
299 +
300 +// TestSortColumnValidation_SQLInjection verifies that SQL injection attempts
301 +// are handled by the validation mechanism
302 +func TestMssqlSortColumnValidation_SQLInjection(t *testing.T) {
303 + c := &Collector{}
304 + availableCols := map[string]bool{"avg_duration": true, "count_executions": true}
305 + cols := c.buildAvailableMSSQLColumns(availableCols)
306 +
307 + maliciousInputs := []string{
308 + "'; DROP TABLE sys.query_store_query; --",
309 + "total_time_ms; DELETE FROM master.dbo.sysdatabases",
310 + "1 OR 1=1",
311 + "WAITFOR DELAY '00:00:10'",
312 + "xp_cmdshell 'whoami'",
313 + }
314 +
315 + for _, input := range maliciousInputs {
316 + result := c.mapAndValidateMSSQLSortColumn(input, cols)
317 + // All malicious inputs should fall back to first available sort option
318 + assert.Equal(t, "calls", result,
319 + "malicious input should fall back to safe default: %s -> %s", input, result)
320 + }
321 +}
src/go/plugin/go.d/collector/mssql/metadata.yaml
+28
@@ -136,6 +136,22 @@ modules:
136 required: false
137 group: Target
138
139 + - name: query_store_function_enabled
140 + description: |
141 + Enable the Query Store function to expose top queries via Netdata Functions.
142 + **WARNING**: Query Store may contain unmasked literal values (customer names, emails, IDs).
143 + Only enable after ensuring proper access controls to the Netdata dashboard.
144 + default_value: false
145 + required: false
146 + group: Query Store
147 + - name: query_store_time_window_days
148 + description: |
149 + Number of days of Query Store data to analyze. Set to 0 to include all available data.
150 + Smaller values improve query performance but show less history.
151 + default_value: 7
152 + required: false
153 + group: Query Store
154 +
155 - name: vnode
156 description: Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes).
157 default_value: ""
@@ -182,6 +198,18 @@ modules:
198
199 - name: development
200 dsn: "sqlserver://netdata_user:password@dev-sql:1433"
201 + - name: With Query Store function
202 + description: |
203 + Enable the Query Store function to view top queries in the Netdata dashboard.
204 +
205 + > **Warning**: Query Store may contain unmasked literal values (PII).
206 + > Only enable in environments with proper access controls.
207 + config: |
208 + jobs:
209 + - name: local
210 + dsn: "sqlserver://netdata_user:password@localhost:1433"
211 + query_store_function_enabled: true
212 + query_store_time_window_days: 7
213 troubleshooting:
214 problems:
215 list:
src/go/plugin/go.d/collector/mysql/collect_global_vars.go
+2
@@ -39,7 +39,9 @@ func (c *Collector) collectGlobalVariables() error {
39 case "max_connections":
40 c.varMaxConns = parseInt(value)
41 case "performance_schema":
42 + c.varPerfSchemaMu.Lock()
43 c.varPerformanceSchema = value
44 + c.varPerfSchemaMu.Unlock()
45 case "table_open_cache":
46 c.varTableOpenCache = parseInt(value)
47 }
src/go/plugin/go.d/collector/mysql/collect_process_list.go
+4 -1
@@ -42,7 +42,10 @@ ORDER BY
42 func (c *Collector) collectProcessListStatistics(mx map[string]int64) error {
43 var q string
44 mysqlMinVer := semver.Version{Major: 8, Minor: 0, Patch: 22}
45 - if !c.isMariaDB && c.version.GTE(mysqlMinVer) && c.varPerformanceSchema == "ON" {
45 + c.varPerfSchemaMu.RLock()
46 + perfSchema := c.varPerformanceSchema
47 + c.varPerfSchemaMu.RUnlock()
48 + if !c.isMariaDB && c.version.GTE(mysqlMinVer) && perfSchema == "ON" {
49 q = queryShowProcessListPS
50 } else {
51 q = queryShowProcessList
src/go/plugin/go.d/collector/mysql/collector.go
+8
@@ -27,6 +27,9 @@ func init() {
27 JobConfigSchema: configSchema,
28 Create: func() module.Module { return New() },
29 Config: func() any { return &Config{} },
30 + Methods: mysqlMethods,
31 + MethodParams: mysqlMethodParams,
32 + HandleMethod: mysqlHandleMethod,
33 })
34 }
35
@@ -67,6 +70,7 @@ type Config struct {
70 DSN string `yaml:"dsn" json:"dsn"`
71 MyCNF string `yaml:"my.cnf,omitempty" json:"my.cnf"`
72 Timeout confopt.Duration `yaml:"timeout,omitempty" json:"timeout"`
73 + TopQueriesLimit int `yaml:"top_queries_limit,omitempty" json:"top_queries_limit,omitempty"`
74 }
75
76 type Collector struct {
@@ -105,6 +109,10 @@ type Collector struct {
109 varDisabledStorageEngine string
110 varLogBin string
111 varPerformanceSchema string
112 + varPerfSchemaMu sync.RWMutex // protects varPerformanceSchema for concurrent access
113 +
114 + stmtSummaryCols map[string]bool // cached column names from events_statements_summary_by_digest
115 + stmtSummaryColsMu sync.RWMutex // protects stmtSummaryCols for concurrent access
116 }
117
118 func (c *Collector) Configuration() any {
src/go/plugin/go.d/collector/mysql/config_schema.json
+8
@@ -40,6 +40,14 @@
40 "title": "Vnode",
41 "description": "Associates this data collection job with a [Virtual Node](https://learn.netdata.cloud/docs/netdata-agent/configuration/organize-systems-metrics-and-alerts#virtual-nodes).",
42 "type": "string"
43 + },
44 + "top_queries_limit": {
45 + "title": "Top Queries Limit",
46 + "description": "Maximum number of queries to return in the top-queries function response.",
47 + "type": "integer",
48 + "minimum": 1,
49 + "maximum": 5000,
50 + "default": 500
51 }
52 },
53 "required": [
src/go/plugin/go.d/collector/mysql/functions.go new
+635
@@ -0,0 +1,635 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package mysql
4 +
5 +import (
6 + "context"
7 + "database/sql"
8 + "fmt"
9 + "strings"
10 +
11 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
12 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
13 + "github.com/netdata/netdata/go/plugins/plugin/go.d/pkg/strmutil"
14 +)
15 +
16 +const maxQueryTextLength = 4096
17 +
18 +const (
19 + paramSort = "__sort"
20 +
21 + ftString = funcapi.FieldTypeString
22 + ftInteger = funcapi.FieldTypeInteger
23 + ftDuration = funcapi.FieldTypeDuration
24 +
25 + trNone = funcapi.FieldTransformNone
26 + trNumber = funcapi.FieldTransformNumber
27 + trDuration = funcapi.FieldTransformDuration
28 +
29 + sortAsc = funcapi.FieldSortAscending
30 + sortDesc = funcapi.FieldSortDescending
31 +
32 + summaryCount = funcapi.FieldSummaryCount
33 + summarySum = funcapi.FieldSummarySum
34 + summaryMin = funcapi.FieldSummaryMin
35 + summaryMax = funcapi.FieldSummaryMax
36 + summaryMean = funcapi.FieldSummaryMean
37 +
38 + filterMulti = funcapi.FieldFilterMultiselect
39 + filterRange = funcapi.FieldFilterRange
40 +)
41 +
42 +// mysqlColumnMeta defines metadata for a single column
43 +type mysqlColumnMeta struct {
44 + dbColumn string // Column name in database (e.g., "SUM_TIMER_WAIT")
45 + uiKey string // Canonical name used everywhere: SQL alias, UI key, sort key
46 + displayName string // Display name in UI (e.g., "Total Time")
47 + dataType funcapi.FieldType // "string", "integer", "float", "duration"
48 + units string // Unit for duration/numeric types (e.g., "seconds")
49 + visible bool // Default visibility
50 + transform funcapi.FieldTransform // Transform for value_options (e.g., "duration", "number", "none")
51 + decimalPoints int // Decimal points for display
52 + sortDir funcapi.FieldSort // Sort direction: "ascending" or "descending"
53 + summary funcapi.FieldSummary // Summary function: "sum", "count", "max", "min", "mean" (UI aggregations)
54 + filter funcapi.FieldFilter // Filter type: "multiselect" or "range"
55 + isPicoseconds bool // Needs picoseconds to seconds conversion
56 + isSortOption bool // Show in sort dropdown
57 + sortLabel string // Label for sort option
58 + isDefaultSort bool // Is this the default sort option
59 + isUniqueKey bool // Is this column a unique key
60 + isSticky bool // Is this column sticky in UI
61 + fullWidth bool // Should column take full width
62 +}
63 +
64 +// mysqlAllColumns defines ALL possible columns from events_statements_summary_by_digest
65 +// Columns that don't exist in certain MySQL/MariaDB versions will be filtered at runtime
66 +var mysqlAllColumns = []mysqlColumnMeta{
67 + // Identity columns - always available
68 + {dbColumn: "DIGEST", uiKey: "digest", displayName: "Digest", dataType: ftString, visible: false, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, isUniqueKey: true},
69 + {dbColumn: "DIGEST_TEXT", uiKey: "query", displayName: "Query", dataType: ftString, visible: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, isSticky: true, fullWidth: true},
70 + {dbColumn: "SCHEMA_NAME", uiKey: "schema", displayName: "Schema", dataType: ftString, visible: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
71 +
72 + // Execution counts
73 + {dbColumn: "COUNT_STAR", uiKey: "calls", displayName: "Calls", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Number of Calls"},
74 +
75 + // Timer metrics (picoseconds -> seconds)
76 + {dbColumn: "SUM_TIMER_WAIT", uiKey: "totalTime", displayName: "Total Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange, isPicoseconds: true, isSortOption: true, sortLabel: "Top queries by Total Execution Time", isDefaultSort: true},
77 + {dbColumn: "MIN_TIMER_WAIT", uiKey: "minTime", displayName: "Min Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMin, filter: filterRange, isPicoseconds: true},
78 + {dbColumn: "AVG_TIMER_WAIT", uiKey: "avgTime", displayName: "Avg Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMean, filter: filterRange, isPicoseconds: true, isSortOption: true, sortLabel: "Top queries by Average Execution Time"},
79 + {dbColumn: "MAX_TIMER_WAIT", uiKey: "maxTime", displayName: "Max Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isPicoseconds: true},
80 +
81 + // Lock time (picoseconds -> seconds)
82 + {dbColumn: "SUM_LOCK_TIME", uiKey: "lockTime", displayName: "Lock Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange, isPicoseconds: true, isSortOption: true, sortLabel: "Top queries by Lock Time"},
83 +
84 + // Error and warning counts
85 + {dbColumn: "SUM_ERRORS", uiKey: "errors", displayName: "Errors", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Errors"},
86 + {dbColumn: "SUM_WARNINGS", uiKey: "warnings", displayName: "Warnings", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Warnings"},
87 +
88 + // Row operations
89 + {dbColumn: "SUM_ROWS_AFFECTED", uiKey: "rowsAffected", displayName: "Rows Affected", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Rows Affected"},
90 + {dbColumn: "SUM_ROWS_SENT", uiKey: "rowsSent", displayName: "Rows Sent", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Rows Sent"},
91 + {dbColumn: "SUM_ROWS_EXAMINED", uiKey: "rowsExamined", displayName: "Rows Examined", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Rows Examined"},
92 +
93 + // Temp table usage
94 + {dbColumn: "SUM_CREATED_TMP_DISK_TABLES", uiKey: "tmpDiskTables", displayName: "Temp Disk Tables", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Temp Disk Tables"},
95 + {dbColumn: "SUM_CREATED_TMP_TABLES", uiKey: "tmpTables", displayName: "Temp Tables", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Temp Tables"},
96 +
97 + // Join operations
98 + {dbColumn: "SUM_SELECT_FULL_JOIN", uiKey: "fullJoin", displayName: "Full Joins", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Full Joins"},
99 + {dbColumn: "SUM_SELECT_FULL_RANGE_JOIN", uiKey: "fullRangeJoin", displayName: "Full Range Joins", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
100 + {dbColumn: "SUM_SELECT_RANGE", uiKey: "selectRange", displayName: "Select Range", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
101 + {dbColumn: "SUM_SELECT_RANGE_CHECK", uiKey: "selectRangeCheck", displayName: "Select Range Check", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
102 + {dbColumn: "SUM_SELECT_SCAN", uiKey: "selectScan", displayName: "Select Scan", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Table Scans"},
103 +
104 + // Sort operations
105 + {dbColumn: "SUM_SORT_MERGE_PASSES", uiKey: "sortMergePasses", displayName: "Sort Merge Passes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
106 + {dbColumn: "SUM_SORT_RANGE", uiKey: "sortRange", displayName: "Sort Range", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
107 + {dbColumn: "SUM_SORT_ROWS", uiKey: "sortRows", displayName: "Sort Rows", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Rows Sorted"},
108 + {dbColumn: "SUM_SORT_SCAN", uiKey: "sortScan", displayName: "Sort Scan", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
109 +
110 + // Index usage
111 + {dbColumn: "SUM_NO_INDEX_USED", uiKey: "noIndexUsed", displayName: "No Index Used", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Top queries by No Index Used"},
112 + {dbColumn: "SUM_NO_GOOD_INDEX_USED", uiKey: "noGoodIndexUsed", displayName: "No Good Index Used", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
113 +
114 + // Timestamp columns
115 + {dbColumn: "FIRST_SEEN", uiKey: "firstSeen", displayName: "First Seen", dataType: ftString, visible: false, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
116 + {dbColumn: "LAST_SEEN", uiKey: "lastSeen", displayName: "Last Seen", dataType: ftString, visible: false, transform: trNone, sortDir: sortDesc, summary: summaryCount, filter: filterMulti},
117 +
118 + // MySQL 8.0+ quantile columns
119 + {dbColumn: "QUANTILE_95", uiKey: "p95Time", displayName: "P95 Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isPicoseconds: true, isSortOption: true, sortLabel: "Top queries by 95th Percentile Time"},
120 + {dbColumn: "QUANTILE_99", uiKey: "p99Time", displayName: "P99 Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isPicoseconds: true, isSortOption: true, sortLabel: "Top queries by 99th Percentile Time"},
121 + {dbColumn: "QUANTILE_999", uiKey: "p999Time", displayName: "P99.9 Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isPicoseconds: true},
122 +
123 + // MySQL 8.0+ sample query
124 + {dbColumn: "QUERY_SAMPLE_TEXT", uiKey: "sampleQuery", displayName: "Sample Query", dataType: ftString, visible: false, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, fullWidth: true},
125 + {dbColumn: "QUERY_SAMPLE_SEEN", uiKey: "sampleSeen", displayName: "Sample Seen", dataType: ftString, visible: false, transform: trNone, sortDir: sortDesc, summary: summaryCount, filter: filterMulti},
126 + {dbColumn: "QUERY_SAMPLE_TIMER_WAIT", uiKey: "sampleTime", displayName: "Sample Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isPicoseconds: true},
127 +
128 + // MySQL 8.0.28+ CPU time
129 + {dbColumn: "SUM_CPU_TIME", uiKey: "cpuTime", displayName: "CPU Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange, isPicoseconds: true, isSortOption: true, sortLabel: "Top queries by CPU Time"},
130 +
131 + // MySQL 8.0.31+ memory columns
132 + {dbColumn: "MAX_CONTROLLED_MEMORY", uiKey: "maxControlledMemory", displayName: "Max Controlled Memory", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Max Controlled Memory"},
133 + {dbColumn: "MAX_TOTAL_MEMORY", uiKey: "maxTotalMemory", displayName: "Max Total Memory", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isSortOption: true, sortLabel: "Top queries by Max Total Memory"},
134 +}
135 +
136 +// mysqlMethods returns the available function methods for MySQL
137 +func mysqlMethods() []module.MethodConfig {
138 + // Build sort options from column metadata
139 + var sortOptions []funcapi.ParamOption
140 + sortDir := funcapi.FieldSortDescending
141 + for _, col := range mysqlAllColumns {
142 + if col.isSortOption {
143 + sortOptions = append(sortOptions, funcapi.ParamOption{
144 + ID: col.uiKey,
145 + Column: col.dbColumn,
146 + Name: col.sortLabel,
147 + Default: col.isDefaultSort,
148 + Sort: &sortDir,
149 + })
150 + }
151 + }
152 +
153 + return []module.MethodConfig{{
154 + ID: "top-queries",
155 + Name: "Top Queries",
156 + Help: "Top SQL queries from performance_schema",
157 + RequiredParams: []funcapi.ParamConfig{
158 + {
159 + ID: paramSort,
160 + Name: "Filter By",
161 + Help: "Select the primary sort column",
162 + Selection: funcapi.ParamSelect,
163 + Options: sortOptions,
164 + UniqueView: true,
165 + },
166 + },
167 + }}
168 +}
169 +
170 +func mysqlMethodParams(ctx context.Context, job *module.Job, method string) ([]funcapi.ParamConfig, error) {
171 + collector, ok := job.Module().(*Collector)
172 + if !ok {
173 + return nil, fmt.Errorf("invalid module type")
174 + }
175 + if collector.db == nil {
176 + return nil, fmt.Errorf("collector is still initializing")
177 + }
178 + switch method {
179 + case "top-queries":
180 + return collector.topQueriesParams(ctx)
181 + default:
182 + return nil, fmt.Errorf("unknown method: %s", method)
183 + }
184 +}
185 +
186 +// mysqlHandleMethod handles function requests for MySQL
187 +func mysqlHandleMethod(ctx context.Context, job *module.Job, method string, params funcapi.ResolvedParams) *module.FunctionResponse {
188 + collector, ok := job.Module().(*Collector)
189 + if !ok {
190 + return &module.FunctionResponse{Status: 500, Message: "internal error: invalid module type"}
191 + }
192 +
193 + // Check if collector is initialized (first collect() may not have run yet)
194 + if collector.db == nil {
195 + return &module.FunctionResponse{
196 + Status: 503,
197 + Message: "collector is still initializing, please retry in a few seconds",
198 + }
199 + }
200 +
201 + switch method {
202 + case "top-queries":
203 + return collector.collectTopQueries(ctx, params.Column(paramSort))
204 + default:
205 + return &module.FunctionResponse{Status: 404, Message: fmt.Sprintf("unknown method: %s", method)}
206 + }
207 +}
208 +
209 +// detectMySQLStatementsColumns queries the database to discover available columns
210 +func (c *Collector) detectMySQLStatementsColumns(ctx context.Context) (map[string]bool, error) {
211 + // Fast path: return cached result
212 + c.stmtSummaryColsMu.RLock()
213 + if c.stmtSummaryCols != nil {
214 + cols := c.stmtSummaryCols
215 + c.stmtSummaryColsMu.RUnlock()
216 + return cols, nil
217 + }
218 + c.stmtSummaryColsMu.RUnlock()
219 +
220 + // Slow path: query and cache
221 + c.stmtSummaryColsMu.Lock()
222 + defer c.stmtSummaryColsMu.Unlock()
223 +
224 + // Double-check after acquiring write lock
225 + if c.stmtSummaryCols != nil {
226 + return c.stmtSummaryCols, nil
227 + }
228 +
229 + // Query information_schema to get available columns
230 + query := `
231 + SELECT COLUMN_NAME
232 + FROM information_schema.COLUMNS
233 + WHERE TABLE_SCHEMA = 'performance_schema'
234 + AND TABLE_NAME = 'events_statements_summary_by_digest'
235 + `
236 + rows, err := c.db.QueryContext(ctx, query)
237 + if err != nil {
238 + return nil, fmt.Errorf("failed to query column information: %w", err)
239 + }
240 + defer rows.Close()
241 +
242 + cols := make(map[string]bool)
243 + for rows.Next() {
244 + var colName string
245 + if err := rows.Scan(&colName); err != nil {
246 + return nil, fmt.Errorf("failed to scan column name: %w", err)
247 + }
248 + cols[colName] = true
249 + }
250 +
251 + if err := rows.Err(); err != nil {
252 + return nil, fmt.Errorf("error iterating columns: %w", err)
253 + }
254 +
255 + // Cache the result
256 + c.stmtSummaryCols = cols
257 +
258 + return cols, nil
259 +}
260 +
261 +// buildAvailableColumns filters columns based on what's available in the database
262 +func (c *Collector) buildAvailableMySQLColumns(availableCols map[string]bool) []mysqlColumnMeta {
263 + var cols []mysqlColumnMeta
264 + for _, col := range mysqlAllColumns {
265 + if availableCols[col.dbColumn] {
266 + cols = append(cols, col)
267 + }
268 + }
269 + return cols
270 +}
271 +
272 +// mapAndValidateMySQLSortColumn validates sort key and returns the uiKey to use
273 +func (c *Collector) mapAndValidateMySQLSortColumn(sortKey string, availableCols map[string]bool) string {
274 + // Find the column by uiKey or dbColumn
275 + for _, col := range mysqlAllColumns {
276 + if (col.uiKey == sortKey || col.dbColumn == sortKey) && availableCols[col.dbColumn] {
277 + return col.uiKey
278 + }
279 + }
280 + // Default to totalTime if available
281 + if availableCols["SUM_TIMER_WAIT"] {
282 + return "totalTime"
283 + }
284 + return "calls" // Ultimate fallback
285 +}
286 +
287 +// buildMySQLDynamicSQL builds the SQL query with only available columns
288 +func (c *Collector) buildMySQLDynamicSQL(cols []mysqlColumnMeta, sortColumn string, limit int) string {
289 + var selectParts []string
290 + for _, col := range cols {
291 + // Use backticks to handle reserved keywords
292 + if col.isPicoseconds {
293 + // Convert picoseconds to milliseconds (divide by 10^9)
294 + selectParts = append(selectParts, fmt.Sprintf("%s/1000000000 AS `%s`", col.dbColumn, col.uiKey))
295 + } else if col.dbColumn == "SCHEMA_NAME" {
296 + selectParts = append(selectParts, fmt.Sprintf("IFNULL(%s, '') AS `%s`", col.dbColumn, col.uiKey))
297 + } else {
298 + selectParts = append(selectParts, fmt.Sprintf("%s AS `%s`", col.dbColumn, col.uiKey))
299 + }
300 + }
301 +
302 + return fmt.Sprintf(`
303 +SELECT %s
304 +FROM performance_schema.events_statements_summary_by_digest
305 +WHERE DIGEST IS NOT NULL
306 +ORDER BY `+"`%s`"+` DESC
307 +LIMIT %d
308 +`, strings.Join(selectParts, ", "), sortColumn, limit)
309 +}
310 +
311 +// scanMySQLDynamicRows scans rows dynamically based on column types
312 +func (c *Collector) scanMySQLDynamicRows(rows mysqlRowScanner, cols []mysqlColumnMeta) ([][]any, error) {
313 + data := make([][]any, 0, 500)
314 +
315 + // Create value holders for scanning
316 + valuePtrs := make([]any, len(cols))
317 + values := make([]any, len(cols))
318 +
319 + for rows.Next() {
320 + // Reset value holders for each row
321 + for i, col := range cols {
322 + switch col.dataType {
323 + case ftString:
324 + var v sql.NullString
325 + values[i] = &v
326 + case ftInteger:
327 + var v sql.NullInt64
328 + values[i] = &v
329 + case ftDuration:
330 + var v sql.NullFloat64
331 + values[i] = &v
332 + default:
333 + var v any
334 + values[i] = &v
335 + }
336 + valuePtrs[i] = values[i]
337 + }
338 +
339 + if err := rows.Scan(valuePtrs...); err != nil {
340 + return nil, fmt.Errorf("row scan failed: %w", err)
341 + }
342 +
343 + // Convert scanned values to output format
344 + row := make([]any, len(cols))
345 + for i, col := range cols {
346 + switch v := values[i].(type) {
347 + case *sql.NullString:
348 + if v.Valid {
349 + s := v.String
350 + // Truncate query text
351 + if col.uiKey == "query" || col.uiKey == "sampleQuery" {
352 + s = strmutil.TruncateText(s, maxQueryTextLength)
353 + }
354 + row[i] = s
355 + } else {
356 + row[i] = ""
357 + }
358 + case *sql.NullInt64:
359 + if v.Valid {
360 + row[i] = v.Int64
361 + } else {
362 + row[i] = int64(0)
363 + }
364 + case *sql.NullFloat64:
365 + if v.Valid {
366 + row[i] = v.Float64
367 + } else {
368 + row[i] = float64(0)
369 + }
370 + default:
371 + row[i] = nil
372 + }
373 + }
374 + data = append(data, row)
375 + }
376 +
377 + if err := rows.Err(); err != nil {
378 + return nil, fmt.Errorf("rows iteration error: %w", err)
379 + }
380 +
381 + return data, nil
382 +}
383 +
384 +// buildMySQLDynamicColumns builds column definitions for the response
385 +func (c *Collector) buildMySQLDynamicColumns(cols []mysqlColumnMeta) map[string]any {
386 + columns := make(map[string]any)
387 + for i, col := range cols {
388 + visual := funcapi.FieldVisualValue
389 + if col.dataType == ftDuration {
390 + visual = funcapi.FieldVisualBar
391 + }
392 + colDef := funcapi.Column{
393 + Index: i,
394 + Name: col.displayName,
395 + Type: col.dataType,
396 + Units: col.units,
397 + Visualization: visual,
398 + Sort: col.sortDir,
399 + Sortable: true,
400 + Sticky: col.isSticky,
401 + Summary: col.summary,
402 + Filter: col.filter,
403 + FullWidth: col.fullWidth,
404 + Wrap: false,
405 + DefaultExpandedFilter: false,
406 + UniqueKey: col.isUniqueKey,
407 + Visible: col.visible,
408 + ValueOptions: funcapi.ValueOptions{
409 + Transform: col.transform,
410 + DecimalPoints: col.decimalPoints,
411 + DefaultValue: nil,
412 + },
413 + }
414 + columns[col.uiKey] = colDef.BuildColumn()
415 + }
416 + return columns
417 +}
418 +
419 +// buildMySQLDynamicSortOptions builds sort options from available columns
420 +// Returns only sort options for columns that actually exist in the database
421 +func (c *Collector) buildMySQLDynamicSortOptions(cols []mysqlColumnMeta) []funcapi.ParamOption {
422 + var sortOpts []funcapi.ParamOption
423 + seen := make(map[string]bool)
424 + sortDir := funcapi.FieldSortDescending
425 +
426 + for _, col := range cols {
427 + if col.isSortOption && !seen[col.uiKey] {
428 + seen[col.uiKey] = true
429 + sortOpts = append(sortOpts, funcapi.ParamOption{
430 + ID: col.uiKey,
431 + Column: col.dbColumn,
432 + Name: col.sortLabel,
433 + Default: col.isDefaultSort,
434 + Sort: &sortDir,
435 + })
436 + }
437 + }
438 + return sortOpts
439 +}
440 +
441 +func (c *Collector) topQueriesSortParam(cols []mysqlColumnMeta) (funcapi.ParamConfig, []funcapi.ParamOption) {
442 + sortOptions := c.buildMySQLDynamicSortOptions(cols)
443 + sortParam := funcapi.ParamConfig{
444 + ID: paramSort,
445 + Name: "Filter By",
446 + Help: "Select the primary sort column",
447 + Selection: funcapi.ParamSelect,
448 + Options: sortOptions,
449 + UniqueView: true,
450 + }
451 + return sortParam, sortOptions
452 +}
453 +
454 +func (c *Collector) topQueriesParams(ctx context.Context) ([]funcapi.ParamConfig, error) {
455 + available, err := c.checkPerformanceSchema(ctx)
456 + if err != nil {
457 + return nil, err
458 + }
459 + if !available {
460 + return nil, fmt.Errorf("performance_schema is not enabled")
461 + }
462 +
463 + availableCols, err := c.detectMySQLStatementsColumns(ctx)
464 + if err != nil {
465 + return nil, err
466 + }
467 + cols := c.buildAvailableMySQLColumns(availableCols)
468 + if len(cols) == 0 {
469 + return nil, fmt.Errorf("no columns available in events_statements_summary_by_digest")
470 + }
471 +
472 + sortParam, _ := c.topQueriesSortParam(cols)
473 + return []funcapi.ParamConfig{sortParam}, nil
474 +}
475 +
476 +// mysqlRowScanner interface for testing
477 +type mysqlRowScanner interface {
478 + Next() bool
479 + Scan(dest ...any) error
480 + Err() error
481 +}
482 +
483 +// collectTopQueries queries performance_schema for top queries using dynamic columns
484 +func (c *Collector) collectTopQueries(ctx context.Context, sortColumn string) *module.FunctionResponse {
485 + // Check if performance_schema is enabled
486 + available, err := c.checkPerformanceSchema(ctx)
487 + if err != nil {
488 + return &module.FunctionResponse{
489 + Status: 500,
490 + Message: fmt.Sprintf("failed to check performance_schema availability: %v", err),
491 + }
492 + }
493 + if !available {
494 + return &module.FunctionResponse{
495 + Status: 503,
496 + Message: "performance_schema is not enabled",
497 + }
498 + }
499 +
500 + // Detect available columns
501 + availableCols, err := c.detectMySQLStatementsColumns(ctx)
502 + if err != nil {
503 + return &module.FunctionResponse{
504 + Status: 500,
505 + Message: fmt.Sprintf("failed to detect available columns: %v", err),
506 + }
507 + }
508 +
509 + // Build list of available columns
510 + cols := c.buildAvailableMySQLColumns(availableCols)
511 + if len(cols) == 0 {
512 + return &module.FunctionResponse{
513 + Status: 500,
514 + Message: "no columns available in events_statements_summary_by_digest",
515 + }
516 + }
517 +
518 + // Validate and map sort column
519 + dbSortColumn := c.mapAndValidateMySQLSortColumn(sortColumn, availableCols)
520 +
521 + // Get query limit (default 500)
522 + limit := c.TopQueriesLimit
523 + if limit <= 0 {
524 + limit = 500
525 + }
526 +
527 + // Build and execute query
528 + query := c.buildMySQLDynamicSQL(cols, dbSortColumn, limit)
529 +
530 + rows, err := c.db.QueryContext(ctx, query)
531 + if err != nil {
532 + if ctx.Err() == context.DeadlineExceeded {
533 + return &module.FunctionResponse{Status: 504, Message: "query timed out"}
534 + }
535 + return &module.FunctionResponse{Status: 500, Message: fmt.Sprintf("query failed: %v", err)}
536 + }
537 + defer rows.Close()
538 +
539 + // Scan rows dynamically
540 + data, err := c.scanMySQLDynamicRows(rows, cols)
541 + if err != nil {
542 + return &module.FunctionResponse{Status: 500, Message: err.Error()}
543 + }
544 +
545 + // Build dynamic sort options from available columns (only those actually detected)
546 + sortParam, sortOptions := c.topQueriesSortParam(cols)
547 +
548 + // Find default sort column UI key
549 + defaultSort := ""
550 + for _, col := range cols {
551 + if col.isDefaultSort && col.isSortOption {
552 + defaultSort = col.uiKey
553 + break
554 + }
555 + }
556 + // Fallback to first sort option if no default
557 + if defaultSort == "" && len(sortOptions) > 0 {
558 + defaultSort = sortOptions[0].ID
559 + }
560 +
561 + return &module.FunctionResponse{
562 + Status: 200,
563 + Help: "Top SQL queries from performance_schema.events_statements_summary_by_digest",
564 + Columns: c.buildMySQLDynamicColumns(cols),
565 + Data: data,
566 + DefaultSortColumn: defaultSort,
567 + RequiredParams: []funcapi.ParamConfig{sortParam},
568 +
569 + // Charts for aggregated visualization
570 + Charts: map[string]module.ChartConfig{
571 + "Calls": {
572 + Name: "Number of Calls",
573 + Type: "stacked-bar",
574 + Columns: []string{"calls"},
575 + },
576 + "Time": {
577 + Name: "Execution Time",
578 + Type: "stacked-bar",
579 + Columns: []string{"totalTime", "avgTime"},
580 + },
581 + "Rows": {
582 + Name: "Rows",
583 + Type: "stacked-bar",
584 + Columns: []string{"rowsSent", "rowsExamined", "rowsAffected"},
585 + },
586 + "Errors": {
587 + Name: "Errors & Warnings",
588 + Type: "stacked-bar",
589 + Columns: []string{"errors", "warnings"},
590 + },
591 + },
592 + DefaultCharts: [][]string{
593 + {"Time", "schema"},
594 + {"Calls", "schema"},
595 + },
596 + GroupBy: map[string]module.GroupByConfig{
597 + "schema": {
598 + Name: "Group by Schema",
599 + Columns: []string{"schema"},
600 + },
601 + },
602 + }
603 +}
604 +
605 +// checkPerformanceSchema checks if performance_schema is enabled (cached)
606 +func (c *Collector) checkPerformanceSchema(ctx context.Context) (bool, error) {
607 + // Fast path: return cached result if already checked
608 + c.varPerfSchemaMu.RLock()
609 + cached := c.varPerformanceSchema
610 + c.varPerfSchemaMu.RUnlock()
611 + if cached != "" {
612 + return cached == "ON" || cached == "1", nil
613 + }
614 +
615 + // Slow path: query and cache the result
616 + // Use write lock for the entire operation to prevent duplicate queries
617 + c.varPerfSchemaMu.Lock()
618 + defer c.varPerfSchemaMu.Unlock()
619 +
620 + // Double-check after acquiring write lock (another goroutine may have set it)
621 + if c.varPerformanceSchema != "" {
622 + return c.varPerformanceSchema == "ON" || c.varPerformanceSchema == "1", nil
623 + }
624 +
625 + var value string
626 + query := "SELECT @@performance_schema"
627 + err := c.db.QueryRowContext(ctx, query).Scan(&value)
628 + if err != nil {
629 + return false, err
630 + }
631 +
632 + // Cache the result
633 + c.varPerformanceSchema = value
634 + return value == "ON" || value == "1", nil
635 +}
src/go/plugin/go.d/collector/mysql/functions_test.go new
+273
@@ -0,0 +1,273 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package mysql
4 +
5 +import (
6 + "testing"
7 +
8 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
9 + "github.com/stretchr/testify/assert"
10 +)
11 +
12 +func TestMysqlMethods(t *testing.T) {
13 + methods := mysqlMethods()
14 +
15 + require := assert.New(t)
16 + require.Len(methods, 1)
17 + require.Equal("top-queries", methods[0].ID)
18 + require.Equal("Top Queries", methods[0].Name)
19 + require.NotEmpty(methods[0].RequiredParams)
20 +
21 + // Verify at least one default sort option exists
22 + var sortParam *funcapi.ParamConfig
23 + for i := range methods[0].RequiredParams {
24 + if methods[0].RequiredParams[i].ID == "__sort" {
25 + sortParam = &methods[0].RequiredParams[i]
26 + break
27 + }
28 + }
29 + require.NotNil(sortParam, "expected __sort required param")
30 + require.NotEmpty(sortParam.Options)
31 +
32 + hasDefault := false
33 + for _, opt := range sortParam.Options {
34 + if opt.Default {
35 + hasDefault = true
36 + require.Equal("totalTime", opt.ID) // camelCase for UI
37 + break
38 + }
39 + }
40 + require.True(hasDefault, "should have a default sort option")
41 +}
42 +
43 +func TestMysqlAllColumns_HasRequiredColumns(t *testing.T) {
44 + // Verify all required base columns are defined
45 + requiredUIKeys := []string{
46 + "digest", "query", "schema", "calls",
47 + "totalTime", "avgTime", "minTime", "maxTime",
48 + "rowsSent", "rowsExamined", "noIndexUsed",
49 + }
50 +
51 + uiKeys := make(map[string]bool)
52 + for _, col := range mysqlAllColumns {
53 + uiKeys[col.uiKey] = true
54 + }
55 +
56 + for _, key := range requiredUIKeys {
57 + assert.True(t, uiKeys[key], "column %s should be defined in mysqlAllColumns", key)
58 + }
59 +}
60 +
61 +func TestMysqlAllColumns_HasValidMetadata(t *testing.T) {
62 + for _, col := range mysqlAllColumns {
63 + // Every column must have a UI key
64 + assert.NotEmpty(t, col.uiKey, "column %s must have uiKey", col.dbColumn)
65 +
66 + // Every column must have a display name
67 + assert.NotEmpty(t, col.displayName, "column %s must have displayName", col.uiKey)
68 +
69 + // Every column must have a data type
70 + assert.NotEqual(t, funcapi.FieldTypeNone, col.dataType, "column %s must have dataType", col.uiKey)
71 +
72 + // Duration columns must have units
73 + if col.dataType == ftDuration {
74 + assert.NotEmpty(t, col.units, "duration column %s must have units", col.uiKey)
75 + }
76 +
77 + // Sort options must have labels
78 + if col.isSortOption {
79 + assert.NotEmpty(t, col.sortLabel, "sort option column %s must have sortLabel", col.uiKey)
80 + }
81 + }
82 +}
83 +
84 +func TestCollector_mapAndValidateMySQLSortColumn(t *testing.T) {
85 + tests := map[string]struct {
86 + availableCols map[string]bool
87 + input string
88 + expected string
89 + }{
90 + "totalTime maps correctly": {
91 + availableCols: map[string]bool{"SUM_TIMER_WAIT": true, "COUNT_STAR": true},
92 + input: "totalTime",
93 + expected: "totalTime",
94 + },
95 + "calls maps correctly": {
96 + availableCols: map[string]bool{"SUM_TIMER_WAIT": true, "COUNT_STAR": true},
97 + input: "calls",
98 + expected: "calls",
99 + },
100 + "invalid column falls back to totalTime": {
101 + availableCols: map[string]bool{"SUM_TIMER_WAIT": true, "COUNT_STAR": true},
102 + input: "invalid_column",
103 + expected: "totalTime",
104 + },
105 + "SQL injection attempt falls back to totalTime": {
106 + availableCols: map[string]bool{"SUM_TIMER_WAIT": true, "COUNT_STAR": true},
107 + input: "'; DROP TABLE users;--",
108 + expected: "totalTime",
109 + },
110 + "falls back to calls when SUM_TIMER_WAIT unavailable": {
111 + availableCols: map[string]bool{"COUNT_STAR": true},
112 + input: "invalid_column",
113 + expected: "calls",
114 + },
115 + }
116 +
117 + for name, tc := range tests {
118 + t.Run(name, func(t *testing.T) {
119 + c := &Collector{}
120 + result := c.mapAndValidateMySQLSortColumn(tc.input, tc.availableCols)
121 + assert.Equal(t, tc.expected, result)
122 + })
123 + }
124 +}
125 +
126 +func TestCollector_buildAvailableMySQLColumns(t *testing.T) {
127 + tests := map[string]struct {
128 + availableCols map[string]bool
129 + expectCols []string // UI keys we expect to see
130 + notExpectCols []string // UI keys we don't expect
131 + }{
132 + "Basic MySQL 5.7 columns": {
133 + availableCols: map[string]bool{
134 + "DIGEST": true, "DIGEST_TEXT": true, "SCHEMA_NAME": true, "COUNT_STAR": true,
135 + "SUM_TIMER_WAIT": true, "MIN_TIMER_WAIT": true, "AVG_TIMER_WAIT": true, "MAX_TIMER_WAIT": true,
136 + "SUM_ROWS_SENT": true, "SUM_ROWS_EXAMINED": true, "SUM_NO_INDEX_USED": true,
137 + },
138 + expectCols: []string{"digest", "query", "schema", "calls", "totalTime", "rowsSent"},
139 + notExpectCols: []string{"p95Time", "cpuTime", "maxTotalMemory"}, // MySQL 8.0+ only
140 + },
141 + "MySQL 8.0 with quantiles": {
142 + availableCols: map[string]bool{
143 + "DIGEST": true, "DIGEST_TEXT": true, "SCHEMA_NAME": true, "COUNT_STAR": true,
144 + "SUM_TIMER_WAIT": true, "QUANTILE_95": true, "QUANTILE_99": true,
145 + "QUERY_SAMPLE_TEXT": true,
146 + },
147 + expectCols: []string{"digest", "query", "calls", "p95Time", "p99Time", "sampleQuery"},
148 + },
149 + }
150 +
151 + for name, tc := range tests {
152 + t.Run(name, func(t *testing.T) {
153 + c := &Collector{}
154 + cols := c.buildAvailableMySQLColumns(tc.availableCols)
155 +
156 + // Build map of UI keys for easy lookup
157 + uiKeys := make(map[string]bool)
158 + for _, col := range cols {
159 + uiKeys[col.uiKey] = true
160 + }
161 +
162 + for _, key := range tc.expectCols {
163 + assert.True(t, uiKeys[key], "expected column %s to be present", key)
164 + }
165 + for _, key := range tc.notExpectCols {
166 + assert.False(t, uiKeys[key], "did not expect column %s to be present", key)
167 + }
168 + })
169 + }
170 +}
171 +
172 +func TestCollector_buildMySQLDynamicSQL(t *testing.T) {
173 + c := &Collector{}
174 +
175 + cols := []mysqlColumnMeta{
176 + {dbColumn: "DIGEST", uiKey: "digest", dataType: ftString},
177 + {dbColumn: "DIGEST_TEXT", uiKey: "query", dataType: ftString},
178 + {dbColumn: "COUNT_STAR", uiKey: "calls", dataType: ftInteger},
179 + {dbColumn: "SUM_TIMER_WAIT", uiKey: "totalTime", dataType: ftDuration, isPicoseconds: true},
180 + }
181 +
182 + sql := c.buildMySQLDynamicSQL(cols, "totalTime", 500)
183 +
184 + assert.Contains(t, sql, "performance_schema.events_statements_summary_by_digest")
185 + assert.Contains(t, sql, "ORDER BY `totalTime` DESC")
186 + assert.Contains(t, sql, "LIMIT 500")
187 + assert.Contains(t, sql, "AS `digest`")
188 + assert.Contains(t, sql, "AS `calls`")
189 + assert.Contains(t, sql, "AS `totalTime`")
190 + // Picosecond columns should have conversion
191 + assert.Contains(t, sql, "SUM_TIMER_WAIT/1000000000 AS `totalTime`")
192 +}
193 +
194 +func TestCollector_buildMySQLDynamicColumns(t *testing.T) {
195 + c := &Collector{}
196 +
197 + cols := []mysqlColumnMeta{
198 + {uiKey: "digest", displayName: "Digest", dataType: ftString, visible: false, isUniqueKey: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
199 + {uiKey: "query", displayName: "Query", dataType: ftString, visible: true, isSticky: true, fullWidth: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
200 + {uiKey: "totalTime", displayName: "Total Time", dataType: ftDuration, units: "seconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
201 + }
202 +
203 + columns := c.buildMySQLDynamicColumns(cols)
204 +
205 + // Verify column count
206 + assert.Len(t, columns, 3)
207 +
208 + // Verify digest column
209 + digestCol := columns["digest"].(map[string]any)
210 + assert.Equal(t, "Digest", digestCol["name"])
211 + assert.Equal(t, "string", digestCol["type"])
212 + assert.True(t, digestCol["unique_key"].(bool))
213 + assert.False(t, digestCol["visible"].(bool))
214 + assert.Equal(t, 0, digestCol["index"])
215 +
216 + // Verify query column
217 + queryCol := columns["query"].(map[string]any)
218 + assert.Equal(t, "Query", queryCol["name"])
219 + assert.True(t, queryCol["sticky"].(bool))
220 + assert.True(t, queryCol["full_width"].(bool))
221 + assert.Equal(t, 1, queryCol["index"])
222 +
223 + // Verify totalTime column
224 + totalTimeCol := columns["totalTime"].(map[string]any)
225 + assert.Equal(t, "Total Time", totalTimeCol["name"])
226 + assert.Equal(t, "duration", totalTimeCol["type"])
227 + assert.Equal(t, "seconds", totalTimeCol["units"])
228 + assert.Equal(t, "bar", totalTimeCol["visualization"]) // duration uses bar
229 + assert.Equal(t, 2, totalTimeCol["index"])
230 +}
231 +
232 +// Test that method config sort options have valid column references
233 +func TestMysqlMethods_SortOptionsHaveLabels(t *testing.T) {
234 + methods := mysqlMethods()
235 +
236 + for _, method := range methods {
237 + var sortParam *funcapi.ParamConfig
238 + for i := range method.RequiredParams {
239 + if method.RequiredParams[i].ID == "__sort" {
240 + sortParam = &method.RequiredParams[i]
241 + break
242 + }
243 + }
244 + assert.NotNil(t, sortParam)
245 + for _, opt := range sortParam.Options {
246 + assert.NotEmpty(t, opt.ID, "sort option must have ID")
247 + assert.NotEmpty(t, opt.Name, "sort option %s must have Name", opt.ID)
248 + assert.Contains(t, opt.Name, "Top queries by", "label should have standard prefix")
249 + }
250 + }
251 +}
252 +
253 +// TestSortColumnValidation_SQLInjection verifies that SQL injection attempts
254 +// are handled by the validation mechanism
255 +func TestSortColumnValidation_SQLInjection(t *testing.T) {
256 + c := &Collector{}
257 + availableCols := map[string]bool{"SUM_TIMER_WAIT": true, "COUNT_STAR": true}
258 +
259 + maliciousInputs := []string{
260 + "'; DROP TABLE performance_schema; --",
261 + "COUNT_STAR; DELETE FROM mysql.user",
262 + "1 OR 1=1",
263 + "SLEEP(10)",
264 + "BENCHMARK(10000000,SHA1('test'))",
265 + }
266 +
267 + for _, input := range maliciousInputs {
268 + result := c.mapAndValidateMySQLSortColumn(input, availableCols)
269 + // All malicious inputs should fall back to safe default
270 + assert.True(t, result == "totalTime" || result == "calls",
271 + "malicious input should fall back to safe default: %s -> %s", input, result)
272 + }
273 +}
src/go/plugin/go.d/collector/postgres/collect.go
+1
@@ -18,6 +18,7 @@ const (
18 pgVersion94 = 9_04_00
19 pgVersion10 = 10_00_00
20 pgVersion11 = 11_00_00
21 + pgVersion13 = 13_00_00
22 pgVersion17 = 17_00_00
23 )
24
src/go/plugin/go.d/collector/postgres/collector.go
+15 -8
@@ -27,6 +27,9 @@ func init() {
27 JobConfigSchema: configSchema,
28 Create: func() module.Module { return New() },
29 Config: func() any { return &Config{} },
30 + Methods: pgMethods,
31 + MethodParams: pgMethodParams,
32 + HandleMethod: pgHandleMethod,
33 })
34 }
35
@@ -69,6 +72,7 @@ type Config struct {
72 QueryTimeHistogram []float64 `yaml:"query_time_histogram,omitempty" json:"query_time_histogram"`
73 MaxDBTables int64 `yaml:"max_db_tables" json:"max_db_tables"`
74 MaxDBIndexes int64 `yaml:"max_db_indexes" json:"max_db_indexes"`
75 + TopQueriesLimit int `yaml:"top_queries_limit,omitempty" json:"top_queries_limit,omitempty"`
76 }
77
78 type (
@@ -83,14 +87,17 @@ type (
87 db *sql.DB
88 dbConns map[string]*dbConn
89
86 - superUser *bool
87 - pgIsInRecovery *bool
88 - pgVersion int
89 - dbSr matcher.Matcher
90 - recheckSettingsTime time.Time
91 - recheckSettingsEvery time.Duration
92 - doSlowTime time.Time
93 - doSlowEvery time.Duration
90 + superUser *bool
91 + pgIsInRecovery *bool
92 + pgVersion int
93 + pgStatStatementsAvail bool // cached positive result only
94 + pgStatStatementsColumns map[string]bool // cached column names from pg_stat_statements
95 + pgStatStatementsMu sync.RWMutex // protects pgStatStatements* fields for concurrent access
96 + dbSr matcher.Matcher
97 + recheckSettingsTime time.Time
98 + recheckSettingsEvery time.Duration
99 + doSlowTime time.Time
100 + doSlowEvery time.Duration
101
102 mx *pgMetrics
103 }
src/go/plugin/go.d/collector/postgres/config_schema.json
+8
@@ -98,6 +98,14 @@
98 5,
99 10
100 ]
101 + },
102 + "top_queries_limit": {
103 + "title": "Top Queries Limit",
104 + "description": "Maximum number of queries to return in the top-queries function response.",
105 + "type": "integer",
106 + "minimum": 1,
107 + "maximum": 5000,
108 + "default": 500
109 }
110 },
111 "required": [
src/go/plugin/go.d/collector/postgres/functions.go new
+755
@@ -0,0 +1,755 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package postgres
4 +
5 +import (
6 + "context"
7 + "database/sql"
8 + "fmt"
9 + "strings"
10 +
11 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
12 + "github.com/netdata/netdata/go/plugins/plugin/go.d/agent/module"
13 + "github.com/netdata/netdata/go/plugins/plugin/go.d/pkg/strmutil"
14 +)
15 +
16 +const maxQueryTextLength = 4096
17 +
18 +const (
19 + paramSort = "__sort"
20 +
21 + ftString = funcapi.FieldTypeString
22 + ftInteger = funcapi.FieldTypeInteger
23 + ftFloat = funcapi.FieldTypeFloat
24 + ftDuration = funcapi.FieldTypeDuration
25 +
26 + trNone = funcapi.FieldTransformNone
27 + trNumber = funcapi.FieldTransformNumber
28 + trDuration = funcapi.FieldTransformDuration
29 +
30 + sortAsc = funcapi.FieldSortAscending
31 + sortDesc = funcapi.FieldSortDescending
32 +
33 + summaryCount = funcapi.FieldSummaryCount
34 + summarySum = funcapi.FieldSummarySum
35 + summaryMin = funcapi.FieldSummaryMin
36 + summaryMax = funcapi.FieldSummaryMax
37 + summaryMean = funcapi.FieldSummaryMean
38 + summaryMedian = funcapi.FieldSummaryMedian
39 +
40 + filterMulti = funcapi.FieldFilterMultiselect
41 + filterRange = funcapi.FieldFilterRange
42 +)
43 +
44 +// pgColumnMeta defines metadata for a pg_stat_statements column
45 +type pgColumnMeta struct {
46 + // Database column name (may vary by version)
47 + dbColumn string
48 + // Canonical name used everywhere: SQL alias, UI key, sort key
49 + uiKey string
50 + // Display name in UI
51 + displayName string
52 + // Data type: "string", "integer", "float", "duration"
53 + dataType funcapi.FieldType
54 + // Unit for duration/numeric types
55 + units string
56 + // Whether visible by default
57 + visible bool
58 + // Transform for value_options
59 + transform funcapi.FieldTransform
60 + // Decimal points for display
61 + decimalPoints int
62 + // Sort direction preference
63 + sortDir funcapi.FieldSort
64 + // Summary function
65 + summary funcapi.FieldSummary
66 + // Filter type
67 + filter funcapi.FieldFilter
68 + // Whether this is a sortable option for the sort dropdown
69 + isSortOption bool
70 + // Sort option label (if isSortOption)
71 + sortLabel string
72 + // Whether this is the default sort
73 + isDefaultSort bool
74 + // Whether this is the unique key
75 + isUniqueKey bool
76 + // Whether this column is sticky (stays visible when scrolling)
77 + isSticky bool
78 + // Whether this column should take full width
79 + fullWidth bool
80 +}
81 +
82 +// pgAllColumns defines ALL possible columns from pg_stat_statements
83 +// Order matters - this determines column index in the response
84 +var pgAllColumns = []pgColumnMeta{
85 + // Core identification columns (always present)
86 + {dbColumn: "s.queryid::text", uiKey: "queryid", displayName: "Query ID", dataType: ftString, visible: false, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, isUniqueKey: true},
87 + {dbColumn: "s.query", uiKey: "query", displayName: "Query", dataType: ftString, visible: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti, isSticky: true, fullWidth: true},
88 + {dbColumn: "d.datname", uiKey: "database", displayName: "Database", dataType: ftString, visible: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
89 + {dbColumn: "u.usename", uiKey: "user", displayName: "User", dataType: ftString, visible: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
90 +
91 + // Execution count (always present)
92 + {dbColumn: "s.calls", uiKey: "calls", displayName: "Calls", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Number of Calls"},
93 +
94 + // Execution time columns (names vary by version - detected dynamically)
95 + // PG <13: total_time, mean_time, min_time, max_time, stddev_time
96 + // PG 13+: total_exec_time, mean_exec_time, min_exec_time, max_exec_time, stddev_exec_time
97 + {dbColumn: "total_time", uiKey: "totalTime", displayName: "Total Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Total Execution Time", isDefaultSort: true},
98 + {dbColumn: "mean_time", uiKey: "meanTime", displayName: "Mean Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange, isSortOption: true, sortLabel: "Average Execution Time"},
99 + {dbColumn: "min_time", uiKey: "minTime", displayName: "Min Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
100 + {dbColumn: "max_time", uiKey: "maxTime", displayName: "Max Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
101 + {dbColumn: "stddev_time", uiKey: "stddevTime", displayName: "Stddev Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
102 +
103 + // Planning time columns (PG 13+ only)
104 + {dbColumn: "s.plans", uiKey: "plans", displayName: "Plans", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
105 + {dbColumn: "total_plan_time", uiKey: "totalPlanTime", displayName: "Total Plan Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
106 + {dbColumn: "mean_plan_time", uiKey: "meanPlanTime", displayName: "Mean Plan Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
107 + {dbColumn: "min_plan_time", uiKey: "minPlanTime", displayName: "Min Plan Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMin, filter: filterRange},
108 + {dbColumn: "max_plan_time", uiKey: "maxPlanTime", displayName: "Max Plan Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
109 + {dbColumn: "stddev_plan_time", uiKey: "stddevPlanTime", displayName: "Stddev Plan Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summaryMax, filter: filterRange},
110 +
111 + // Row count (always present)
112 + {dbColumn: "s.rows", uiKey: "rows", displayName: "Rows", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Rows Returned"},
113 +
114 + // Shared buffer statistics (always present)
115 + {dbColumn: "s.shared_blks_hit", uiKey: "sharedBlksHit", displayName: "Shared Blocks Hit", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Shared Blocks Hit (Cache)"},
116 + {dbColumn: "s.shared_blks_read", uiKey: "sharedBlksRead", displayName: "Shared Blocks Read", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Shared Blocks Read (Disk I/O)"},
117 + {dbColumn: "s.shared_blks_dirtied", uiKey: "sharedBlksDirtied", displayName: "Shared Blocks Dirtied", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
118 + {dbColumn: "s.shared_blks_written", uiKey: "sharedBlksWritten", displayName: "Shared Blocks Written", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
119 +
120 + // Local buffer statistics (always present)
121 + {dbColumn: "s.local_blks_hit", uiKey: "localBlksHit", displayName: "Local Blocks Hit", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
122 + {dbColumn: "s.local_blks_read", uiKey: "localBlksRead", displayName: "Local Blocks Read", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
123 + {dbColumn: "s.local_blks_dirtied", uiKey: "localBlksDirtied", displayName: "Local Blocks Dirtied", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
124 + {dbColumn: "s.local_blks_written", uiKey: "localBlksWritten", displayName: "Local Blocks Written", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
125 +
126 + // Temp buffer statistics (always present)
127 + {dbColumn: "s.temp_blks_read", uiKey: "tempBlksRead", displayName: "Temp Blocks Read", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
128 + {dbColumn: "s.temp_blks_written", uiKey: "tempBlksWritten", displayName: "Temp Blocks Written", dataType: ftInteger, visible: true, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange, isSortOption: true, sortLabel: "Temp Blocks Written"},
129 +
130 + // I/O timing (requires track_io_timing, always present but may be 0)
131 + {dbColumn: "s.blk_read_time", uiKey: "blkReadTime", displayName: "Block Read Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
132 + {dbColumn: "s.blk_write_time", uiKey: "blkWriteTime", displayName: "Block Write Time", dataType: ftDuration, units: "milliseconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
133 +
134 + // WAL statistics (PG 13+ only)
135 + {dbColumn: "s.wal_records", uiKey: "walRecords", displayName: "WAL Records", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
136 + {dbColumn: "s.wal_fpi", uiKey: "walFpi", displayName: "WAL Full Page Images", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
137 + {dbColumn: "s.wal_bytes", uiKey: "walBytes", displayName: "WAL Bytes", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
138 +
139 + // JIT statistics (PG 15+ only)
140 + {dbColumn: "s.jit_functions", uiKey: "jitFunctions", displayName: "JIT Functions", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
141 + {dbColumn: "s.jit_generation_time", uiKey: "jitGenerationTime", displayName: "JIT Generation Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
142 + {dbColumn: "s.jit_inlining_count", uiKey: "jitInliningCount", displayName: "JIT Inlining Count", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
143 + {dbColumn: "s.jit_inlining_time", uiKey: "jitInliningTime", displayName: "JIT Inlining Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
144 + {dbColumn: "s.jit_optimization_count", uiKey: "jitOptimizationCount", displayName: "JIT Optimization Count", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
145 + {dbColumn: "s.jit_optimization_time", uiKey: "jitOptimizationTime", displayName: "JIT Optimization Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
146 + {dbColumn: "s.jit_emission_count", uiKey: "jitEmissionCount", displayName: "JIT Emission Count", dataType: ftInteger, visible: false, transform: trNumber, sortDir: sortDesc, summary: summarySum, filter: filterRange},
147 + {dbColumn: "s.jit_emission_time", uiKey: "jitEmissionTime", displayName: "JIT Emission Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
148 +
149 + // Temp file statistics (PG 15+ only)
150 + {dbColumn: "s.temp_blk_read_time", uiKey: "tempBlkReadTime", displayName: "Temp Block Read Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
151 + {dbColumn: "s.temp_blk_write_time", uiKey: "tempBlkWriteTime", displayName: "Temp Block Write Time", dataType: ftDuration, units: "milliseconds", visible: false, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
152 +}
153 +
154 +// pgMethods returns the available function methods for PostgreSQL
155 +// Sort options are built dynamically based on available columns
156 +func pgMethods() []module.MethodConfig {
157 + // Build sort options from column metadata
158 + var sortOptions []funcapi.ParamOption
159 + sortDir := funcapi.FieldSortDescending
160 + for _, col := range pgAllColumns {
161 + if col.isSortOption {
162 + sortOptions = append(sortOptions, funcapi.ParamOption{
163 + ID: col.uiKey,
164 + Column: col.uiKey,
165 + Name: "Top queries by " + col.sortLabel,
166 + Default: col.isDefaultSort,
167 + Sort: &sortDir,
168 + })
169 + }
170 + }
171 +
172 + return []module.MethodConfig{{
173 + ID: "top-queries",
174 + Name: "Top Queries",
175 + Help: "Top SQL queries from pg_stat_statements",
176 + RequiredParams: []funcapi.ParamConfig{
177 + {
178 + ID: paramSort,
179 + Name: "Filter By",
180 + Help: "Select the primary sort column",
181 + Selection: funcapi.ParamSelect,
182 + Options: sortOptions,
183 + UniqueView: true,
184 + },
185 + },
186 + }}
187 +}
188 +
189 +func pgMethodParams(ctx context.Context, job *module.Job, method string) ([]funcapi.ParamConfig, error) {
190 + collector, ok := job.Module().(*Collector)
191 + if !ok {
192 + return nil, fmt.Errorf("invalid module type")
193 + }
194 + if collector.db == nil {
195 + return nil, fmt.Errorf("collector is still initializing")
196 + }
197 + switch method {
198 + case "top-queries":
199 + return collector.topQueriesParams(ctx)
200 + default:
201 + return nil, fmt.Errorf("unknown method: %s", method)
202 + }
203 +}
204 +
205 +// pgHandleMethod handles function requests for PostgreSQL
206 +func pgHandleMethod(ctx context.Context, job *module.Job, method string, params funcapi.ResolvedParams) *module.FunctionResponse {
207 + collector, ok := job.Module().(*Collector)
208 + if !ok {
209 + return &module.FunctionResponse{Status: 500, Message: "internal error: invalid module type"}
210 + }
211 +
212 + // Check if collector is initialized (first collect() may not have run yet)
213 + if collector.db == nil {
214 + return &module.FunctionResponse{
215 + Status: 503,
216 + Message: "collector is still initializing, please retry in a few seconds",
217 + }
218 + }
219 +
220 + switch method {
221 + case "top-queries":
222 + return collector.collectTopQueries(ctx, params.Column(paramSort))
223 + default:
224 + return &module.FunctionResponse{Status: 404, Message: fmt.Sprintf("unknown method: %s", method)}
225 + }
226 +}
227 +
228 +// collectTopQueries queries pg_stat_statements for top queries
229 +func (c *Collector) collectTopQueries(ctx context.Context, sortColumn string) *module.FunctionResponse {
230 + // Check pg_stat_statements availability (lazy check)
231 + available, err := c.checkPgStatStatements(ctx)
232 + if err != nil {
233 + return &module.FunctionResponse{
234 + Status: 500,
235 + Message: fmt.Sprintf("failed to check pg_stat_statements availability: %v", err),
236 + }
237 + }
238 + if !available {
239 + return &module.FunctionResponse{
240 + Status: 503,
241 + Message: "pg_stat_statements extension is not installed in this database. " +
242 + "Run 'CREATE EXTENSION pg_stat_statements;' in the database the collector connects to.",
243 + }
244 + }
245 +
246 + // Detect available columns (lazy detection, cached)
247 + availableCols, err := c.detectPgStatStatementsColumns(ctx)
248 + if err != nil {
249 + return &module.FunctionResponse{
250 + Status: 500,
251 + Message: fmt.Sprintf("failed to detect available columns: %v", err),
252 + }
253 + }
254 +
255 + // Build list of columns to query based on what's available
256 + queryCols := c.buildAvailableColumns(availableCols)
257 + if len(queryCols) == 0 {
258 + return &module.FunctionResponse{
259 + Status: 500,
260 + Message: "no queryable columns found in pg_stat_statements",
261 + }
262 + }
263 +
264 + // Map and validate sort column
265 + actualSortCol := c.mapAndValidateSortColumn(sortColumn, availableCols)
266 +
267 + // Get query limit (default 500)
268 + limit := c.TopQueriesLimit
269 + if limit <= 0 {
270 + limit = 500
271 + }
272 +
273 + // Build and execute query
274 + query := c.buildDynamicSQL(queryCols, actualSortCol, limit)
275 + rows, err := c.db.QueryContext(ctx, query)
276 + if err != nil {
277 + if ctx.Err() == context.DeadlineExceeded {
278 + return &module.FunctionResponse{Status: 504, Message: "query timed out"}
279 + }
280 + return &module.FunctionResponse{Status: 500, Message: fmt.Sprintf("query failed: %v", err)}
281 + }
282 + defer rows.Close()
283 +
284 + // Process rows and build response
285 + data, err := c.scanDynamicRows(rows, queryCols)
286 + if err != nil {
287 + return &module.FunctionResponse{Status: 500, Message: err.Error()}
288 + }
289 +
290 + if err := rows.Err(); err != nil {
291 + return &module.FunctionResponse{Status: 500, Message: fmt.Sprintf("rows iteration error: %v", err)}
292 + }
293 +
294 + // Build dynamic sort options from available columns (only those actually detected)
295 + sortParam, sortOptions := c.topQueriesSortParam(queryCols)
296 +
297 + // Find default sort column UI key from metadata
298 + defaultSort := ""
299 + for _, col := range queryCols {
300 + if col.isDefaultSort && col.isSortOption {
301 + defaultSort = col.uiKey
302 + break
303 + }
304 + }
305 + // Fallback to first sort option if no default
306 + if defaultSort == "" && len(sortOptions) > 0 {
307 + defaultSort = sortOptions[0].ID
308 + }
309 +
310 + return &module.FunctionResponse{
311 + Status: 200,
312 + Help: "Top SQL queries from pg_stat_statements",
313 + Columns: c.buildDynamicColumns(queryCols),
314 + Data: data,
315 + DefaultSortColumn: defaultSort,
316 + RequiredParams: []funcapi.ParamConfig{sortParam},
317 +
318 + // Charts for aggregated visualization
319 + Charts: map[string]module.ChartConfig{
320 + "Calls": {
321 + Name: "Number of Calls",
322 + Type: "stacked-bar",
323 + Columns: []string{"calls"},
324 + },
325 + "Time": {
326 + Name: "Execution Time",
327 + Type: "stacked-bar",
328 + Columns: []string{"totalTime", "meanTime"},
329 + },
330 + "Rows": {
331 + Name: "Rows Returned",
332 + Type: "stacked-bar",
333 + Columns: []string{"rows"},
334 + },
335 + "IO": {
336 + Name: "Block I/O",
337 + Type: "stacked-bar",
338 + Columns: []string{"sharedBlksHit", "sharedBlksRead"},
339 + },
340 + },
341 + DefaultCharts: [][]string{
342 + {"Time", "database"},
343 + {"Calls", "database"},
344 + },
345 + GroupBy: map[string]module.GroupByConfig{
346 + "database": {
347 + Name: "Group by Database",
348 + Columns: []string{"database"},
349 + },
350 + "user": {
351 + Name: "Group by User",
352 + Columns: []string{"user"},
353 + },
354 + },
355 + }
356 +}
357 +
358 +// detectPgStatStatementsColumns queries the database to find available columns
359 +func (c *Collector) detectPgStatStatementsColumns(ctx context.Context) (map[string]bool, error) {
360 + // Fast path: return cached result
361 + c.pgStatStatementsMu.RLock()
362 + if c.pgStatStatementsColumns != nil {
363 + cols := c.pgStatStatementsColumns
364 + c.pgStatStatementsMu.RUnlock()
365 + return cols, nil
366 + }
367 + c.pgStatStatementsMu.RUnlock()
368 +
369 + // Slow path: query and cache
370 + c.pgStatStatementsMu.Lock()
371 + defer c.pgStatStatementsMu.Unlock()
372 +
373 + // Double-check after acquiring write lock
374 + if c.pgStatStatementsColumns != nil {
375 + return c.pgStatStatementsColumns, nil
376 + }
377 +
378 + // Query available columns from pg_stat_statements
379 + query := `
380 + SELECT column_name
381 + FROM information_schema.columns
382 + WHERE table_name = 'pg_stat_statements'
383 + AND table_schema = 'public'
384 + `
385 + rows, err := c.db.QueryContext(ctx, query)
386 + if err != nil {
387 + return nil, fmt.Errorf("failed to query columns: %v", err)
388 + }
389 + defer rows.Close()
390 +
391 + cols := make(map[string]bool)
392 + for rows.Next() {
393 + var colName string
394 + if err := rows.Scan(&colName); err != nil {
395 + return nil, fmt.Errorf("failed to scan column name: %v", err)
396 + }
397 + cols[colName] = true
398 + }
399 +
400 + if err := rows.Err(); err != nil {
401 + return nil, fmt.Errorf("rows iteration error: %v", err)
402 + }
403 +
404 + // Cache the result
405 + c.pgStatStatementsColumns = cols
406 + return cols, nil
407 +}
408 +
409 +// buildAvailableColumns returns column metadata for columns that exist in this PG version
410 +func (c *Collector) buildAvailableColumns(availableCols map[string]bool) []pgColumnMeta {
411 + var result []pgColumnMeta
412 +
413 + for _, col := range pgAllColumns {
414 + // Extract the actual column name (remove table prefix and type cast)
415 + colName := col.dbColumn
416 + if idx := strings.LastIndex(colName, "."); idx != -1 {
417 + colName = colName[idx+1:]
418 + }
419 + // Remove PostgreSQL type cast suffix (e.g., "::text")
420 + if idx := strings.Index(colName, "::"); idx != -1 {
421 + colName = colName[:idx]
422 + }
423 +
424 + // Handle version-specific column names for time columns
425 + // PG 13+ renamed time columns: total_time -> total_exec_time, etc.
426 + actualColName := colName
427 + if c.pgVersion >= pgVersion13 {
428 + switch colName {
429 + case "total_time":
430 + actualColName = "total_exec_time"
431 + case "mean_time":
432 + actualColName = "mean_exec_time"
433 + case "min_time":
434 + actualColName = "min_exec_time"
435 + case "max_time":
436 + actualColName = "max_exec_time"
437 + case "stddev_time":
438 + actualColName = "stddev_exec_time"
439 + }
440 + }
441 +
442 + // Check if column exists (either directly or via join)
443 + // Join columns (database, user) come from other tables (d.datname, u.usename)
444 + isJoinCol := col.uiKey == "database" || col.uiKey == "user"
445 + if isJoinCol || availableCols[actualColName] {
446 + // Create a copy with the actual column name for this version
447 + colCopy := col
448 + if actualColName != colName {
449 + // Update dbColumn to use the version-specific name with alias
450 + prefix := "s."
451 + if strings.HasPrefix(col.dbColumn, "s.") {
452 + prefix = ""
453 + colCopy.dbColumn = "s." + actualColName
454 + }
455 + _ = prefix // suppress unused warning
456 + }
457 + result = append(result, colCopy)
458 + }
459 + }
460 +
461 + return result
462 +}
463 +
464 +// mapAndValidateSortColumn maps the semantic sort column to actual SQL column
465 +func (c *Collector) mapAndValidateSortColumn(sortColumn string, availableCols map[string]bool) string {
466 + // Map UI key back to dbColumn
467 + for _, col := range pgAllColumns {
468 + if col.uiKey == sortColumn || col.dbColumn == sortColumn {
469 + // Get actual column name (strip table prefix and type cast)
470 + colName := col.dbColumn
471 + if idx := strings.LastIndex(colName, "."); idx != -1 {
472 + colName = colName[idx+1:]
473 + }
474 + if idx := strings.Index(colName, "::"); idx != -1 {
475 + colName = colName[:idx]
476 + }
477 +
478 + // Handle version-specific mapping
479 + if c.pgVersion >= pgVersion13 {
480 + switch colName {
481 + case "total_time":
482 + colName = "total_exec_time"
483 + case "mean_time":
484 + colName = "mean_exec_time"
485 + case "min_time":
486 + colName = "min_exec_time"
487 + case "max_time":
488 + colName = "max_exec_time"
489 + case "stddev_time":
490 + colName = "stddev_exec_time"
491 + }
492 + }
493 +
494 + // Validate column exists
495 + if availableCols[colName] {
496 + return colName
497 + }
498 + }
499 + }
500 +
501 + // Default fallback
502 + if c.pgVersion >= pgVersion13 {
503 + return "total_exec_time"
504 + }
505 + return "total_time"
506 +}
507 +
508 +// buildDynamicSQL builds the SQL query with only available columns
509 +func (c *Collector) buildDynamicSQL(cols []pgColumnMeta, sortColumn string, limit int) string {
510 + var selectCols []string
511 +
512 + for _, col := range cols {
513 + colExpr := col.dbColumn
514 +
515 + // Handle version-specific column names
516 + if c.pgVersion >= pgVersion13 {
517 + switch {
518 + case strings.HasSuffix(colExpr, ".total_time"):
519 + colExpr = strings.Replace(colExpr, ".total_time", ".total_exec_time", 1)
520 + case strings.HasSuffix(colExpr, ".mean_time"):
521 + colExpr = strings.Replace(colExpr, ".mean_time", ".mean_exec_time", 1)
522 + case strings.HasSuffix(colExpr, ".min_time"):
523 + colExpr = strings.Replace(colExpr, ".min_time", ".min_exec_time", 1)
524 + case strings.HasSuffix(colExpr, ".max_time"):
525 + colExpr = strings.Replace(colExpr, ".max_time", ".max_exec_time", 1)
526 + case strings.HasSuffix(colExpr, ".stddev_time"):
527 + colExpr = strings.Replace(colExpr, ".stddev_time", ".stddev_exec_time", 1)
528 + case colExpr == "total_time":
529 + colExpr = "total_exec_time"
530 + case colExpr == "mean_time":
531 + colExpr = "mean_exec_time"
532 + case colExpr == "min_time":
533 + colExpr = "min_exec_time"
534 + case colExpr == "max_time":
535 + colExpr = "max_exec_time"
536 + case colExpr == "stddev_time":
537 + colExpr = "stddev_exec_time"
538 + }
539 + }
540 +
541 + // Always use uiKey as the SQL alias for consistent naming
542 + // Use double quotes to handle reserved keywords like "database", "user"
543 + selectCols = append(selectCols, fmt.Sprintf("%s AS \"%s\"", colExpr, col.uiKey))
544 + }
545 +
546 + return fmt.Sprintf(`
547 +SELECT %s
548 +FROM pg_stat_statements s
549 +JOIN pg_database d ON s.dbid = d.oid
550 +JOIN pg_user u ON s.userid = u.usesysid
551 +ORDER BY "%s" DESC
552 +LIMIT %d
553 +`, strings.Join(selectCols, ", "), sortColumn, limit)
554 +}
555 +
556 +// scanDynamicRows scans rows into the data array based on column types
557 +// Uses sql.Null* types to handle NULL values safely
558 +func (c *Collector) scanDynamicRows(rows dbRows, cols []pgColumnMeta) ([][]any, error) {
559 + data := make([][]any, 0, 500)
560 +
561 + // Create value holders for scanning (reuse across rows for efficiency)
562 + valuePtrs := make([]any, len(cols))
563 + values := make([]any, len(cols))
564 +
565 + for rows.Next() {
566 + // Reset value holders for each row
567 + for i, col := range cols {
568 + switch col.dataType {
569 + case ftString:
570 + var v sql.NullString
571 + values[i] = &v
572 + case ftInteger:
573 + var v sql.NullInt64
574 + values[i] = &v
575 + case ftFloat, ftDuration:
576 + var v sql.NullFloat64
577 + values[i] = &v
578 + default:
579 + var v sql.NullString
580 + values[i] = &v
581 + }
582 + valuePtrs[i] = values[i]
583 + }
584 +
585 + if err := rows.Scan(valuePtrs...); err != nil {
586 + return nil, fmt.Errorf("row scan failed: %v", err)
587 + }
588 +
589 + // Convert scanned values to output format
590 + row := make([]any, len(cols))
591 + for i, col := range cols {
592 + switch v := values[i].(type) {
593 + case *sql.NullString:
594 + if v.Valid {
595 + s := v.String
596 + if col.uiKey == "query" {
597 + row[i] = strmutil.TruncateText(s, maxQueryTextLength)
598 + } else {
599 + row[i] = s
600 + }
601 + } else {
602 + row[i] = ""
603 + }
604 + case *sql.NullInt64:
605 + if v.Valid {
606 + row[i] = v.Int64
607 + } else {
608 + row[i] = int64(0)
609 + }
610 + case *sql.NullFloat64:
611 + if v.Valid {
612 + row[i] = v.Float64
613 + } else {
614 + row[i] = float64(0)
615 + }
616 + }
617 + }
618 +
619 + data = append(data, row)
620 + }
621 +
622 + return data, nil
623 +}
624 +
625 +// buildDynamicColumns builds column definitions for the response
626 +func (c *Collector) buildDynamicColumns(cols []pgColumnMeta) map[string]any {
627 + result := make(map[string]any)
628 +
629 + for i, col := range cols {
630 + visual := funcapi.FieldVisualValue
631 + if col.dataType == ftDuration {
632 + visual = funcapi.FieldVisualBar
633 + }
634 + colDef := funcapi.Column{
635 + Index: i,
636 + Name: col.displayName,
637 + Type: col.dataType,
638 + Units: col.units,
639 + Visualization: visual,
640 + Sort: col.sortDir,
641 + Sortable: true,
642 + Sticky: col.isSticky,
643 + Summary: col.summary,
644 + Filter: col.filter,
645 + FullWidth: col.fullWidth,
646 + Wrap: false,
647 + DefaultExpandedFilter: false,
648 + UniqueKey: col.isUniqueKey,
649 + Visible: col.visible,
650 + ValueOptions: funcapi.ValueOptions{
651 + Transform: col.transform,
652 + DecimalPoints: col.decimalPoints,
653 + DefaultValue: nil,
654 + },
655 + }
656 + result[col.uiKey] = colDef.BuildColumn()
657 + }
658 +
659 + return result
660 +}
661 +
662 +// buildDynamicSortOptions builds sort options from available columns
663 +// Returns only sort options for columns that actually exist in the database
664 +func (c *Collector) buildDynamicSortOptions(cols []pgColumnMeta) []funcapi.ParamOption {
665 + var sortOpts []funcapi.ParamOption
666 + seen := make(map[string]bool)
667 + sortDir := funcapi.FieldSortDescending
668 +
669 + for _, col := range cols {
670 + if col.isSortOption && !seen[col.uiKey] {
671 + seen[col.uiKey] = true
672 + sortOpts = append(sortOpts, funcapi.ParamOption{
673 + ID: col.uiKey,
674 + Column: col.uiKey,
675 + Name: col.sortLabel,
676 + Default: col.isDefaultSort,
677 + Sort: &sortDir,
678 + })
679 + }
680 + }
681 + return sortOpts
682 +}
683 +
684 +func (c *Collector) topQueriesSortParam(queryCols []pgColumnMeta) (funcapi.ParamConfig, []funcapi.ParamOption) {
685 + sortOptions := c.buildDynamicSortOptions(queryCols)
686 + sortParam := funcapi.ParamConfig{
687 + ID: paramSort,
688 + Name: "Filter By",
689 + Help: "Select the primary sort column",
690 + Selection: funcapi.ParamSelect,
691 + Options: sortOptions,
692 + UniqueView: true,
693 + }
694 + return sortParam, sortOptions
695 +}
696 +
697 +func (c *Collector) topQueriesParams(ctx context.Context) ([]funcapi.ParamConfig, error) {
698 + available, err := c.checkPgStatStatements(ctx)
699 + if err != nil {
700 + return nil, err
701 + }
702 + if !available {
703 + return nil, fmt.Errorf("pg_stat_statements extension is not installed")
704 + }
705 +
706 + availableCols, err := c.detectPgStatStatementsColumns(ctx)
707 + if err != nil {
708 + return nil, err
709 + }
710 +
711 + queryCols := c.buildAvailableColumns(availableCols)
712 + if len(queryCols) == 0 {
713 + return nil, fmt.Errorf("no queryable columns found in pg_stat_statements")
714 + }
715 +
716 + sortParam, _ := c.topQueriesSortParam(queryCols)
717 + return []funcapi.ParamConfig{sortParam}, nil
718 +}
719 +
720 +// checkPgStatStatements checks if pg_stat_statements extension is available
721 +// Only positive results are cached - negative results are re-checked each time
722 +// so users don't need to restart after installing the extension
723 +func (c *Collector) checkPgStatStatements(ctx context.Context) (bool, error) {
724 + // Fast path: return cached positive result
725 + c.pgStatStatementsMu.RLock()
726 + avail := c.pgStatStatementsAvail
727 + c.pgStatStatementsMu.RUnlock()
728 + if avail {
729 + return true, nil
730 + }
731 +
732 + // Slow path: query the database
733 + var exists bool
734 + query := `SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements')`
735 + err := c.db.QueryRowContext(ctx, query).Scan(&exists)
736 + if err != nil {
737 + return false, err
738 + }
739 +
740 + // Only cache positive results
741 + if exists {
742 + c.pgStatStatementsMu.Lock()
743 + c.pgStatStatementsAvail = true
744 + c.pgStatStatementsMu.Unlock()
745 + }
746 +
747 + return exists, nil
748 +}
749 +
750 +// dbRows interface for testing
751 +type dbRows interface {
752 + Next() bool
753 + Scan(dest ...any) error
754 + Err() error
755 +}
src/go/plugin/go.d/collector/postgres/functions_test.go new
+288
@@ -0,0 +1,288 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package postgres
4 +
5 +import (
6 + "testing"
7 +
8 + "github.com/netdata/netdata/go/plugins/pkg/funcapi"
9 + "github.com/stretchr/testify/assert"
10 +)
11 +
12 +// pgVersionOld is used only for tests (not in production code)
13 +const pgVersionOld = 12_00_00 // PG 12 - before total_exec_time was introduced
14 +
15 +func TestPgMethods(t *testing.T) {
16 + methods := pgMethods()
17 +
18 + require := assert.New(t)
19 + require.Len(methods, 1)
20 + require.Equal("top-queries", methods[0].ID)
21 + require.Equal("Top Queries", methods[0].Name)
22 + require.NotEmpty(methods[0].RequiredParams)
23 +
24 + // Verify at least one default sort option exists
25 + var sortParam *funcapi.ParamConfig
26 + for i := range methods[0].RequiredParams {
27 + if methods[0].RequiredParams[i].ID == "__sort" {
28 + sortParam = &methods[0].RequiredParams[i]
29 + break
30 + }
31 + }
32 + require.NotNil(sortParam, "expected __sort required param")
33 + require.NotEmpty(sortParam.Options)
34 +
35 + hasDefault := false
36 + for _, opt := range sortParam.Options {
37 + if opt.Default {
38 + hasDefault = true
39 + require.Equal("totalTime", opt.ID) // camelCase for UI
40 + break
41 + }
42 + }
43 + require.True(hasDefault, "should have a default sort option")
44 +}
45 +
46 +func TestPgAllColumns_HasRequiredColumns(t *testing.T) {
47 + // Verify all required base columns are defined
48 + requiredUIKeys := []string{
49 + "queryid", "query", "database", "user", "calls",
50 + "totalTime", "meanTime", "minTime", "maxTime",
51 + "rows", "sharedBlksHit", "sharedBlksRead", "tempBlksWritten",
52 + }
53 +
54 + uiKeys := make(map[string]bool)
55 + for _, col := range pgAllColumns {
56 + uiKeys[col.uiKey] = true
57 + }
58 +
59 + for _, key := range requiredUIKeys {
60 + assert.True(t, uiKeys[key], "column %s should be defined in pgAllColumns", key)
61 + }
62 +}
63 +
64 +func TestPgAllColumns_HasValidMetadata(t *testing.T) {
65 + for _, col := range pgAllColumns {
66 + // Every column must have a UI key
67 + assert.NotEmpty(t, col.uiKey, "column %s must have uiKey", col.dbColumn)
68 +
69 + // Every column must have a display name
70 + assert.NotEmpty(t, col.displayName, "column %s must have displayName", col.uiKey)
71 +
72 + // Every column must have a data type
73 + assert.NotEqual(t, funcapi.FieldTypeNone, col.dataType, "column %s must have dataType", col.uiKey)
74 +
75 + // Duration columns must have units
76 + if col.dataType == ftDuration {
77 + assert.NotEmpty(t, col.units, "duration column %s must have units", col.uiKey)
78 + }
79 +
80 + // Sort options must have labels
81 + if col.isSortOption {
82 + assert.NotEmpty(t, col.sortLabel, "sort option column %s must have sortLabel", col.uiKey)
83 + }
84 + }
85 +}
86 +
87 +func TestCollector_mapAndValidateSortColumn(t *testing.T) {
88 + tests := map[string]struct {
89 + pgVersion int
90 + availableCols map[string]bool
91 + input string
92 + expected string
93 + }{
94 + "totalTime on PG12 maps to total_time": {
95 + pgVersion: pgVersionOld,
96 + availableCols: map[string]bool{"total_time": true, "calls": true},
97 + input: "totalTime",
98 + expected: "total_time",
99 + },
100 + "totalTime on PG13 maps to total_exec_time": {
101 + pgVersion: pgVersion13,
102 + availableCols: map[string]bool{"total_exec_time": true, "calls": true},
103 + input: "totalTime",
104 + expected: "total_exec_time",
105 + },
106 + "calls unchanged on any version": {
107 + pgVersion: pgVersion13,
108 + availableCols: map[string]bool{"total_exec_time": true, "calls": true},
109 + input: "calls",
110 + expected: "calls",
111 + },
112 + "invalid column falls back to default (PG12)": {
113 + pgVersion: pgVersionOld,
114 + availableCols: map[string]bool{"total_time": true},
115 + input: "invalid_column",
116 + expected: "total_time",
117 + },
118 + "invalid column falls back to default (PG13)": {
119 + pgVersion: pgVersion13,
120 + availableCols: map[string]bool{"total_exec_time": true},
121 + input: "invalid_column",
122 + expected: "total_exec_time",
123 + },
124 + "SQL injection attempt falls back to default": {
125 + pgVersion: pgVersion13,
126 + availableCols: map[string]bool{"total_exec_time": true},
127 + input: "'; DROP TABLE users;--",
128 + expected: "total_exec_time",
129 + },
130 + }
131 +
132 + for name, tc := range tests {
133 + t.Run(name, func(t *testing.T) {
134 + c := &Collector{pgVersion: tc.pgVersion}
135 + result := c.mapAndValidateSortColumn(tc.input, tc.availableCols)
136 + assert.Equal(t, tc.expected, result)
137 + })
138 + }
139 +}
140 +
141 +func TestCollector_buildAvailableColumns(t *testing.T) {
142 + tests := map[string]struct {
143 + pgVersion int
144 + availableCols map[string]bool
145 + expectCols []string // UI keys we expect to see
146 + notExpectCols []string // UI keys we don't expect
147 + }{
148 + "PG12 with basic columns": {
149 + pgVersion: pgVersionOld,
150 + availableCols: map[string]bool{
151 + "queryid": true, "query": true, "calls": true,
152 + "total_time": true, "mean_time": true, "min_time": true, "max_time": true,
153 + "rows": true, "shared_blks_hit": true, "shared_blks_read": true,
154 + },
155 + expectCols: []string{"queryid", "query", "calls", "totalTime", "meanTime", "rows"},
156 + notExpectCols: []string{"plans", "totalPlanTime", "walRecords"}, // PG13+ only
157 + },
158 + "PG13 with exec_time columns": {
159 + pgVersion: pgVersion13,
160 + availableCols: map[string]bool{
161 + "queryid": true, "query": true, "calls": true,
162 + "total_exec_time": true, "mean_exec_time": true,
163 + "rows": true, "plans": true, "total_plan_time": true,
164 + "wal_records": true,
165 + },
166 + expectCols: []string{"queryid", "query", "calls", "totalTime", "plans", "walRecords"},
167 + },
168 + }
169 +
170 + for name, tc := range tests {
171 + t.Run(name, func(t *testing.T) {
172 + c := &Collector{pgVersion: tc.pgVersion}
173 + cols := c.buildAvailableColumns(tc.availableCols)
174 +
175 + // Build map of UI keys for easy lookup
176 + uiKeys := make(map[string]bool)
177 + for _, col := range cols {
178 + uiKeys[col.uiKey] = true
179 + }
180 +
181 + for _, key := range tc.expectCols {
182 + assert.True(t, uiKeys[key], "expected column %s to be present", key)
183 + }
184 + for _, key := range tc.notExpectCols {
185 + assert.False(t, uiKeys[key], "did not expect column %s to be present", key)
186 + }
187 + })
188 + }
189 +}
190 +
191 +func TestCollector_buildDynamicSQL(t *testing.T) {
192 + tests := map[string]struct {
193 + pgVersion int
194 + sortColumn string
195 + checkPG13 bool
196 + }{
197 + "PG12 builds valid SQL": {
198 + pgVersion: pgVersionOld,
199 + sortColumn: "total_time",
200 + checkPG13: false,
201 + },
202 + "PG13 builds valid SQL with exec_time": {
203 + pgVersion: pgVersion13,
204 + sortColumn: "total_exec_time",
205 + checkPG13: true,
206 + },
207 + }
208 +
209 + for name, tc := range tests {
210 + t.Run(name, func(t *testing.T) {
211 + c := &Collector{pgVersion: tc.pgVersion}
212 +
213 + // Build minimal column set for test
214 + cols := []pgColumnMeta{
215 + {dbColumn: "s.queryid", uiKey: "queryid", dataType: ftString},
216 + {dbColumn: "s.query", uiKey: "query", dataType: ftString},
217 + {dbColumn: "s.calls", uiKey: "calls", dataType: ftInteger},
218 + {dbColumn: "total_time", uiKey: "totalTime", dataType: ftDuration},
219 + }
220 +
221 + sql := c.buildDynamicSQL(cols, tc.sortColumn, 500)
222 +
223 + assert.Contains(t, sql, "pg_stat_statements")
224 + assert.Contains(t, sql, tc.sortColumn)
225 + assert.Contains(t, sql, "LIMIT 500")
226 + assert.Contains(t, sql, "s.queryid")
227 + })
228 + }
229 +}
230 +
231 +func TestCollector_buildDynamicColumns(t *testing.T) {
232 + c := &Collector{}
233 +
234 + cols := []pgColumnMeta{
235 + {uiKey: "queryid", displayName: "Query ID", dataType: ftString, visible: false, isUniqueKey: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
236 + {uiKey: "query", displayName: "Query", dataType: ftString, visible: true, isSticky: true, fullWidth: true, transform: trNone, sortDir: sortAsc, summary: summaryCount, filter: filterMulti},
237 + {uiKey: "totalTime", displayName: "Total Time", dataType: ftDuration, units: "seconds", visible: true, transform: trDuration, decimalPoints: 2, sortDir: sortDesc, summary: summarySum, filter: filterRange},
238 + }
239 +
240 + columns := c.buildDynamicColumns(cols)
241 +
242 + // Verify column count
243 + assert.Len(t, columns, 3)
244 +
245 + // Verify queryid column
246 + queryidCol := columns["queryid"].(map[string]any)
247 + assert.Equal(t, "Query ID", queryidCol["name"])
248 + assert.Equal(t, "string", queryidCol["type"])
249 + assert.True(t, queryidCol["unique_key"].(bool))
250 + assert.False(t, queryidCol["visible"].(bool))
251 + assert.Equal(t, 0, queryidCol["index"])
252 +
253 + // Verify query column
254 + queryCol := columns["query"].(map[string]any)
255 + assert.Equal(t, "Query", queryCol["name"])
256 + assert.True(t, queryCol["sticky"].(bool))
257 + assert.True(t, queryCol["full_width"].(bool))
258 + assert.Equal(t, 1, queryCol["index"])
259 +
260 + // Verify totalTime column
261 + totalTimeCol := columns["totalTime"].(map[string]any)
262 + assert.Equal(t, "Total Time", totalTimeCol["name"])
263 + assert.Equal(t, "duration", totalTimeCol["type"])
264 + assert.Equal(t, "seconds", totalTimeCol["units"])
265 + assert.Equal(t, "bar", totalTimeCol["visualization"]) // duration uses bar
266 + assert.Equal(t, 2, totalTimeCol["index"])
267 +}
268 +
269 +// Test that method config sort options have valid column references
270 +func TestPgMethods_SortOptionsHaveLabels(t *testing.T) {
271 + methods := pgMethods()
272 +
273 + for _, method := range methods {
274 + var sortParam *funcapi.ParamConfig
275 + for i := range method.RequiredParams {
276 + if method.RequiredParams[i].ID == "__sort" {
277 + sortParam = &method.RequiredParams[i]
278 + break
279 + }
280 + }
281 + assert.NotNil(t, sortParam)
282 + for _, opt := range sortParam.Options {
283 + assert.NotEmpty(t, opt.ID, "sort option must have ID")
284 + assert.NotEmpty(t, opt.Name, "sort option %s must have Name", opt.ID)
285 + assert.Contains(t, opt.Name, "Top queries by", "label should have standard prefix")
286 + }
287 + }
288 +}
src/go/plugin/go.d/pkg/strmutil/truncate.go new
+36
@@ -0,0 +1,36 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +// Package strmutil provides string manipulation utilities.
4 +package strmutil
5 +
6 +import "unicode/utf8"
7 +
8 +// TruncateText limits text length to maxLen bytes.
9 +// UTF-8 safe - does not split multi-byte characters.
10 +// Appends "..." when truncation occurs.
11 +func TruncateText(text string, maxLen int) string {
12 + if maxLen <= 0 {
13 + return ""
14 + }
15 + if len(text) <= maxLen {
16 + return text
17 + }
18 + // UTF-8 safe truncation
19 + cutoff := 0
20 + ellipsis := "..."
21 + reserveForEllipsis := 3
22 + if maxLen < 3 {
23 + // Too small for ellipsis - just truncate without it
24 + ellipsis = ""
25 + reserveForEllipsis = 0
26 + }
27 + for i := 0; i < len(text); {
28 + _, size := utf8.DecodeRuneInString(text[i:])
29 + if cutoff+size > maxLen-reserveForEllipsis {
30 + break
31 + }
32 + cutoff += size
33 + i += size
34 + }
35 + return text[:cutoff] + ellipsis
36 +}
src/go/plugin/go.d/pkg/strmutil/truncate_test.go new
+110
@@ -0,0 +1,110 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package strmutil
4 +
5 +import (
6 + "strings"
7 + "testing"
8 +
9 + "github.com/stretchr/testify/assert"
10 +)
11 +
12 +func TestTruncateText(t *testing.T) {
13 + tests := map[string]struct {
14 + input string
15 + maxLen int
16 + expected string
17 + }{
18 + "short text unchanged": {
19 + input: "hello",
20 + maxLen: 100,
21 + expected: "hello",
22 + },
23 + "text exactly at limit": {
24 + input: "hello",
25 + maxLen: 5,
26 + expected: "hello",
27 + },
28 + "text truncated with ellipsis": {
29 + input: "hello world",
30 + maxLen: 8,
31 + expected: "hello...",
32 + },
33 + "empty string": {
34 + input: "",
35 + maxLen: 100,
36 + expected: "",
37 + },
38 + "UTF-8 characters preserved": {
39 + input: "日本語テスト",
40 + maxLen: 13, // 3 chars * 3 bytes each = 9 bytes + "..." = 12 bytes fits in 13
41 + expected: "日本語...",
42 + },
43 + "mixed UTF-8 and ASCII": {
44 + input: "hello世界test",
45 + maxLen: 13, // "hello" (5) + "世" (3) + "界" (3) = 11 bytes + "..." = 14, so truncate to fit
46 + expected: "hello世...",
47 + },
48 + "very small maxLen": {
49 + input: "hello",
50 + maxLen: 4, // only room for 1 char + "..."
51 + expected: "h...",
52 + },
53 + "emoji handling": {
54 + input: "test🚀emoji",
55 + maxLen: 8, // "test" (4) + "..." (3) = 7 bytes fits in 8
56 + expected: "test...",
57 + },
58 + "maxLen zero": {
59 + input: "hello",
60 + maxLen: 0,
61 + expected: "",
62 + },
63 + "maxLen negative": {
64 + input: "hello",
65 + maxLen: -1,
66 + expected: "",
67 + },
68 + "maxLen one ASCII": {
69 + input: "hello",
70 + maxLen: 1,
71 + expected: "h",
72 + },
73 + "maxLen two ASCII": {
74 + input: "hello",
75 + maxLen: 2,
76 + expected: "he",
77 + },
78 + "maxLen one UTF-8 too small": {
79 + input: "日本語", // each char is 3 bytes
80 + maxLen: 1, // can't fit any full rune
81 + expected: "",
82 + },
83 + "maxLen two UTF-8 too small": {
84 + input: "日本語",
85 + maxLen: 2, // still can't fit a 3-byte rune
86 + expected: "",
87 + },
88 + "maxLen three with long text": {
89 + input: "日本語",
90 + maxLen: 3, // only room for ellipsis, no content
91 + expected: "...",
92 + },
93 + }
94 +
95 + for name, tc := range tests {
96 + t.Run(name, func(t *testing.T) {
97 + result := TruncateText(tc.input, tc.maxLen)
98 + assert.Equal(t, tc.expected, result)
99 + })
100 + }
101 +}
102 +
103 +func TestTruncateText_LongQuery(t *testing.T) {
104 + // Simulate a real SQL query scenario
105 + longQuery := strings.Repeat("SELECT * FROM table WHERE id = ?; ", 200)
106 + result := TruncateText(longQuery, 4096)
107 +
108 + assert.LessOrEqual(t, len(result), 4096)
109 + assert.True(t, strings.HasSuffix(result, "..."))
110 +}
src/go/tools/functions-validation/README.md new
+56
@@ -0,0 +1,56 @@
1 +# Functions Validation (CLI + Containers)
2 +
3 +## TL;DR
4 +- Bring up databases with Docker Compose.
5 +- Use `go.d.plugin --function` with the configs in `./config`.
6 +- Config files live under `./config/go.d`.
7 +- Validate output against the embedded schema.
8 +- Use `./e2e.sh` for automated end-to-end checks in `/tmp`.
9 +
10 +## Start containers
11 +```
12 +docker compose up -d
13 +```
14 +
15 +## Example CLI run (Postgres)
16 +```
17 +cd ../../../
18 +src/go/go.d.plugin \
19 + --config-dir src/go/tools/functions-validation/config \
20 + --function postgres:top-queries \
21 + --function-args info
22 +```
23 +
24 +## Validate output
25 +```
26 +echo '{"status":200,"type":"table","columns":{},"data":[]}' | \
27 + (cd src/go && go run ./tools/functions-validation/validate)
28 +```
29 +
30 +## Validate output (require rows)
31 +```
32 +src/go/go.d.plugin \
33 + --config-dir src/go/tools/functions-validation/config \
34 + --function postgres:top-queries \
35 + --function-args __job:local \
36 + > /tmp/pg.json
37 +
38 +(cd src/go && go run ./tools/functions-validation/validate --input /tmp/pg.json --min-rows 1)
39 +```
40 +
41 +## E2E runner (recommended)
42 +```
43 +./e2e.sh
44 +```
45 +
46 +### Behavior
47 +- Creates a workspace under `/tmp` and runs Docker Compose there.
48 +- Builds `go.d.plugin` into the `/tmp` workspace.
49 +- Validates schema **and** that data rows are returned for top-queries.
50 +- Cleans up the `/tmp` workspace on success; keeps it on failure for debugging.
51 +
52 +## Notes
53 +- The compose file sets credentials that match the sample configs in `./config`.
54 +- MSSQL uses an init container to enable Query Store and seed data.
55 +- MongoDB enables the profiler to populate `system.profile`.
56 +- The validator reads the canonical schema at `src/plugins.d/FUNCTION_UI_SCHEMA.json`.
src/go/tools/functions-validation/config/go.d/mongodb.conf new
+5
@@ -0,0 +1,5 @@
1 +jobs:
2 + - name: local
3 + uri: "mongodb://root:rootpw@127.0.0.1:27017"
4 + top_queries_function_enabled: true
5 + top_queries_limit: 100
src/go/tools/functions-validation/config/go.d/mssql.conf new
+4
@@ -0,0 +1,4 @@
1 +jobs:
2 + - name: local
3 + dsn: "sqlserver://sa:Netdata123!@127.0.0.1:1433?database=netdata"
4 + top_queries_limit: 100
src/go/tools/functions-validation/config/go.d/mysql.conf new
+4
@@ -0,0 +1,4 @@
1 +jobs:
2 + - name: local
3 + dsn: "netdata:netdata@tcp(127.0.0.1:3306)/netdata"
4 + top_queries_limit: 100
src/go/tools/functions-validation/config/go.d/postgres.conf new
+4
@@ -0,0 +1,4 @@
1 +jobs:
2 + - name: local
3 + dsn: "postgres://netdata:netdata@127.0.0.1:5432/netdata?sslmode=disable"
4 + top_queries_limit: 100
src/go/tools/functions-validation/docker-compose.yml new
+88
@@ -0,0 +1,88 @@
1 +services:
2 + postgres:
3 + image: postgres:16
4 + environment:
5 + POSTGRES_USER: netdata
6 + POSTGRES_PASSWORD: netdata
7 + POSTGRES_DB: netdata
8 + command: ["postgres", "-c", "shared_preload_libraries=pg_stat_statements"]
9 + ports:
10 + - "5432:5432"
11 + volumes:
12 + - ./seed/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
13 + healthcheck:
14 + test: ["CMD-SHELL", "pg_isready -U netdata -d netdata"]
15 + interval: 5s
16 + timeout: 5s
17 + retries: 10
18 +
19 + mysql:
20 + image: mysql:8.0
21 + environment:
22 + MYSQL_ROOT_PASSWORD: rootpw
23 + MYSQL_DATABASE: netdata
24 + MYSQL_USER: netdata
25 + MYSQL_PASSWORD: netdata
26 + command: ["--performance_schema=ON"]
27 + ports:
28 + - "3306:3306"
29 + volumes:
30 + - ./seed/mysql/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
31 + healthcheck:
32 + test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -u root -p$$MYSQL_ROOT_PASSWORD > /dev/null"]
33 + interval: 5s
34 + timeout: 5s
35 + retries: 10
36 +
37 + mssql:
38 + image: mcr.microsoft.com/mssql/server:2022-latest
39 + environment:
40 + ACCEPT_EULA: "Y"
41 + MSSQL_SA_PASSWORD: "Netdata123!"
42 + ports:
43 + - "1433:1433"
44 + healthcheck:
45 + test: ["CMD-SHELL", "if [ -x /opt/mssql-tools18/bin/sqlcmd ]; then /opt/mssql-tools18/bin/sqlcmd -C -S localhost -U sa -P $$MSSQL_SA_PASSWORD -Q \"SELECT 1\" > /dev/null; else /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P $$MSSQL_SA_PASSWORD -Q \"SELECT 1\" > /dev/null; fi"]
46 + interval: 10s
47 + timeout: 5s
48 + retries: 12
49 +
50 + mssql-init:
51 + image: mcr.microsoft.com/mssql-tools
52 + depends_on:
53 + mssql:
54 + condition: service_healthy
55 + environment:
56 + MSSQL_SA_PASSWORD: "Netdata123!"
57 + volumes:
58 + - ./seed/mssql/init.sql:/seed/init.sql:ro
59 + - ./seed/mssql/init.sh:/seed/init.sh:ro
60 + entrypoint: ["/bin/bash", "/seed/init.sh"]
61 +
62 + mongo:
63 + image: mongo:6
64 + environment:
65 + MONGO_INITDB_ROOT_USERNAME: root
66 + MONGO_INITDB_ROOT_PASSWORD: rootpw
67 + ports:
68 + - "27017:27017"
69 + volumes:
70 + - ./seed/mongodb/init.js:/docker-entrypoint-initdb.d/init.js:ro
71 + healthcheck:
72 + test: ["CMD", "mongosh", "--quiet", "--username", "root", "--password", "rootpw", "--authenticationDatabase", "admin", "--eval", "db.runCommand({ping:1})"]
73 + interval: 5s
74 + timeout: 5s
75 + retries: 10
76 +
77 + mongo-init:
78 + image: mongo:6
79 + depends_on:
80 + mongo:
81 + condition: service_healthy
82 + environment:
83 + MONGO_INITDB_ROOT_USERNAME: root
84 + MONGO_INITDB_ROOT_PASSWORD: rootpw
85 + volumes:
86 + - ./seed/mongodb/init.js:/seed/init.js:ro
87 + - ./seed/mongodb/init.sh:/seed/init.sh:ro
88 + entrypoint: ["/bin/bash", "/seed/init.sh"]
src/go/tools/functions-validation/e2e.sh new
+143
@@ -0,0 +1,143 @@
1 +#!/usr/bin/env bash
2 +set -euo pipefail
3 +
4 +# Colors for output
5 +RED='\033[0;31m'
6 +GREEN='\033[0;32m'
7 +YELLOW='\033[1;33m'
8 +GRAY='\033[0;90m'
9 +NC='\033[0m' # No Color
10 +
11 +# Execute command with visibility
12 +run() {
13 + # Print the command being executed
14 + printf >&2 "${GRAY}$(pwd) >${NC} "
15 + printf >&2 "${YELLOW}"
16 + printf >&2 "%q " "$@"
17 + printf >&2 "${NC}\n"
18 +
19 + # Execute the command
20 + set +e
21 + "$@"
22 + local exit_code=$?
23 + set -e
24 + if [ $exit_code -ne 0 ]; then
25 + echo -e >&2 "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
26 + echo -e >&2 "${RED}[ERROR]${NC} Command failed with exit code ${exit_code}: ${YELLOW}$1${NC}"
27 + echo -e >&2 "${RED} Full command:${NC} $*"
28 + echo -e >&2 "${RED} Working dir:${NC} $(pwd)"
29 + echo -e >&2 "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
30 + return $exit_code
31 + fi
32 +}
33 +
34 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
35 +REPO_ROOT="$(cd "$SCRIPT_DIR/../../../.." && pwd)"
36 +WORKDIR="$(mktemp -d /tmp/netdata-functions-e2e.XXXXXX)"
37 +PROJECT_SUFFIX="$(basename "$WORKDIR")"
38 +PROJECT_SUFFIX="$(printf '%s' "$PROJECT_SUFFIX" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9_-' '-')"
39 +PROJECT_SUFFIX="${PROJECT_SUFFIX%-}"
40 +PROJECT="netdata-func-e2e-$PROJECT_SUFFIX"
41 +COMPOSE=(docker compose -f "$WORKDIR/docker-compose.yml" -p "$PROJECT")
42 +COMPOSE_STARTED=""
43 +
44 +cleanup() {
45 + local exit_code=$?
46 + set +e
47 + if [ -n "$COMPOSE_STARTED" ]; then
48 + run "${COMPOSE[@]}" down -v --remove-orphans
49 + fi
50 + if [ "$exit_code" -eq 0 ]; then
51 + run rm -rf "$WORKDIR"
52 + else
53 + echo "E2E failed. Keeping workspace: $WORKDIR" >&2
54 + fi
55 + exit $exit_code
56 +}
57 +trap cleanup EXIT
58 +
59 +wait_healthy() {
60 + local service="$1"
61 + local timeout="${2:-60}"
62 + local start=$SECONDS
63 +
64 + while true; do
65 + local cid
66 + cid=$("${COMPOSE[@]}" ps -q "$service")
67 + if [ -z "$cid" ]; then
68 + if [ $((SECONDS - start)) -ge "$timeout" ]; then
69 + echo "No container found for service: $service" >&2
70 + return 1
71 + fi
72 + sleep 2
73 + continue
74 + fi
75 +
76 + local status
77 + status="$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' "$cid")"
78 + if [ "$status" = "healthy" ]; then
79 + return 0
80 + fi
81 + if [ $((SECONDS - start)) -ge "$timeout" ]; then
82 + echo "Timed out waiting for $service to be healthy" >&2
83 + return 1
84 + fi
85 + sleep 2
86 + done
87 +}
88 +
89 +run cp -a "$SCRIPT_DIR/docker-compose.yml" "$SCRIPT_DIR/seed" "$SCRIPT_DIR/config" "$WORKDIR/"
90 +
91 +run "${COMPOSE[@]}" up -d
92 +COMPOSE_STARTED="yes"
93 +
94 +wait_healthy postgres 90
95 +wait_healthy mysql 90
96 +wait_healthy mssql 120
97 +wait_healthy mongo 90
98 +
99 +run "${COMPOSE[@]}" run --rm mongo-init
100 +
101 +run bash -c "cd \"$REPO_ROOT/src/go\" && go build -o \"$WORKDIR/go.d.plugin\" ./cmd/godplugin"
102 +
103 +validate() {
104 + local input="$1"
105 + shift
106 + (cd "$REPO_ROOT/src/go" && run go run ./tools/functions-validation/validate --input "$input" "$@")
107 +}
108 +
109 +run_info() {
110 + local module="$1"
111 + local output="$WORKDIR/${module}-info.json"
112 + run "$WORKDIR/go.d.plugin" \
113 + --config-dir "$WORKDIR/config" \
114 + --function "${module}:top-queries" \
115 + --function-args info \
116 + > "$output"
117 + validate "$output"
118 +}
119 +
120 +run_top_queries() {
121 + local module="$1"
122 + local output="$WORKDIR/${module}-top-queries.json"
123 + run "$WORKDIR/go.d.plugin" \
124 + --config-dir "$WORKDIR/config" \
125 + --function "${module}:top-queries" \
126 + --function-args __job:local \
127 + > "$output"
128 + validate "$output" --min-rows 1
129 +}
130 +
131 +run_info postgres
132 +run_top_queries postgres
133 +
134 +run_info mysql
135 +run_top_queries mysql
136 +
137 +run_info mssql
138 +run_top_queries mssql
139 +
140 +run_info mongodb
141 +run_top_queries mongodb
142 +
143 +echo "E2E checks passed." >&2
src/go/tools/functions-validation/seed/mongodb/init.js new
+11
@@ -0,0 +1,11 @@
1 +db = db.getSiblingDB("netdata")
2 +db.setProfilingLevel(2)
3 +
4 +db.sample.insertMany([
5 + { name: "alpha", value: 10 },
6 + { name: "beta", value: 20 },
7 + { name: "gamma", value: 30 }
8 +])
9 +
10 +db.sample.find({ value: { $gt: 15 } }).toArray()
11 +db.sample.updateOne({ name: "alpha" }, { $inc: { value: 1 } })
src/go/tools/functions-validation/seed/mongodb/init.sh new
+8
@@ -0,0 +1,8 @@
1 +#!/usr/bin/env bash
2 +set -euo pipefail
3 +
4 +host="${MONGO_HOST:-mongo}"
5 +user="${MONGO_INITDB_ROOT_USERNAME:-root}"
6 +pass="${MONGO_INITDB_ROOT_PASSWORD:-rootpw}"
7 +
8 +mongosh --quiet "mongodb://${user}:${pass}@${host}:27017/admin" /seed/init.js
src/go/tools/functions-validation/seed/mssql/init.sh new
+35
@@ -0,0 +1,35 @@
1 +#!/usr/bin/env bash
2 +set -euo pipefail
3 +
4 +# Colors for output
5 +RED='\033[0;31m'
6 +GREEN='\033[0;32m'
7 +YELLOW='\033[1;33m'
8 +GRAY='\033[0;90m'
9 +NC='\033[0m' # No Color
10 +
11 +# Execute command with visibility
12 +run() {
13 + # Print the command being executed
14 + printf >&2 "${GRAY}$(pwd) >${NC} "
15 + printf >&2 "${YELLOW}"
16 + printf >&2 "%q " "$@"
17 + printf >&2 "${NC}\n"
18 +
19 + # Execute the command
20 + if ! "$@"; then
21 + local exit_code=$?
22 + echo -e >&2 "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
23 + echo -e >&2 "${RED}[ERROR]${NC} Command failed with exit code ${exit_code}: ${YELLOW}$1${NC}"
24 + echo -e >&2 "${RED} Full command:${NC} $*"
25 + echo -e >&2 "${RED} Working dir:${NC} $(pwd)"
26 + echo -e >&2 "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
27 + return $exit_code
28 + fi
29 +}
30 +
31 +if [ -x /opt/mssql-tools18/bin/sqlcmd ]; then
32 + run /opt/mssql-tools18/bin/sqlcmd -C -S mssql -U sa -P "${MSSQL_SA_PASSWORD}" -i /seed/init.sql
33 +else
34 + run /opt/mssql-tools/bin/sqlcmd -S mssql -U sa -P "${MSSQL_SA_PASSWORD}" -i /seed/init.sql
35 +fi
src/go/tools/functions-validation/seed/mssql/init.sql new
+30
@@ -0,0 +1,30 @@
1 +IF DB_ID('netdata') IS NULL
2 +BEGIN
3 + CREATE DATABASE netdata;
4 +END
5 +GO
6 +
7 +ALTER DATABASE netdata SET QUERY_STORE = ON;
8 +GO
9 +
10 +USE netdata;
11 +GO
12 +
13 +IF OBJECT_ID('dbo.sample', 'U') IS NULL
14 +BEGIN
15 + CREATE TABLE dbo.sample (
16 + id INT IDENTITY(1,1) PRIMARY KEY,
17 + name NVARCHAR(64) NOT NULL,
18 + value INT NOT NULL
19 + );
20 +END
21 +GO
22 +
23 +INSERT INTO dbo.sample (name, value)
24 +VALUES ('alpha', 10), ('beta', 20), ('gamma', 30);
25 +GO
26 +
27 +SELECT COUNT(*) FROM dbo.sample;
28 +SELECT * FROM dbo.sample WHERE value > 15;
29 +UPDATE dbo.sample SET value = value + 1 WHERE name = 'alpha';
30 +GO
src/go/tools/functions-validation/seed/mysql/init.sql new
+34
@@ -0,0 +1,34 @@
1 +CREATE TABLE IF NOT EXISTS sample (
2 + id INT AUTO_INCREMENT PRIMARY KEY,
3 + name VARCHAR(64) NOT NULL,
4 + value INT NOT NULL
5 +);
6 +
7 +-- Ensure statement digest collection is enabled.
8 +UPDATE performance_schema.setup_consumers
9 + SET ENABLED = 'YES'
10 + WHERE NAME IN (
11 + 'events_statements_summary_by_digest',
12 + 'events_statements_summary_by_program',
13 + 'events_statements_summary_by_user_by_event_name',
14 + 'events_statements_summary_by_host_by_event_name',
15 + 'events_statements_summary_by_thread_by_event_name'
16 + );
17 +UPDATE performance_schema.setup_instruments
18 + SET ENABLED = 'YES', TIMED = 'YES'
19 + WHERE NAME LIKE 'statement/%';
20 +
21 +GRANT USAGE, REPLICATION CLIENT, PROCESS ON *.* TO 'netdata'@'%';
22 +GRANT SELECT ON performance_schema.* TO 'netdata'@'%';
23 +FLUSH PRIVILEGES;
24 +
25 +INSERT INTO sample (name, value)
26 +VALUES
27 + ('alpha', 10),
28 + ('beta', 20),
29 + ('gamma', 30);
30 +
31 +SELECT COUNT(*) FROM sample;
32 +SELECT * FROM sample WHERE value > 15;
33 +UPDATE sample SET value = value + 1 WHERE name = 'alpha';
34 +SELECT * FROM sample WHERE name = 'beta';
src/go/tools/functions-validation/seed/postgres/init.sql new
+17
@@ -0,0 +1,17 @@
1 +CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
2 +
3 +CREATE TABLE IF NOT EXISTS public.sample (
4 + id SERIAL PRIMARY KEY,
5 + name TEXT NOT NULL,
6 + value INTEGER NOT NULL
7 +);
8 +
9 +INSERT INTO public.sample (name, value)
10 +VALUES
11 + ('alpha', 10),
12 + ('beta', 20),
13 + ('gamma', 30);
14 +
15 +SELECT COUNT(*) FROM public.sample;
16 +SELECT * FROM public.sample WHERE value > 15;
17 +UPDATE public.sample SET value = value + 1 WHERE name = 'alpha';
src/go/tools/functions-validation/validate/main.go new
+114
@@ -0,0 +1,114 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +package main
4 +
5 +import (
6 + "bytes"
7 + "encoding/json"
8 + "flag"
9 + "fmt"
10 + "io"
11 + "os"
12 + "path/filepath"
13 +
14 + "github.com/santhosh-tekuri/jsonschema/v6"
15 +)
16 +
17 +func main() {
18 + var schemaPath string
19 + var inputPath string
20 + var minRows int
21 + var requireRows bool
22 +
23 + flag.StringVar(&schemaPath, "schema", "", "schema file path (optional)")
24 + flag.StringVar(&inputPath, "input", "", "input JSON file (default: stdin)")
25 + flag.IntVar(&minRows, "min-rows", 0, "minimum rows required in data responses (0 disables row check)")
26 + flag.BoolVar(&requireRows, "require-rows", false, "require at least one row in data responses")
27 + flag.Parse()
28 +
29 + schemaBytes, err := loadSchema(schemaPath)
30 + if err != nil {
31 + exitErr("load schema: %v", err)
32 + }
33 +
34 + inputBytes, err := loadInput(inputPath)
35 + if err != nil {
36 + exitErr("load input: %v", err)
37 + }
38 +
39 + var payload any
40 + if err := json.Unmarshal(inputBytes, &payload); err != nil {
41 + exitErr("parse input JSON: %v", err)
42 + }
43 +
44 + compiler := jsonschema.NewCompiler()
45 + if err := compiler.AddResource("schema.json", bytes.NewReader(schemaBytes)); err != nil {
46 + exitErr("add schema resource: %v", err)
47 + }
48 + schema, err := compiler.Compile("schema.json")
49 + if err != nil {
50 + exitErr("compile schema: %v", err)
51 + }
52 +
53 + if err := schema.Validate(payload); err != nil {
54 + exitErr("validation failed: %v", err)
55 + }
56 +
57 + if requireRows && minRows == 0 {
58 + minRows = 1
59 + }
60 + if minRows > 0 {
61 + rows, err := countRows(payload)
62 + if err != nil {
63 + exitErr("row check failed: %v", err)
64 + }
65 + if rows < minRows {
66 + exitErr("row check failed: expected at least %d rows, got %d", minRows, rows)
67 + }
68 + }
69 +}
70 +
71 +func countRows(payload any) (int, error) {
72 + obj, ok := payload.(map[string]any)
73 + if !ok {
74 + return 0, fmt.Errorf("expected JSON object")
75 + }
76 +
77 + if errMsg, ok := obj["errorMessage"]; ok {
78 + if s, ok := errMsg.(string); ok && s != "" {
79 + return 0, fmt.Errorf("error response: %s", s)
80 + }
81 + return 0, fmt.Errorf("error response without message")
82 + }
83 +
84 + data, ok := obj["data"]
85 + if !ok {
86 + return 0, fmt.Errorf("missing data field")
87 + }
88 +
89 + rows, ok := data.([]any)
90 + if !ok {
91 + return 0, fmt.Errorf("data is not an array")
92 + }
93 +
94 + return len(rows), nil
95 +}
96 +
97 +func loadSchema(path string) ([]byte, error) {
98 + if path == "" {
99 + path = filepath.Clean(filepath.Join("..", "plugins.d", "FUNCTION_UI_SCHEMA.json"))
100 + }
101 + return os.ReadFile(path)
102 +}
103 +
104 +func loadInput(path string) ([]byte, error) {
105 + if path == "" || path == "-" {
106 + return io.ReadAll(os.Stdin)
107 + }
108 + return os.ReadFile(path)
109 +}
110 +
111 +func exitErr(format string, args ...any) {
112 + fmt.Fprintf(os.Stderr, format+"\n", args...)
113 + os.Exit(1)
114 +}
src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md
+5 -4
@@ -376,7 +376,6 @@ Log explorer functions (`has_history: true`) provide advanced log analysis with
376 "id": "priority",
377 "name": "Log Level",
378 "order": 1,
379 - "defaultExpanded": true,
379 "options": [
380 {"id": "ERROR", "name": "ERROR", "count": 45, "order": 1},
381 {"id": "INFO", "name": "INFO", "count": 234, "order": 3}
@@ -416,9 +415,8 @@ Facets provide dynamic filtering with real-time counts computed by the backend:
415 "facets": [
416 {
417 "id": "level",
419 - "name": "Log Level",
418 + "name": "Log Level",
419 "order": 1,
421 - "defaultExpanded": true,
420 "options": [
421 {"id": "ERROR", "name": "ERROR", "count": 45, "order": 1},
422 {"id": "WARN", "name": "WARN", "count": 123, "order": 2},
@@ -875,10 +873,13 @@ For errors, return:
873 ```json
874 {
875 "status": 400,
878 - "error_message": "Descriptive error message"
876 + "errorMessage": "Descriptive error message"
877 }
878 ```
879
880 +**Compatibility note (cloud-frontend):**
881 +- The Functions UI expects `errorMessage` (camelCase) and does **not** camelize error payloads.
882 +
883 ### Performance Optimization
884
885 **For Large Datasets:**
src/plugins.d/FUNCTION_UI_REFERENCE.md
+58 -21
@@ -110,11 +110,22 @@ GET /api/v3/function?function=systemd-journal info after:1234567890 before:12345
110 ```
111
112 **Frontend Processing:**
113 -- `accepted_params`: Validates which parameters can be sent to function
113 +- `accepted_params`: **drives outgoing payload** (only params in this list are sent). When facets are selected, filters are restricted to `accepted_params` plus required param IDs.
114 - `required_params`: Generates filter UI, prevents execution if missing
115 - `v: 3`: Enables POST requests with JSON payloads
116 - Missing required parameters show user-friendly error messages
117
118 +**Parameter shapes (wire format):**
119 +- `accepted_params`: array of **strings** (parameter IDs). Example: `["sockets"]`, `["group"]`.
120 +- `required_params`: array of **objects** that define UI selectors.
121 + - Required fields: `id`, `name`, `type`, `options`
122 + - Common fields: `help`, `unique_view`
123 + - `options[]`: `id`, `name`, optional `defaultSelected`, `disabled`, `sort`
124 +
125 +**Cloud-frontend UI notes (verified):**
126 +- `type: "select"` renders as a **single-select**.
127 +- If no `defaultSelected`, UI selects the **first** option by default.
128 +
129 ### Backend Implementation
130
131 ```c
@@ -283,16 +294,22 @@ Row 3: message="error info", category="testing"
294 "visible": true, // Default visibility
295 "sticky": false, // Pin when scrolling
296 "visualization": "value", // How to render
286 - "transform": "none", // Value transformation
287 - "decimal_points": 2, // For numbers
288 - "units": "bytes", // Display units
297 + "value_options": { // Value formatting options
298 + "units": "bytes",
299 + "transform": "none",
300 + "decimal_points": 2,
301 + "default_value": ""
302 + },
303 "max": 100, // For bar types
304 + "pointer_to": "col_id", // Optional reference target
305 "sort": "descending", // Default sort
306 "sortable": true, // User can sort
307 "filter": "multiselect", // Filter type
308 "full_width": false, // Expand to fill
309 "wrap": false, // Text wrapping
295 - "summary": "sum" // Aggregation (backend only)
310 + "default_expanded_filter": false,
311 + "summary": "sum", // Aggregation (backend only)
312 + "dummy": false // True for hidden/internal columns
313 }
314 },
315
@@ -315,13 +332,19 @@ Row 3: message="error info", category="testing"
332 "columns": ["col1", "col2"]
333 }
334 },
318 - "accepted_params": [{
319 - "id": "param_id",
320 - "name": "Parameter Name",
321 - "type": "string|integer|boolean",
322 - "default": "default_value"
323 - }],
324 - "required_params": ["param1", "param2"]
335 + "accepted_params": ["param_id"],
336 + "required_params": [
337 + {
338 + "id": "param_id",
339 + "name": "Parameter Name",
340 + "type": "select",
341 + "unique_view": true,
342 + "options": [
343 + {"id": "opt1", "name": "Option 1", "defaultSelected": true},
344 + {"id": "opt2", "name": "Option 2"}
345 + ]
346 + }
347 + ]
348 }
349 ```
350
@@ -362,7 +385,6 @@ Special last element in data array for row styling:
385 "id": "priority", // Plain field name (not hash)
386 "name": "Priority",
387 "order": 1,
365 - "defaultExpanded": true,
388 "options": [
389 {
390 "id": "ERROR",
@@ -754,6 +776,7 @@ To add aggregated count support to a function:
776 | Type | Behavior | Notes |
777 |------|----------|-------|
778 | `boolean` | ValueCell | No special boolean UI |
779 +| `float` | ValueCell | Go functions may emit floats; UI falls back to default |
780 | `detail-string` | ValueCell | No expandable functionality |
781 | `array` | ValueCell | Works with `pill` visualization |
782 | `none` | ValueCell | Avoid using |
@@ -773,6 +796,8 @@ To add aggregated count support to a function:
796
797 ### Transform Types (RRDF_FIELD_TRANSFORM_*)
798
799 +**JSON path:** `columns[*].value_options.transform` (and `columns[*].value_options.decimal_points` for numeric formatting).
800 +
801 | Type | Input | Output | Notes |
802 |------|-------|--------|-------|
803 | `none` | Any | Unchanged | Default |
@@ -781,6 +806,7 @@ To add aggregated count support to a function:
806 | `datetime` | Epoch ms | Localized date/time | |
807 | `datetime_usec` | Epoch μs | Localized date/time | For logs |
808 | `xml` | XML string | Formatted XML | No specialized UI |
809 +| `text` | Any | Unchanged | UI falls back to default |
810
811 ### Conditional Patterns and Dependencies
812
@@ -831,7 +857,7 @@ buffer_rrdf_table_add_field(wb, field_id++, "row_options", "Row Options",
857 | `full_width` | 0x08 | Expand to fill space |
858 | `wrap` | 0x10 | Enable text wrapping |
859 | `dummy` | 0x20 | Internal use only |
834 -| `expanded_filter` | 0x40 | Expand filter by default |
860 +| `default_expanded_filter` | 0x40 | Expand filter by default |
861
862 ### Sort Options (RRDF_FIELD_SORT_*)
863
@@ -1370,18 +1396,26 @@ Frontend handles 304 responses gracefully without showing errors to users.
1396 - `update_every`: Default 1
1397 - `expires`: Cache control
1398 - `default_sort_column`: Initial sort
1399 +- `accepted_params`, `required_params`: Some backends include these in data responses (e.g., go.d functions)
1400 - All column options except `index`, `name`, `type`
1401
1402 +### UI-required fields (cloud-frontend)
1403 +- **Info response**: `v`, `type`, `has_history`, `accepted_params`, `required_params`, `help`
1404 +- **Data response**: `type`, `columns`, `data`
1405 +- **Error response**: `errorMessage` (camelCase) is used by the Functions UI
1406 +
1407 +**Note on casing:** cloud-frontend camelizes **successful** responses (info/data) before use, but **does not** camelize error payloads.
1408 +
1409 ## Error Handling
1410
1411 ### Error Response Format
1412
1379 -When a function encounters an error, the backend returns a JSON object. The primary error generation function (`rrd_call_function_error`) produces the following minimal format:
1413 +When a function encounters an error, the backend returns a JSON object. The primary error generator (`rrd_call_function_error`) produces the following minimal format:
1414
1415 ```json
1416 {
1383 - "status": 400, // The HTTP status code (e.g., 400, 404, 500)
1384 - "error_message": "A descriptive error message" // A human-readable message explaining the error
1417 + "status": 400, // The HTTP status code (e.g., 400, 404, 500)
1418 + "errorMessage": "A descriptive error message"
1419 }
1420 ```
1421
@@ -1389,15 +1423,18 @@ When a function encounters an error, the backend returns a JSON object. The prim
1423 The frontend is designed to handle a more comprehensive error structure, allowing for richer error display and localization. When an error occurs, the frontend will attempt to extract information from the received error object using the following hierarchy:
1424
1425 * **`status`**: The HTTP status code, used for general error classification (e.g., 400 for bad request, 404 for not found).
1392 -* **`error_message`**: The primary detailed message from the backend. This is the most consistently populated field by current backend functions.
1426 +* **`errorMessage`**: Primary detailed message used by the cloud-frontend Functions UI.
1427 * **`error`**: A short, machine-readable error identifier (e.g., "MissingParameter"). While not consistently generated by `rrd_call_function_error`, other parts of the system or future backend implementations might provide this.
1394 -* **`message`**: A user-friendly message. The frontend often maps `error_message` or an internal `errorMsgKey` to this for display.
1428 +* **`message`**: A user-friendly message. The frontend often maps `errorMessage` or an internal `errorMsgKey` to this for display.
1429 * **`help`**: Optional additional guidance for resolving the error. This field is not currently generated by `rrd_call_function_error`.
1430
1431 +**Compatibility note (cloud-frontend):**
1432 +- Cloud-frontend Functions UI expects `errorMessage` (camelCase) and does **not** camelize error payloads.
1433 +
1434 **Example of Frontend Interpretation (Conceptual):**
1398 -The frontend might internally map specific `error_message` strings to predefined `errorMsgKey` values to provide localized or more context-specific messages to the user. For instance, a backend `error_message` like "The 'time_range' parameter is required" might be mapped to an `errorMsgKey` of "ErrMissingTimeRange" in the frontend, which then displays a user-friendly message like "Please specify a time range for this function."
1435 +The frontend might internally map specific `errorMessage` strings to predefined `errorMsgKey` values to provide localized or more context-specific messages to the user. For instance, a backend `errorMessage` like "The 'time_range' parameter is required" might be mapped to an `errorMsgKey` of "ErrMissingTimeRange" in the frontend, which then displays a user-friendly message like "Please specify a time range for this function."
1436
1400 -Therefore, while the backend currently provides `status` and `error_message`, developers should be aware that the frontend's error handling is capable of utilizing the more detailed fields (`error`, `message`, `help`) if they are provided by the backend in the future or by other API endpoints.
1437 +Therefore, while the backend currently provides `status` and `errorMessage`, developers should be aware that the frontend's error handling is capable of utilizing the more detailed fields (`error`, `message`, `help`) if they are provided by the backend in the future or by other API endpoints.
1438
1439 **Common Error Codes:**
1440
src/plugins.d/FUNCTION_UI_SCHEMA.json new
+253
@@ -0,0 +1,253 @@
1 +{
2 + "$schema": "http://json-schema.org/draft-07/schema#",
3 + "title": "Netdata Functions UI Response",
4 + "type": "object",
5 + "required": ["status"],
6 + "properties": {
7 + "status": {
8 + "type": "integer"
9 + }
10 + },
11 + "oneOf": [
12 + { "$ref": "#/definitions/info_response" },
13 + { "$ref": "#/definitions/data_response" },
14 + { "$ref": "#/definitions/error_response" },
15 + { "$ref": "#/definitions/not_modified_response" }
16 + ],
17 + "definitions": {
18 + "field_type": {
19 + "type": "string",
20 + "enum": [
21 + "none",
22 + "integer",
23 + "float",
24 + "boolean",
25 + "string",
26 + "detail-string",
27 + "bar-with-integer",
28 + "duration",
29 + "timestamp",
30 + "array",
31 + "feedTemplate"
32 + ]
33 + },
34 + "field_visual": {
35 + "type": "string",
36 + "enum": [
37 + "value",
38 + "bar",
39 + "pill",
40 + "richValue",
41 + "rowOptions",
42 + "feedTemplate"
43 + ]
44 + },
45 + "field_transform": {
46 + "type": "string",
47 + "enum": [
48 + "none",
49 + "number",
50 + "duration",
51 + "datetime",
52 + "datetime_usec",
53 + "xml",
54 + "text"
55 + ]
56 + },
57 + "field_sort": {
58 + "type": "string",
59 + "enum": ["ascending", "descending"]
60 + },
61 + "field_summary": {
62 + "type": "string",
63 + "enum": ["count", "uniqueCount", "sum", "min", "max", "mean", "median"]
64 + },
65 + "field_filter": {
66 + "type": "string",
67 + "enum": ["none", "range", "multiselect", "facet"]
68 + },
69 + "value_options": {
70 + "type": "object",
71 + "properties": {
72 + "units": { "type": ["string", "null"] },
73 + "transform": { "$ref": "#/definitions/field_transform" },
74 + "decimal_points": { "type": "integer", "minimum": 0 },
75 + "default_value": { "type": ["string", "number", "boolean", "null"] }
76 + },
77 + "additionalProperties": true
78 + },
79 + "column": {
80 + "type": "object",
81 + "required": ["index", "name", "type"],
82 + "properties": {
83 + "index": { "type": "integer", "minimum": 0 },
84 + "unique_key": { "type": "boolean" },
85 + "name": { "type": "string" },
86 + "type": { "$ref": "#/definitions/field_type" },
87 + "units": { "type": ["string", "null"] },
88 + "visualization": { "$ref": "#/definitions/field_visual" },
89 + "value_options": { "$ref": "#/definitions/value_options" },
90 + "max": { "type": "number" },
91 + "pointer_to": { "type": ["string", "null"] },
92 + "sort": { "$ref": "#/definitions/field_sort" },
93 + "sortable": { "type": "boolean" },
94 + "sticky": { "type": "boolean" },
95 + "summary": { "$ref": "#/definitions/field_summary" },
96 + "filter": { "$ref": "#/definitions/field_filter" },
97 + "full_width": { "type": "boolean" },
98 + "wrap": { "type": "boolean" },
99 + "default_expanded_filter": { "type": "boolean" },
100 + "dummy": { "type": "boolean" }
101 + },
102 + "additionalProperties": true
103 + },
104 + "columns": {
105 + "type": "object",
106 + "additionalProperties": { "$ref": "#/definitions/column" }
107 + },
108 + "required_param_option": {
109 + "type": "object",
110 + "required": ["id", "name"],
111 + "properties": {
112 + "id": { "type": "string" },
113 + "name": { "type": "string" },
114 + "defaultSelected": { "type": "boolean" },
115 + "disabled": { "type": "boolean" },
116 + "sort": { "$ref": "#/definitions/field_sort" }
117 + },
118 + "additionalProperties": true
119 + },
120 + "required_param": {
121 + "type": "object",
122 + "required": ["id", "name", "type", "options"],
123 + "properties": {
124 + "id": { "type": "string" },
125 + "name": { "type": "string" },
126 + "type": { "enum": ["select", "multiselect"] },
127 + "options": {
128 + "type": "array",
129 + "items": { "$ref": "#/definitions/required_param_option" }
130 + },
131 + "help": { "type": "string" },
132 + "unique_view": { "type": "boolean" }
133 + },
134 + "additionalProperties": true
135 + },
136 + "accepted_params": {
137 + "type": "array",
138 + "items": { "type": "string" }
139 + },
140 + "required_params": {
141 + "type": "array",
142 + "items": { "$ref": "#/definitions/required_param" }
143 + },
144 + "chart_config": {
145 + "type": "object",
146 + "required": ["name", "type", "columns"],
147 + "properties": {
148 + "name": { "type": "string" },
149 + "type": { "type": "string" },
150 + "columns": {
151 + "type": "array",
152 + "items": { "type": "string" }
153 + }
154 + },
155 + "additionalProperties": true
156 + },
157 + "group_by_config": {
158 + "type": "object",
159 + "required": ["name", "columns"],
160 + "properties": {
161 + "name": { "type": "string" },
162 + "columns": {
163 + "type": "array",
164 + "items": { "type": "string" }
165 + }
166 + },
167 + "additionalProperties": true
168 + },
169 + "info_response": {
170 + "type": "object",
171 + "required": ["status", "type", "has_history", "accepted_params", "required_params", "v"],
172 + "properties": {
173 + "status": { "type": "integer" },
174 + "type": { "type": "string" },
175 + "has_history": { "type": "boolean" },
176 + "accepted_params": { "$ref": "#/definitions/accepted_params" },
177 + "required_params": { "$ref": "#/definitions/required_params" },
178 + "v": { "type": "integer" },
179 + "help": { "type": "string" },
180 + "update_every": { "type": "integer" },
181 + "expires": { "type": "integer" }
182 + },
183 + "allOf": [
184 + { "not": { "required": ["columns"] } },
185 + { "not": { "required": ["data"] } }
186 + ],
187 + "additionalProperties": true
188 + },
189 + "data_response": {
190 + "type": "object",
191 + "required": ["status", "type", "columns", "data"],
192 + "properties": {
193 + "status": { "type": "integer" },
194 + "type": { "type": "string", "enum": ["table", "log"] },
195 + "columns": { "$ref": "#/definitions/columns" },
196 + "data": {
197 + "type": "array",
198 + "items": {
199 + "type": "array",
200 + "items": {}
201 + }
202 + },
203 + "has_history": { "type": "boolean" },
204 + "accepted_params": { "$ref": "#/definitions/accepted_params" },
205 + "required_params": { "$ref": "#/definitions/required_params" },
206 + "help": { "type": "string" },
207 + "update_every": { "type": "integer" },
208 + "expires": { "type": "integer" },
209 + "default_sort_column": { "type": "string" },
210 + "group_by": {
211 + "type": "object",
212 + "additionalProperties": { "$ref": "#/definitions/group_by_config" }
213 + },
214 + "charts": {
215 + "type": "object",
216 + "additionalProperties": { "$ref": "#/definitions/chart_config" }
217 + },
218 + "default_charts": {
219 + "type": "array",
220 + "items": {
221 + "type": "array",
222 + "items": { "type": "string" }
223 + }
224 + },
225 + "facets": { "type": ["array", "object"] },
226 + "histogram": { "type": "object" },
227 + "items": { "type": "object" },
228 + "pagination": { "type": "object" }
229 + },
230 + "additionalProperties": true
231 + },
232 + "error_response": {
233 + "type": "object",
234 + "required": ["status", "errorMessage"],
235 + "properties": {
236 + "status": { "type": "integer", "minimum": 400 },
237 + "errorMessage": { "type": "string" },
238 + "error": { "type": "string" },
239 + "message": { "type": "string" },
240 + "help": { "type": "string" }
241 + },
242 + "additionalProperties": true
243 + },
244 + "not_modified_response": {
245 + "type": "object",
246 + "required": ["status"],
247 + "properties": {
248 + "status": { "const": 304 }
249 + },
250 + "additionalProperties": false
251 + }
252 + }
253 +}