@cryptotaxi247 / netdata-1 / commits / ffe23567b

Model Context Protocol (MCP) Part 2 (#20287)

* fix compilation warnings * added nd-mcp bridges to allow LLMs use stdio to communicate with Netdata over websockets * fixed bridges build scripts * fixed responses according to schema; added mcp ping; fixed mcp batch requests * added missing mcp-ping files * removed all non-standard methods; implemented logging and added completion mockup * adjusted websocket max sizes to reflect what browsers do; the bridges now use the sizes modern web browsers use; * netdata websockets implementation now fragments outbound messages when they exceed 4MiB. The max outbound websocket frame size can be negotiated per client * added more discovery methods, including full text search; bridges now never give up and reconnect when the websocket breaks * more tools * universal USER_AUTH structure that is now propagates from web client to websocket to mcp; added support for querying functions * prevent deadlock while shutting down websocket threads * functions managements at the mcp level * it should be const char pointer * revert apps.plugin changes for mcp * minor * more descriptive function failures * remove meaningless host match when searching for contexts * strict control on mcp execution parameters; rewrote and optimized functions managements on mcp * added the ability to query metrics * added cardinality_limit to data queries * expanded jsonwrap keys and mcp-info * move rrdcontexts to modes; added instances and dimensions to context details * added aclk logs when abnormally closing the connection * added dimensions, instances and labels per context * simplified contexts queries * query_metrics filtering instances and mcp instructions * fix labels filtering * cleanup resources and obsolete mcp responses * make it clear that a pipe is needed to separate patters * fix memory leaks * consolidated all metadata tools to one set of functions * working list_functions * support rfc3339 timestamps in all APIs * give only 1 option for filtering labels * mcp query metric now accepts structured parameters for lists of things * support scope_instances and scope_labels in data queries and weights * added scope_dimensions to data and weights endpoints * updated swagger * new contexts search that shows all items matches * make execute_function be a little smarter when a column is not found * fix memory leaks * like renamed to match to avoid relation with SQL * apply column transformations in mcp functions * properly parse transform * reworked functions error handling * enable rfc3339 in api/v23/data responses * fix contexts * support full text search in functions * fix strncpy() cases; isolate json_keys in libnetdata * fix codacy findings * renames * fix rrdlabels unittest to not leak memory on errors * no need to keep parser->user.chart_rrdlabels_linked_temporarily pointer, we just need a count * manually select dictionary traversal method * fix rrdlabels bug * fix memory leaks in new query target code * force the LLM to provide all the details of each query * weights endpoint for MCP * fix weights endpoints to be mcp/jsonrpc compatible * properly close the json of weights mcp endpoints * unify parameters parsing the schema generation * list_metrics and get_nodes_details support array of nodes * better list_nodes description * more strict parameters * properly support rfc3339 dates in the schema * update function names * make mcp function handle missing has_history node * added has_history:false * fix memory leaks * long keys in alerts * compact active and running alerts * cleanup alerts api * unify schema for all mcp tools * unified parameters and error handling; added new web page tool to test mcp * finished working on alerts * add missing return value to callback * added facets readme * updated facets readme * cleanup of mcp functions * working logs * working logs - added optimizations; cleanup * add cursor support for logs * fix instructions for logs * do not limit functions rows artifically when has_history=true * improve descriptions when functions give empty results * added API key to access full MCP; added user documentation * build nd-mcp as a standard; updated docs * fix the path of the mcp key file * enable mcp in websocket always * updated documentation for multiple mcp netdata connections * log json parsing errors * silently ignore legacy PING * add nd-mcp to spec file; added todo list * added minimum cardinality limit 30 to weights endpoints

Costa Tsaousis committed Jun 2, 2025 at 07:23 UTC ffe23567bfe5569198ce1193cc558e807a19b9a6
219 files changed +27054 -8695
CMakeLists.txt
+63 -12
@@ -839,6 +839,8 @@ set(LIBNETDATA_FILES
839 src/libnetdata/bitmap/bitmap64.h
840 src/libnetdata/buffer/buffer.c
841 src/libnetdata/buffer/buffer.h
842 + src/libnetdata/buffer/functions_fields.c
843 + src/libnetdata/buffer/functions_fields.h
844 src/libnetdata/ringbuffer/ringbuffer.c
845 src/libnetdata/ringbuffer/ringbuffer.h
846 src/libnetdata/circular_buffer/circular_buffer.c
@@ -876,6 +878,8 @@ set(LIBNETDATA_FILES
878 src/libnetdata/inlined.h
879 src/libnetdata/json/json.c
880 src/libnetdata/json/json.h
881 + src/libnetdata/json/json-keys.c
882 + src/libnetdata/json/json-keys.h
883 src/libnetdata/json/vendored/jsmn.c
884 src/libnetdata/json/vendored/jsmn.h
885 src/libnetdata/libnetdata.c
@@ -923,8 +927,10 @@ set(LIBNETDATA_FILES
927 src/libnetdata/string/utf8.h
928 src/libnetdata/worker_utilization/worker_utilization.c
929 src/libnetdata/worker_utilization/worker_utilization.h
926 - src/libnetdata/http/http_access.c
927 - src/libnetdata/http/http_access.h
930 + src/libnetdata/user-auth/http-access.c
931 + src/libnetdata/user-auth/http-access.h
932 + src/libnetdata/user-auth/user-auth.c
933 + src/libnetdata/user-auth/user-auth.h
934 src/libnetdata/http/http_defs.c
935 src/libnetdata/http/http_defs.h
936 src/libnetdata/http/content_type.c
@@ -1432,6 +1438,10 @@ set(API_PLUGIN_FILES
1438 src/web/api/queries/rrdr.h
1439 src/web/api/queries/query.c
1440 src/web/api/queries/query.h
1441 + src/web/api/queries/query-group-by.c
1442 + src/web/api/queries/query-group-over-time.c
1443 + src/web/api/queries/query-internal.h
1444 + src/web/api/queries/query-plan.c
1445 src/web/api/queries/average/average.c
1446 src/web/api/queries/average/average.h
1447 src/web/api/queries/countif/countif.c
@@ -1456,6 +1466,8 @@ set(API_PLUGIN_FILES
1466 src/web/api/queries/des/des.h
1467 src/web/api/queries/trimmed_mean/trimmed_mean.c
1468 src/web/api/queries/trimmed_mean/trimmed_mean.h
1469 + src/web/api/queries/extremes/extremes.c
1470 + src/web/api/queries/extremes/extremes.h
1471 src/web/api/queries/weights.c
1472 src/web/api/queries/weights.h
1473 src/web/api/formatters/rrd2json.c
@@ -1468,8 +1480,19 @@ set(API_PLUGIN_FILES
1480 src/web/api/formatters/ssv/ssv.h
1481 src/web/api/formatters/value/value.c
1482 src/web/api/formatters/value/value.h
1471 - src/web/api/formatters/json_wrapper.c
1472 - src/web/api/formatters/json_wrapper.h
1483 + src/web/api/formatters/jsonwrap.c
1484 + src/web/api/formatters/jsonwrap.h
1485 + src/web/api/formatters/jsonwrap-internal.h
1486 + src/web/api/formatters/jsonwrap-objects-tree.c
1487 + src/web/api/formatters/jsonwrap-query-plan.c
1488 + src/web/api/formatters/jsonwrap-summary-alerts.c
1489 + src/web/api/formatters/jsonwrap-summary-contexts.c
1490 + src/web/api/formatters/jsonwrap-summary-dimensions.c
1491 + src/web/api/formatters/jsonwrap-summary-instances.c
1492 + src/web/api/formatters/jsonwrap-summary-labels.c
1493 + src/web/api/formatters/jsonwrap-summary-nodes.c
1494 + src/web/api/formatters/jsonwrap-v1.c
1495 + src/web/api/formatters/jsonwrap-v2.c
1496 src/web/api/formatters/charts2json.c
1497 src/web/api/formatters/charts2json.h
1498 src/web/api/formatters/rrdset2json.c
@@ -1628,6 +1651,8 @@ set(RRD_PLUGIN_FILES
1651 src/database/rrdfunctions-inline.h
1652 src/database/rrdhost.c
1653 src/database/rrdlabels.c
1654 + src/database/rrdlabels-aggregated.c
1655 + src/database/rrdlabels-aggregated.h
1656 src/database/rrd.c
1657 src/database/rrd.h
1658 src/database/rrd-metadata.c
@@ -1847,26 +1872,45 @@ set(WEB_PLUGIN_FILES
1872 src/web/api/functions/function-metrics-cardinality.h
1873 src/web/api/queries/backfill.c
1874 src/web/api/queries/backfill.h
1850 - src/web/api/request_source.c
1851 - src/web/api/request_source.h
1875 src/web/api/v3/api_v3_stream_info.c
1876 src/web/api/v3/api_v3_stream_path.c
1877 src/web/mcp/adapters/mcp-websocket.c
1878 src/web/mcp/adapters/mcp-websocket.h
1856 - src/web/mcp/mcp-context.c
1857 - src/web/mcp/mcp-context.h
1879 src/web/mcp/mcp-initialize.c
1880 src/web/mcp/mcp-initialize.h
1860 - src/web/mcp/mcp-notifications.c
1861 - src/web/mcp/mcp-notifications.h
1881 src/web/mcp/mcp-prompts.c
1882 src/web/mcp/mcp-prompts.h
1883 src/web/mcp/mcp-resources.c
1884 src/web/mcp/mcp-resources.h
1866 - src/web/mcp/mcp-system.c
1867 - src/web/mcp/mcp-system.h
1885 src/web/mcp/mcp-tools.c
1886 src/web/mcp/mcp-tools.h
1887 + src/web/mcp/mcp-tools-list-metadata.c
1888 + src/web/mcp/mcp-tools-list-metadata.h
1889 + src/web/mcp/mcp-tools-execute-function.c
1890 + src/web/mcp/mcp-tools-execute-function.h
1891 + src/web/mcp/mcp-tools-execute-function-internal.h
1892 + src/web/mcp/mcp-tools-execute-function-registry.c
1893 + src/web/mcp/mcp-tools-execute-function-registry.h
1894 + src/web/mcp/mcp-tools-query-metrics.c
1895 + src/web/mcp/mcp-tools-query-metrics.h
1896 + src/web/mcp/mcp-tools-weights.c
1897 + src/web/mcp/mcp-tools-weights.h
1898 + src/web/mcp/mcp-tools-alert-transitions.c
1899 + src/web/mcp/mcp-tools-alert-transitions.h
1900 + src/web/mcp/mcp-tools-configured-alerts.c
1901 + src/web/mcp/mcp-tools-configured-alerts.h
1902 + src/web/mcp/mcp-params.c
1903 + src/web/mcp/mcp-params.h
1904 + src/web/mcp/mcp-request-id.c
1905 + src/web/mcp/mcp-request-id.h
1906 + src/web/mcp/mcp-ping.c
1907 + src/web/mcp/mcp-ping.h
1908 + src/web/mcp/mcp-logging.c
1909 + src/web/mcp/mcp-logging.h
1910 + src/web/mcp/mcp-completion.c
1911 + src/web/mcp/mcp-completion.h
1912 + src/web/mcp/mcp-api-key.c
1913 + src/web/mcp/mcp-api-key.h
1914 src/web/mcp/mcp.c
1915 src/web/mcp/mcp.h
1916 src/web/server/static/static-threaded.c
@@ -3245,6 +3289,13 @@ if(ENABLE_PLUGIN_GO)
3289 install(PROGRAMS ${CMAKE_BINARY_DIR}/go.d.plugin
3290 COMPONENT plugin-go
3291 DESTINATION usr/libexec/netdata/plugins.d)
3292 +
3293 + # Build and install nd-mcp (stdio-golang bridge) exactly like go.d.plugin
3294 + add_go_target(nd-mcp-target nd-mcp src/web/mcp/bridges/stdio-golang .)
3295 +
3296 + install(PROGRAMS ${CMAKE_BINARY_DIR}/nd-mcp
3297 + COMPONENT plugin-go
3298 + DESTINATION "${BINDIR}")
3299 endif()
3300
3301 #
netdata.spec.in
+1
@@ -525,6 +525,7 @@ rm -rf "${RPM_BUILD_ROOT}"
525 %{_libdir}/%{name}
526 %{_sbindir}/%{name}
527 %{_sbindir}/netdatacli
528 +%{_sbindir}/nd-mcp
529 %if %{_have_log2journal}
530 %{_sbindir}/log2journal
531 %endif
src/aclk/aclk_query.c
+1 -1
@@ -122,7 +122,7 @@ int http_api_v2(mqtt_wss_client client, aclk_query_t *query)
122 web_client_set_conn_cloud(w);
123 w->port_acl = HTTP_ACL_ACLK | default_aclk_http_acl;
124 w->acl = w->port_acl;
125 - web_client_set_permissions(w, HTTP_ACCESS_MAP_OLD_MEMBER, HTTP_USER_ROLE_MEMBER, WEB_CLIENT_FLAG_AUTH_CLOUD);
125 + web_client_set_permissions(w, HTTP_ACCESS_MAP_OLD_MEMBER, HTTP_USER_ROLE_MEMBER, USER_AUTH_METHOD_CLOUD);
126
127 w->mode = HTTP_REQUEST_MODE_GET;
128 w->timings.tv_in = query->created_tv;
src/aclk/mqtt_websockets/mqtt_wss_client.c
+3 -2
@@ -617,8 +617,9 @@ int mqtt_wss_connect(
617 client->poll_fds[POLLFD_SOCKET].events = POLLIN;
618 // wait till MQTT connection is established
619 while (!client->mqtt_connected) {
620 - if(mqtt_wss_service(client, 60 * MSEC_PER_SEC)) {
621 - nd_log(NDLS_DAEMON, NDLP_ERR, "Error connecting to MQTT WSS server \"%s\", port %d.", host, port);
620 + int rc = mqtt_wss_service(client, 60 * MSEC_PER_SEC);
621 + if(rc) {
622 + nd_log(NDLS_DAEMON, NDLP_ERR, "Error connecting to MQTT WSS server \"%s\", port %d. Code: %d", host, port, rc);
623 return 2;
624 }
625 }
src/aclk/mqtt_websockets/ws_client.c
+6 -2
@@ -526,8 +526,11 @@ int ws_client_process_rx_ws(ws_client *client)
526 if (!rbuf_bytes_available(client->buf_read))
527 return WS_CLIENT_NEED_MORE_BYTES;
528 char *insert = rbuf_get_linear_insert_range(client->buf_to_mqtt, &size);
529 - if (!insert)
529 + if (!insert) {
530 + nd_log(NDLS_DAEMON, NDLP_ERR, "ACLK: WebSocket buffer full! Cannot process payload of %"PRIu64" bytes (processed %"PRIu64"/%"PRIu64"). Buffer capacity: %zu bytes",
531 + remaining, client->rx.payload_processed, client->rx.payload_length, rbuf_get_capacity(client->buf_to_mqtt));
532 return WS_CLIENT_BUFFER_FULL;
533 + }
534 size = (size > remaining) ? remaining : size;
535 size = rbuf_pop(client->buf_read, insert, size);
536 rbuf_bump_head(client->buf_to_mqtt, size);
@@ -597,7 +600,8 @@ int ws_client_process_rx_ws(ws_client *client)
600 return WS_CLIENT_PARSING_DONE;
601 case WS_PAYLOAD_PING_REQ_PAYLOAD:
602 if (client->rx.payload_length > rbuf_get_capacity(client->buf_read) / 2) {
600 - nd_log(NDLS_DAEMON, NDLP_ERR, "ACLK: Ping arrived with payload which is too big!");
603 + nd_log(NDLS_DAEMON, NDLP_ERR, "ACLK: Ping payload too big! Received %"PRIu64" bytes, maximum allowed %zu bytes (buffer capacity: %zu bytes)",
604 + client->rx.payload_length, rbuf_get_capacity(client->buf_read) / 2, rbuf_get_capacity(client->buf_read));
605 return WS_CLIENT_INTERNAL_ERROR;
606 }
607 BUF_READ_CHECK_AT_LEAST(client->rx.payload_length);
src/cli/cli.c
+2 -4
@@ -174,10 +174,8 @@ int main(int argc, char **argv)
174 command_string_size = 0;
175 command_string[0] = '\0';
176 for (i = 1 ; i < argc ; ++i) {
177 - size_t to_copy;
178 -
179 - to_copy = MIN(strlen(argv[i]), MAX_COMMAND_LENGTH - 1 - command_string_size);
180 - strncpy(command_string + command_string_size, argv[i], to_copy);
177 + size_t to_copy = MIN(strlen(argv[i]), MAX_COMMAND_LENGTH - 1 - command_string_size);
178 + memcpy(command_string + command_string_size, argv[i], to_copy);
179 command_string_size += to_copy;
180 command_string[command_string_size] = '\0';
181
src/collectors/apps.plugin/apps_functions.c
+5
@@ -361,6 +361,7 @@ void function_processes(const char *transaction, char *function,
361 #endif
362 #endif
363 add_value_field_ndd_with_max(wb, CPU, (NETDATA_DOUBLE)(total_cpu) / cpu_divisor);
364 +
365 add_value_field_ndd_with_max(wb, UserCPU, (NETDATA_DOUBLE)(p->values[PDF_UTIME]) / cpu_divisor);
366 add_value_field_ndd_with_max(wb, SysCPU, (NETDATA_DOUBLE)(p->values[PDF_STIME]) / cpu_divisor);
367 #if (PROCESSES_HAVE_CPU_GUEST_TIME)
@@ -392,6 +393,7 @@ void function_processes(const char *transaction, char *function,
393 #endif
394
395 add_value_field_ndd_with_max(wb, VMSize, (NETDATA_DOUBLE)p->values[PDF_VMSIZE] / memory_divisor);
396 +
397 #if (PROCESSES_HAVE_VMSWAP == 1)
398 add_value_field_ndd_with_max(wb, Swap, (NETDATA_DOUBLE)p->values[PDF_VMSWAP] / memory_divisor);
399 #endif
@@ -435,6 +437,7 @@ void function_processes(const char *transaction, char *function,
437 add_value_field_ndd_with_max(wb, FDsLimitPercent, p->openfds_limits_percent);
438 #endif
439 add_value_field_llu_with_max(wb, FDs, pid_openfds_sum(p));
440 +
441 add_value_field_llu_with_max(wb, Files, p->openfds.files);
442 add_value_field_llu_with_max(wb, Pipes, p->openfds.pipes);
443 add_value_field_llu_with_max(wb, Sockets, p->openfds.sockets);
@@ -543,6 +546,7 @@ void function_processes(const char *transaction, char *function,
546 RRDF_FIELD_TRANSFORM_NUMBER, 2, "%", CPU_max, RRDF_FIELD_SORT_DESCENDING, NULL,
547 RRDF_FIELD_SUMMARY_SUM, RRDF_FIELD_FILTER_RANGE,
548 RRDF_FIELD_OPTS_VISIBLE, NULL);
549 +
550 buffer_rrdf_table_add_field(wb, field_id++, "UserCPU", "User CPU time (100% = 1 core)",
551 RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
552 RRDF_FIELD_VISUAL_BAR, RRDF_FIELD_TRANSFORM_NUMBER, 2, "%", UserCPU_max,
@@ -738,6 +742,7 @@ void function_processes(const char *transaction, char *function,
742 RRDF_FIELD_TRANSFORM_NUMBER, 0, "fds", FDs_max, RRDF_FIELD_SORT_DESCENDING, NULL,
743 RRDF_FIELD_SUMMARY_SUM, RRDF_FIELD_FILTER_RANGE,
744 RRDF_FIELD_OPTS_NONE, NULL);
745 +
746 buffer_rrdf_table_add_field(wb, field_id++, "Files", "Open Files", RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
747 RRDF_FIELD_VISUAL_BAR, RRDF_FIELD_TRANSFORM_NUMBER, 0,
748 "fds",
src/collectors/apps.plugin/apps_plugin.c
+2 -2
@@ -815,14 +815,14 @@ int main(int argc, char **argv) {
815
816 #if (PROCESSES_HAVE_UID == 1)
817 if (enable_users_charts) {
818 - send_charts_updates_to_netdata(users_root_target, "user", "user", "User Processes");
818 + send_charts_updates_to_netdata(users_root_target, "user", "user", "User");
819 send_collected_data_to_netdata(users_root_target, "user", dt);
820 }
821 #endif
822
823 #if (PROCESSES_HAVE_GID == 1)
824 if (enable_groups_charts) {
825 - send_charts_updates_to_netdata(groups_root_target, "usergroup", "user_group", "User Group Processes");
825 + send_charts_updates_to_netdata(groups_root_target, "usergroup", "user_group", "User Group");
826 send_collected_data_to_netdata(groups_root_target, "usergroup", dt);
827 }
828 #endif
src/collectors/apps.plugin/apps_plugin.h
+1 -1
@@ -725,7 +725,7 @@ void send_resource_usage_to_netdata(usec_t dt);
725 void send_proc_states_count(usec_t dt);
726 #endif
727
728 -#define APPS_PLUGIN_PROCESSES_FUNCTION_DESCRIPTION "Detailed information on the currently running processes."
728 +#define APPS_PLUGIN_PROCESSES_FUNCTION_DESCRIPTION "Provides detailed process information including CPU usage, memory consumption, I/O statistics, file descriptors, page faults, and parent-child relationships (PPID) for all processes including those in containers."
729 void function_processes(const char *transaction, char *function,
730 usec_t *stop_monotonic_ut __maybe_unused, bool *cancelled __maybe_unused,
731 BUFFER *payload __maybe_unused, HTTP_ACCESS access,
src/collectors/cgroups.plugin/cgroup-internals.h
+3 -3
@@ -378,7 +378,7 @@ static inline char *cgroup_chart_type(char *buffer, struct cgroup *cg) {
378 buffer[0] = '\0';
379
380 if (cg->chart_id[0] == '\0' || (cg->chart_id[0] == '/' && cg->chart_id[1] == '\0'))
381 - strncpy(buffer, "cgroup_root", RRD_ID_LENGTH_MAX);
381 + strncpyz(buffer, "cgroup_root", RRD_ID_LENGTH_MAX - 1);
382 else if (is_cgroup_systemd_service(cg))
383 snprintfz(buffer, RRD_ID_LENGTH_MAX, "%s%s", services_chart_id_prefix, cg->chart_id);
384 else
@@ -387,8 +387,8 @@ static inline char *cgroup_chart_type(char *buffer, struct cgroup *cg) {
387 return buffer;
388 }
389
390 -#define RRDFUNCTIONS_CGTOP_HELP "View running containers"
391 -#define RRDFUNCTIONS_SYSTEMD_SERVICES_HELP "View systemd services"
390 +#define RRDFUNCTIONS_CGTOP_HELP "Lists active containers and cgroups with resource usage including CPU, memory, disk I/O, and network traffic."
391 +#define RRDFUNCTIONS_SYSTEMD_SERVICES_HELP "Shows systemd service cgroups with their process counts and resource consumption (CPU, memory, I/O)."
392
393 int cgroup_function_cgroup_top(BUFFER *wb, const char *function, BUFFER *payload, const char *source);
394 int cgroup_function_systemd_top(BUFFER *wb, const char *function, BUFFER *payload, const char *source);
src/collectors/diskspace.plugin/plugin_diskspace.c
+1 -1
@@ -9,7 +9,7 @@
9 #define DEFAULT_EXCLUDED_FILESYSTEMS_INODES "msdosfs msdos vfat overlayfs aufs* *unionfs"
10 #define CONFIG_SECTION_DISKSPACE "plugin:proc:diskspace"
11
12 -#define RRDFUNCTIONS_DISKSPACE_HELP "View mount point statistics"
12 +#define RRDFUNCTIONS_DISKSPACE_HELP "Displays filesystem mount points with space utilization, available capacity, and inode usage statistics."
13
14 #define MAX_STAT_USEC 10000LU
15 #define SLOW_UPDATE_EVERY 5
src/collectors/freeipmi.plugin/freeipmi_plugin.c
+1 -1
@@ -20,7 +20,7 @@
20
21 #define FREEIPMI_GLOBAL_FUNCTION_SENSORS() do { \
22 fprintf(stdout, PLUGINSD_KEYWORD_FUNCTION " GLOBAL \"ipmi-sensors\" %d \"%s\" \"top\" "HTTP_ACCESS_FORMAT" %d\n", \
23 - 5, "Displays current sensor state and readings", \
23 + 5, "Shows IPMI hardware sensor readings including temperatures, voltages, fan speeds, and component health status.", \
24 (HTTP_ACCESS_FORMAT_CAST)(HTTP_ACCESS_NONE), 100); \
25 } while(0)
26
src/collectors/network-viewer.plugin/network-viewer.c
+3 -3
@@ -27,7 +27,7 @@ static SPAWN_SERVER *spawn_srv = NULL;
27 #include "libnetdata/os/system-maps/system-services.h"
28
29 #define NETWORK_CONNECTIONS_VIEWER_FUNCTION "network-connections"
30 -#define NETWORK_CONNECTIONS_VIEWER_HELP "Network connections explorer"
30 +#define NETWORK_CONNECTIONS_VIEWER_HELP "Shows active network connections with protocol details, states, addresses, ports, and performance metrics."
31
32 #define SIMPLE_HASHTABLE_VALUE_TYPE LOCAL_SOCKET *
33 #define SIMPLE_HASHTABLE_NAME _AGGREGATED_SOCKETS
@@ -750,7 +750,7 @@ void network_viewer_function(const char *transaction, char *function __maybe_unu
750
751 // RTT
752 buffer_rrdf_table_add_field(wb, field_id++, "RTT", aggregated ? "Max Smoothed Round Trip Time" : "Smoothed Round Trip Time",
753 - RRDF_FIELD_TYPE_DURATION, RRDF_FIELD_VISUAL_VALUE, RRDF_FIELD_TRANSFORM_NUMBER,
753 + RRDF_FIELD_TYPE_DURATION, RRDF_FIELD_VISUAL_VALUE, RRDF_FIELD_TRANSFORM_DURATION_S,
754 2, "ms", st.max.tcpi_rtt / USEC_PER_MS, RRDF_FIELD_SORT_DESCENDING, NULL,
755 RRDF_FIELD_SUMMARY_MAX, RRDF_FIELD_FILTER_RANGE,
756 RRDF_FIELD_OPTS_VISIBLE,
@@ -758,7 +758,7 @@ void network_viewer_function(const char *transaction, char *function __maybe_unu
758
759 // Asymmetry RTT
760 buffer_rrdf_table_add_field(wb, field_id++, "RecvRTT", aggregated ? "Max Receiver ACKs RTT" : "Receiver ACKs RTT",
761 - RRDF_FIELD_TYPE_DURATION, RRDF_FIELD_VISUAL_VALUE, RRDF_FIELD_TRANSFORM_NUMBER,
761 + RRDF_FIELD_TYPE_DURATION, RRDF_FIELD_VISUAL_VALUE, RRDF_FIELD_TRANSFORM_DURATION_S,
762 2, "ms", st.max.tcpi_rcv_rtt / USEC_PER_MS, RRDF_FIELD_SORT_DESCENDING, NULL,
763 RRDF_FIELD_SUMMARY_MAX, RRDF_FIELD_FILTER_RANGE,
764 RRDF_FIELD_OPTS_VISIBLE,
src/collectors/proc.plugin/proc_diskstats.c
+3 -3
@@ -9,7 +9,7 @@
9 #define _COMMON_PLUGIN_MODULE_NAME PLUGIN_PROC_MODULE_DISKSTATS_NAME
10 #include "../common-contexts/common-contexts.h"
11
12 -#define RRDFUNCTIONS_DISKSTATS_HELP "View block device statistics"
12 +#define RRDFUNCTIONS_DISKSTATS_HELP "Displays block device I/O statistics including read/write throughput, operations, latency, and utilization."
13
14 #define DISK_TYPE_UNKNOWN 0
15 #define DISK_TYPE_PHYSICAL 1
@@ -455,10 +455,10 @@ static inline char *get_disk_name(unsigned long major, unsigned long minor, char
455 if(!path_to_device_label || !*path_to_device_label || !get_disk_name_from_path(path_to_device_label, result, FILENAME_MAX + 1, major, minor, disk, NULL, 0))
456 if(!path_to_veritas_volume_groups || !*path_to_veritas_volume_groups || !get_disk_name_from_path(path_to_veritas_volume_groups, result, FILENAME_MAX + 1, major, minor, disk, "vx", 2))
457 if(name_disks_by_id != CONFIG_BOOLEAN_YES || !path_to_device_id || !*path_to_device_id || !get_disk_name_from_path(path_to_device_id, result, FILENAME_MAX + 1, major, minor, disk, NULL, 0))
458 - strncpy(result, disk, FILENAME_MAX);
458 + strncpyz(result, disk, sizeof(result) - 1);
459
460 if(!result[0])
461 - strncpy(result, disk, FILENAME_MAX);
461 + strncpyz(result, disk, sizeof(result) - 1);
462
463 netdata_fix_chart_name(result);
464 return strdup(result);
src/collectors/proc.plugin/proc_net_dev.c
+1 -1
@@ -6,7 +6,7 @@
6 #define PLUGIN_PROC_MODULE_NETDEV_NAME "/proc/net/dev"
7 #define CONFIG_SECTION_PLUGIN_PROC_NETDEV "plugin:" PLUGIN_PROC_CONFIG_NAME ":" PLUGIN_PROC_MODULE_NETDEV_NAME
8
9 -#define RRDFUNCTIONS_NETDEV_HELP "View network interface statistics"
9 +#define RRDFUNCTIONS_NETDEV_HELP "Shows real-time network interface performance including traffic rates, packet counts, drops, and link status."
10
11 #define STATE_LENGTH_MAX 32
12
src/collectors/systemd-journal.plugin/systemd-internals.h
+4
@@ -17,6 +17,10 @@
17 #define ND_SD_JOURNAL_EXECUTE_WATCHER_PENDING_EVERY_MS 250
18 #define ND_SD_JOURNAL_ALL_FILES_SCAN_EVERY_USEC (5 * 60 * USEC_PER_SEC)
19
20 +#define ND_SD_UNITS_FUNCTION_DESCRIPTION "Lists all systemd units (services, timers, mounts, etc.) with their current state and status."
21 +#define ND_SD_UNITS_FUNCTION_NAME "systemd-list-units"
22 +#define ND_SD_UNITS_DEFAULT_TIMEOUT 30
23 +
24 extern __thread size_t fstat_thread_calls;
25 extern __thread size_t fstat_thread_cached_responses;
26 void fstat_cache_enable_on_thread(void);
src/collectors/systemd-journal.plugin/systemd-journal-files.c
+6 -2
@@ -474,14 +474,18 @@ static int nd_journal_file_to_json_array_cb(const DICTIONARY_ITEM *item, void *e
474 "s",
475 true);
476
477 + char last_ut[RFC3339_MAX_LENGTH];
478 + rfc3339_datetime_ut(last_ut, sizeof(last_ut), nd_jfs->last_ut, 0, true);
479 +
480 char info[1024];
481 snprintfz(
482 info,
483 sizeof(info),
481 - "%zu files, with a total size of %s, covering %s",
484 + "%zu files, total size %s, covering %s, last entry at %s",
485 nd_jfs->count,
486 size_for_humans,
484 - duration_for_humans);
487 + duration_for_humans,
488 + last_ut);
489
490 buffer_json_member_add_string(wb, "id", name);
491 buffer_json_member_add_string(wb, "name", name);
src/collectors/systemd-units.plugin/plugin_systemd_units.c
+2
@@ -1172,6 +1172,7 @@ static void netdata_systemd_units_function_info(const char *transaction)
1172
1173 buffer_json_member_add_uint64(wb, "status", HTTP_RESP_OK);
1174 buffer_json_member_add_string(wb, "type", "table");
1175 + buffer_json_member_add_boolean(wb, "has_history", false);
1176 buffer_json_member_add_string(wb, "help", ND_SD_UNITS_FUNCTION_DESCRIPTION);
1177
1178 buffer_json_finalize(wb);
@@ -1647,6 +1648,7 @@ void function_systemd_units(
1648
1649 buffer_json_member_add_uint64(wb, "status", HTTP_RESP_OK);
1650 buffer_json_member_add_string(wb, "type", "table");
1651 + buffer_json_member_add_boolean(wb, "has_history", false);
1652 buffer_json_member_add_time_t(wb, "update_every", 10);
1653 buffer_json_member_add_string(wb, "help", ND_SD_UNITS_FUNCTION_DESCRIPTION);
1654 buffer_json_member_add_array(wb, "data");
src/collectors/windows-events.plugin/windows-events-sources.c
+7 -3
@@ -298,10 +298,14 @@ static int wevt_source_to_json_array_cb(const DICTIONARY_ITEM *item, void *entry
298 char entries_for_humans[128];
299 entries_snprintf(entries_for_humans, sizeof(entries_for_humans), s->entries, "", false);
300
301 + char last_ut[RFC3339_MAX_LENGTH];
302 + rfc3339_datetime_ut(last_ut, sizeof(last_ut), s->last_ut, 0, true);
303 +
304 char info[1024];
302 - snprintfz(info, sizeof(info), "%zu channel%s, with a total size of %s, covering %s%s%s%s",
303 - s->count, s->count > 1 ? "s":"", size_for_humans, duration_for_humans,
304 - s->entries ? ", having " : "", s->entries ? entries_for_humans : "", s->entries ? " entries" : "");
305 + snprintfz(info, sizeof(info), "%zu channel%s, total size %s, covering %s%s%s%s, last entry at %s",
306 + s->count, s->count > 1 ? "s":"", size_for_humans, duration_for_humans,
307 + s->entries ? ", having " : "", s->entries ? entries_for_humans : "", s->entries ? " entries" : "",
308 + last_ut);
309
310 buffer_json_member_add_string(wb, "id", name);
311 buffer_json_member_add_string(wb, "name", name);
src/collectors/windows-events.plugin/windows-events.h
+1 -1
@@ -133,7 +133,7 @@ typedef enum {
133 #define ON_FTS_PRELOAD_XML 0
134 #define ON_FTS_PRELOAD_EVENT_DATA 1
135
136 -#define WEVT_FUNCTION_DESCRIPTION "View, search and analyze the Microsoft Windows Events log."
136 +#define WEVT_FUNCTION_DESCRIPTION "Provides full access to Windows Event logs with search, filtering, and analysis capabilities."
137 #define WEVT_FUNCTION_NAME "windows-events"
138
139 #define WINDOWS_EVENTS_WORKER_THREADS 5
src/collectors/windows.plugin/perflib-services.c
+2
@@ -200,6 +200,8 @@ dict_win_services_charts_cb(const DICTIONARY_ITEM *item __maybe_unused, void *va
200
201 rrdset_done(p->st_service_state);
202 }
203 +
204 + return 1;
205 }
206
207 int do_PerflibServices(int update_every, usec_t dt __maybe_unused)
src/daemon/daemon-shutdown-watcher.c
+4
@@ -142,6 +142,8 @@ void *watcher_main(void *arg)
142 watcher_wait_for_step(WATCHER_STEP_ID_WAIT_FOR_DBENGINE_COLLECTORS_TO_FINISH, shutdown_start_time);
143 watcher_wait_for_step(WATCHER_STEP_ID_STOP_DBENGINE_TIERS, shutdown_start_time);
144 watcher_wait_for_step(WATCHER_STEP_ID_STOP_METASYNC_THREADS, shutdown_start_time);
145 + watcher_wait_for_step(WATCHER_STEP_ID_STOP_WEBSOCKET_THREADS, shutdown_start_time);
146 + watcher_wait_for_step(WATCHER_STEP_ID_JOIN_STATIC_THREADS, shutdown_start_time);
147 watcher_wait_for_step(WATCHER_STEP_ID_CLOSE_SQL_DATABASES, shutdown_start_time);
148 watcher_wait_for_step(WATCHER_STEP_ID_REMOVE_PID_FILE, shutdown_start_time);
149 watcher_wait_for_step(WATCHER_STEP_ID_FREE_OPENSSL_STRUCTURES, shutdown_start_time);
@@ -184,6 +186,8 @@ void watcher_thread_start() {
186 "wait for dbengine collectors to finish";
187 watcher_steps[WATCHER_STEP_ID_STOP_DBENGINE_TIERS].msg = "stop dbengine tiers";
188 watcher_steps[WATCHER_STEP_ID_STOP_METASYNC_THREADS].msg = "stop metasync threads";
189 + watcher_steps[WATCHER_STEP_ID_STOP_WEBSOCKET_THREADS].msg = "stop websocket threads";
190 + watcher_steps[WATCHER_STEP_ID_JOIN_STATIC_THREADS].msg = "join static threads";
191 watcher_steps[WATCHER_STEP_ID_CLOSE_SQL_DATABASES].msg = "close SQL databases";
192 watcher_steps[WATCHER_STEP_ID_REMOVE_PID_FILE].msg = "remove pid file";
193 watcher_steps[WATCHER_STEP_ID_FREE_OPENSSL_STRUCTURES].msg = "free openssl structures";
src/daemon/daemon-shutdown-watcher.h
+2
@@ -23,6 +23,8 @@ typedef enum {
23 WATCHER_STEP_ID_WAIT_FOR_DBENGINE_COLLECTORS_TO_FINISH,
24 WATCHER_STEP_ID_STOP_DBENGINE_TIERS,
25 WATCHER_STEP_ID_STOP_METASYNC_THREADS,
26 + WATCHER_STEP_ID_STOP_WEBSOCKET_THREADS,
27 + WATCHER_STEP_ID_JOIN_STATIC_THREADS,
28 WATCHER_STEP_ID_CLOSE_SQL_DATABASES,
29 WATCHER_STEP_ID_REMOVE_PID_FILE,
30 WATCHER_STEP_ID_FREE_OPENSSL_STRUCTURES,
src/daemon/daemon-shutdown.c
+9 -1
@@ -24,6 +24,7 @@ void cgroup_netdev_link_destroy(void);
24 void bearer_tokens_destroy(void);
25 void alerts_by_x_cleanup(void);
26 void websocket_threads_join(void);
27 +void mcp_functions_registry_cleanup(void);
28
29 static bool abort_on_fatal = true;
30
@@ -300,10 +301,14 @@ static void netdata_cleanup_and_exit(EXIT_REASON reason, bool abnormal, bool exi
301 add_agent_event(EVENT_AGENT_SHUTDOWN_TIME, (int64_t)(now_monotonic_usec() - shutdown_start_time));
302
303 websocket_threads_join();
304 + watcher_step_complete(WATCHER_STEP_ID_STOP_WEBSOCKET_THREADS);
305 +
306 nd_thread_join_threads();
307 + watcher_step_complete(WATCHER_STEP_ID_JOIN_STATIC_THREADS);
308 +
309 sqlite_close_databases();
305 - watcher_step_complete(WATCHER_STEP_ID_CLOSE_SQL_DATABASES);
310 sqlite_library_shutdown();
311 + watcher_step_complete(WATCHER_STEP_ID_CLOSE_SQL_DATABASES);
312
313 // unlink the pid
314 if(pidfile && *pidfile && unlink(pidfile) != 0)
@@ -329,6 +334,7 @@ static void netdata_cleanup_and_exit(EXIT_REASON reason, bool abnormal, bool exi
334 netdata_main_spawn_server_cleanup();
335
336 fprintf(stderr, "Freeing all RRDHOSTs...\n");
337 + mcp_functions_registry_cleanup();
338 rrdhost_free_all();
339 dyncfg_shutdown();
340 rrd_functions_inflight_destroy();
@@ -391,6 +397,8 @@ static void netdata_cleanup_and_exit(EXIT_REASON reason, bool abnormal, bool exi
397 fprintf(stderr, "WARNING: STRING has %zu strings still allocated.\n",
398 strings_referenced);
399
400 + fprintf(stderr, "RRDLABELS remaining in registry: %d.\n", rrdlabels_registry_count());
401 +
402 fprintf(stderr, "All done, exiting...\n");
403 #endif
404
src/daemon/dyncfg/dyncfg-intercept.c
+2 -2
@@ -55,8 +55,8 @@ static void dyncfg_log_user_action(DYNCFG *df, struct dyncfg_call *dc) {
55 break;
56 }
57
58 - PARSED_REQUEST_SOURCE req;
59 - if(!parse_request_source(dc->source, &req)) {
58 + USER_AUTH req;
59 + if(!user_auth_from_source(dc->source, &req)) {
60 ND_LOG_STACK lgs[] = {
61 ND_LOG_FIELD_TXT(NDF_MODULE, "DYNCFG"),
62 ND_LOG_FIELD_STR(NDF_NIDL_NODE, localhost->hostname),
src/daemon/dyncfg/dyncfg-tree.c
+1 -1
@@ -144,7 +144,7 @@ static void dyncfg_tree_for_host(RRDHOST *host, BUFFER *wb, const char *path, co
144 }
145 buffer_json_object_close(wb); // attention
146
147 - buffer_json_agents_v2(wb, NULL, 0, false, false);
147 + buffer_json_agents_v2(wb, NULL, 0, false, false, 0);
148
149 buffer_json_finalize(wb);
150
src/database/contexts/api_v1_contexts.c
+11 -11
@@ -93,8 +93,8 @@ static inline int rrdmetric_to_json_callback(const DICTIONARY_ITEM *item, void *
93 }
94
95 buffer_json_member_add_string(wb, "name", string2str(rm->name));
96 - buffer_json_member_add_time_t(wb, "first_time_t", rm->first_time_s);
97 - buffer_json_member_add_time_t(wb, "last_time_t", rrd_flag_is_collected(rm) ? (long long)t->now : (long long)rm->last_time_s);
96 + buffer_json_member_add_time_t_formatted(wb, "first_time_t", rm->first_time_s, options & RRDCONTEXT_OPTION_RFC3339);
97 + buffer_json_member_add_time_t_formatted(wb, "last_time_t", rrd_flag_is_collected(rm) ? (long long)t->now : (long long)rm->last_time_s, options & RRDCONTEXT_OPTION_RFC3339);
98 buffer_json_member_add_boolean(wb, "collected", rrd_flag_is_collected(rm));
99
100 if(options & RRDCONTEXT_OPTION_SHOW_DELETED)
@@ -205,8 +205,8 @@ static inline int rrdinstance_to_json_callback(const DICTIONARY_ITEM *item, void
205 buffer_json_member_add_string(wb, "chart_type", rrdset_type_name(ri->chart_type));
206 buffer_json_member_add_uint64(wb, "priority", ri->priority);
207 buffer_json_member_add_time_t(wb, "update_every", ri->update_every_s);
208 - buffer_json_member_add_time_t(wb, "first_time_t", first_time_s);
209 - buffer_json_member_add_time_t(wb, "last_time_t", (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_s);
208 + buffer_json_member_add_time_t_formatted(wb, "first_time_t", first_time_s, options & RRDCONTEXT_OPTION_RFC3339);
209 + buffer_json_member_add_time_t_formatted(wb, "last_time_t", (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_s, options & RRDCONTEXT_OPTION_RFC3339);
210 buffer_json_member_add_boolean(wb, "collected", flags & RRD_FLAG_COLLECTED);
211
212 if(options & RRDCONTEXT_OPTION_SHOW_DELETED)
@@ -308,8 +308,8 @@ static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void
308 buffer_json_member_add_string(wb, "family", string2str(rc->family));
309 buffer_json_member_add_string(wb, "chart_type", rrdset_type_name(rc->chart_type));
310 buffer_json_member_add_uint64(wb, "priority", rc->priority);
311 - buffer_json_member_add_time_t(wb, "first_time_t", first_time_s);
312 - buffer_json_member_add_time_t(wb, "last_time_t", (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_s);
311 + buffer_json_member_add_time_t_formatted(wb, "first_time_t", first_time_s, options & RRDCONTEXT_OPTION_RFC3339);
312 + buffer_json_member_add_time_t_formatted(wb, "last_time_t", (flags & RRD_FLAG_COLLECTED) ? (long long)t_parent->now : (long long)last_time_s, options & RRDCONTEXT_OPTION_RFC3339);
313 buffer_json_member_add_boolean(wb, "collected", (flags & RRD_FLAG_COLLECTED));
314
315 if(options & RRDCONTEXT_OPTION_SHOW_DELETED)
@@ -326,9 +326,9 @@ static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void
326 rrd_reasons_to_buffer_json_array_items(rc->queue.queued_flags, wb);
327 buffer_json_array_close(wb);
328
329 - buffer_json_member_add_time_t(wb, "last_queued", (time_t)(rc->queue.queued_ut / USEC_PER_SEC));
330 - buffer_json_member_add_time_t(wb, "scheduled_dispatch", (time_t)(rc->queue.scheduled_dispatch_ut / USEC_PER_SEC));
331 - buffer_json_member_add_time_t(wb, "last_dequeued", (time_t)(rc->queue.dequeued_ut / USEC_PER_SEC));
329 + buffer_json_member_add_time_t_formatted(wb, "last_queued", (time_t)(rc->queue.queued_ut / USEC_PER_SEC), options & RRDCONTEXT_OPTION_RFC3339);
330 + buffer_json_member_add_time_t_formatted(wb, "scheduled_dispatch", (time_t)(rc->queue.scheduled_dispatch_ut / USEC_PER_SEC), options & RRDCONTEXT_OPTION_RFC3339);
331 + buffer_json_member_add_time_t_formatted(wb, "last_dequeued", (time_t)(rc->queue.dequeued_ut / USEC_PER_SEC), options & RRDCONTEXT_OPTION_RFC3339);
332 buffer_json_member_add_uint64(wb, "dispatches", rc->queue.dispatches);
333 buffer_json_member_add_uint64(wb, "hub_version", rc->hub.version);
334 buffer_json_member_add_uint64(wb, "version", rc->version);
@@ -337,8 +337,8 @@ static inline int rrdcontext_to_json_callback(const DICTIONARY_ITEM *item, void
337 rrd_reasons_to_buffer_json_array_items(rc->pp.queued_flags, wb);
338 buffer_json_array_close(wb);
339
340 - buffer_json_member_add_time_t(wb, "pp_last_queued", (time_t)(rc->pp.queued_ut / USEC_PER_SEC));
341 - buffer_json_member_add_time_t(wb, "pp_last_dequeued", (time_t)(rc->pp.dequeued_ut / USEC_PER_SEC));
340 + buffer_json_member_add_time_t_formatted(wb, "pp_last_queued", (time_t)(rc->pp.queued_ut / USEC_PER_SEC), options & RRDCONTEXT_OPTION_RFC3339);
341 + buffer_json_member_add_time_t_formatted(wb, "pp_last_dequeued", (time_t)(rc->pp.dequeued_ut / USEC_PER_SEC), options & RRDCONTEXT_OPTION_RFC3339);
342 buffer_json_member_add_uint64(wb, "pp_executed", rc->pp.executions);
343 }
344
src/database/contexts/api_v2_contexts.c
+732 -259
@@ -1,48 +1,25 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 #include "api_v2_contexts.h"
4 -
4 +#include "../rrdlabels-aggregated.h"
5 #include "aclk/aclk_capas.h"
6 +#include "web/mcp/mcp.h"
7 +#include "libnetdata/json/json-keys.h"
8
9 // ----------------------------------------------------------------------------
10 // /api/v2/contexts API
11
10 -static const char *fts_match_to_string(FTS_MATCH match) {
11 - switch(match) {
12 - case FTS_MATCHED_HOST:
13 - return "HOST";
14 -
15 - case FTS_MATCHED_CONTEXT:
16 - return "CONTEXT";
17 -
18 - case FTS_MATCHED_INSTANCE:
19 - return "INSTANCE";
20 -
21 - case FTS_MATCHED_DIMENSION:
22 - return "DIMENSION";
23 -
24 - case FTS_MATCHED_ALERT:
25 - return "ALERT";
26 -
27 - case FTS_MATCHED_ALERT_INFO:
28 - return "ALERT_INFO";
29 -
30 - case FTS_MATCHED_LABEL:
31 - return "LABEL";
32 -
33 - case FTS_MATCHED_FAMILY:
34 - return "FAMILY";
35 -
36 - case FTS_MATCHED_TITLE:
37 - return "TITLE";
38 -
39 - case FTS_MATCHED_UNITS:
40 - return "UNITS";
41 -
42 - default:
43 - return "NONE";
44 - }
45 -}
12 +// Enum for all match types - using bitmask to track multiple matches
13 +typedef enum {
14 + SEARCH_MATCH_NONE = 0,
15 + SEARCH_MATCH_CONTEXT_ID = (1 << 0),
16 + SEARCH_MATCH_CONTEXT_TITLE = (1 << 1),
17 + SEARCH_MATCH_CONTEXT_UNITS = (1 << 2),
18 + SEARCH_MATCH_CONTEXT_FAMILY = (1 << 3),
19 + SEARCH_MATCH_INSTANCE = (1 << 4),
20 + SEARCH_MATCH_DIMENSION = (1 << 5),
21 + SEARCH_MATCH_LABEL = (1 << 6),
22 +} SEARCH_MATCH_TYPE;
23
24 struct function_v2_entry {
25 size_t size;
@@ -57,16 +34,121 @@ struct function_v2_entry {
34
35 struct context_v2_entry {
36 size_t count;
60 - STRING *id;
37 + STRING *id; // DO NOT FREE THIS, IT IS NOT DUP'd
38 + STRING *title;
39 STRING *family;
40 STRING *units;
41 uint32_t priority;
42 time_t first_time_s;
43 time_t last_time_s;
44 + size_t nodes;
45 + size_t instances;
46 RRD_FLAGS flags;
67 - FTS_MATCH match;
47 + DICTIONARY *instances_dict;
48 + DICTIONARY *dimensions_dict;
49 + RRDLABELS_AGGREGATED *labels_aggregated;
50 +
51 + RRDCONTEXT *rc; // THIS IS TEMPORARY, WHILE REFERENCED, NOT TO BE CLEANED
52 +
53 + // For search results
54 + SEARCH_MATCH_TYPE matched_types; // Bitmask of all match types
55 + DICTIONARY *matched_instances;
56 + DICTIONARY *matched_dimensions;
57 + RRDLABELS_AGGREGATED *matched_labels;
58 +};
59 +
60 +struct category_entry {
61 + size_t count;
62 + DICTIONARY *contexts; // Dictionary to hold context names
63 };
64
65 +static void rrdcontext_categorize_and_output(BUFFER *wb, DICTIONARY *contexts_dict, size_t cardinality_limit) {
66 + size_t total_contexts = dictionary_entries(contexts_dict);
67 +
68 + // First pass: count categories
69 + DICTIONARY *categories = dictionary_create(DICT_OPTION_SINGLE_THREADED);
70 +
71 + struct context_v2_entry *z;
72 + dfe_start_read(contexts_dict, z) {
73 + const char *context_name = string2str(z->id);
74 + char category[256];
75 + const char *dot = strchr(context_name, '.');
76 + if (dot) {
77 + size_t prefix_len = dot - context_name;
78 + if (prefix_len > sizeof(category) - 1) prefix_len = sizeof(category) - 1;
79 + memcpy(category, context_name, prefix_len);
80 + category[prefix_len] = '\0';
81 + } else {
82 + strncpyz(category, context_name, sizeof(category) - 1);
83 + }
84 +
85 + struct category_entry *entry = dictionary_get(categories, category);
86 + if (!entry) {
87 + struct category_entry new_entry = {0};
88 + new_entry.contexts = dictionary_create(DICT_OPTION_SINGLE_THREADED);
89 + entry = dictionary_set(categories, category, &new_entry, sizeof(struct category_entry));
90 + }
91 + entry->count++;
92 + // Store the context name for sampling
93 + dictionary_set(entry->contexts, context_name, NULL, 0);
94 + }
95 + dfe_done(z);
96 +
97 + // Calculate how many samples per category
98 + size_t num_categories = dictionary_entries(categories);
99 + size_t samples_per_category = 3; // Default to 3
100 + if (num_categories > 0 && cardinality_limit > 0) {
101 + samples_per_category = cardinality_limit / num_categories;
102 + if (samples_per_category < 3) samples_per_category = 3;
103 + }
104 +
105 + // Add info object first
106 + buffer_json_member_add_object(wb, "__info__");
107 + buffer_json_member_add_string(wb, "status", "categorized");
108 + buffer_json_member_add_uint64(wb, "total_contexts", total_contexts);
109 + buffer_json_member_add_uint64(wb, "categories", num_categories);
110 + buffer_json_member_add_uint64(wb, "samples_per_category", samples_per_category);
111 + buffer_json_member_add_string(wb, "help", "Results grouped by category with samples. Use 'metrics' parameter with specific patterns like 'system.*' to get full details for a category.");
112 + buffer_json_object_close(wb);
113 +
114 + // Output categorized contexts
115 + struct category_entry *cat_entry;
116 + dfe_start_read(categories, cat_entry) {
117 + buffer_json_member_add_array(wb, cat_entry_dfe.name);
118 +
119 + size_t samples_shown = 0;
120 + void *ctx_name;
121 + dfe_start_read(cat_entry->contexts, ctx_name) {
122 + // Only show samples_per_category - 1 if we need to add "... more"
123 + size_t max_to_show = (cat_entry->count > samples_per_category) ? samples_per_category - 1 : cat_entry->count;
124 + if (samples_shown < max_to_show) {
125 + buffer_json_add_array_item_string(wb, ctx_name_dfe.name);
126 + samples_shown++;
127 + } else {
128 + break;
129 + }
130 + }
131 + dfe_done(ctx_name);
132 +
133 + // Add "... and X more" only if we have more contexts than the limit
134 + if (cat_entry->count > samples_per_category) {
135 + char msg[100];
136 + snprintf(msg, sizeof(msg), "... and %zu more", cat_entry->count - samples_shown);
137 + buffer_json_add_array_item_string(wb, msg);
138 + }
139 +
140 + buffer_json_array_close(wb);
141 + }
142 + dfe_done(cat_entry);
143 +
144 + // Cleanup
145 + dfe_start_write(categories, cat_entry) {
146 + dictionary_destroy(cat_entry->contexts);
147 + }
148 + dfe_done(cat_entry);
149 + dictionary_destroy(categories);
150 +}
151 +
152 static inline bool full_text_search_string(FTS_INDEX *fts, SIMPLE_PATTERN *q, STRING *ptr) {
153 fts->searches++;
154 fts->string_searches++;
@@ -79,75 +161,95 @@ static inline bool full_text_search_char(FTS_INDEX *fts, SIMPLE_PATTERN *q, char
161 return simple_pattern_matches(q, ptr);
162 }
163
82 -static FTS_MATCH rrdcontext_to_json_v2_full_text_search(struct rrdcontext_to_json_v2_data *ctl, RRDCONTEXT *rc, SIMPLE_PATTERN *q) {
83 - if(unlikely(full_text_search_string(&ctl->q.fts, q, rc->id) ||
84 - full_text_search_string(&ctl->q.fts, q, rc->family)))
85 - return FTS_MATCHED_CONTEXT;
86 -
87 - if(unlikely(full_text_search_string(&ctl->q.fts, q, rc->title)))
88 - return FTS_MATCHED_TITLE;
164 +// Structure to hold search results
165 +struct fts_search_results {
166 + SEARCH_MATCH_TYPE matched_types; // Bitmask of all match types
167 + DICTIONARY *matched_instances;
168 + DICTIONARY *matched_dimensions;
169 + RRDLABELS_AGGREGATED *matched_labels;
170 +};
171
90 - if(unlikely(full_text_search_string(&ctl->q.fts, q, rc->units)))
91 - return FTS_MATCHED_UNITS;
172 +static void rrdcontext_to_json_v2_full_text_search(struct rrdcontext_to_json_v2_data *ctl, RRDCONTEXT *rc, SIMPLE_PATTERN *q, struct fts_search_results *results) {
173 + // Initialize results
174 + results->matched_types = SEARCH_MATCH_NONE;
175 + results->matched_instances = NULL;
176 + results->matched_dimensions = NULL;
177 + results->matched_labels = NULL;
178 +
179 + // Check context-level matches
180 + // Always search ID - it's the primary identifier
181 + if(unlikely(full_text_search_string(&ctl->q.fts, q, rc->id))) {
182 + results->matched_types |= SEARCH_MATCH_CONTEXT_ID;
183 + }
184 +
185 + // Only search family if the option is enabled
186 + if((ctl->options & CONTEXTS_OPTION_FAMILY) && unlikely(full_text_search_string(&ctl->q.fts, q, rc->family))) {
187 + results->matched_types |= SEARCH_MATCH_CONTEXT_FAMILY;
188 + }
189 +
190 + // Only search title if the option is enabled
191 + if((ctl->options & CONTEXTS_OPTION_TITLES) && unlikely(full_text_search_string(&ctl->q.fts, q, rc->title))) {
192 + results->matched_types |= SEARCH_MATCH_CONTEXT_TITLE;
193 + }
194 +
195 + // Only search units if the option is enabled
196 + if((ctl->options & CONTEXTS_OPTION_UNITS) && unlikely(full_text_search_string(&ctl->q.fts, q, rc->units))) {
197 + results->matched_types |= SEARCH_MATCH_CONTEXT_UNITS;
198 + }
199
93 - FTS_MATCH matched = FTS_MATCHED_NONE;
200 +
201 RRDINSTANCE *ri;
202 dfe_start_read(rc->rrdinstances, ri) {
96 - if(matched) break;
97 -
203 if(ctl->window.enabled && !query_matches_retention(ctl->window.after, ctl->window.before, ri->first_time_s, (ri->flags & RRD_FLAG_COLLECTED) ? ctl->now : ri->last_time_s, 0))
204 continue;
205
101 - if(unlikely(full_text_search_string(&ctl->q.fts, q, ri->id)) ||
102 - (ri->name != ri->id && full_text_search_string(&ctl->q.fts, q, ri->name))) {
103 - matched = FTS_MATCHED_INSTANCE;
104 - break;
206 + // Check instance name match only if instances option is enabled
207 + if((ctl->options & CONTEXTS_OPTION_INSTANCES) &&
208 + (unlikely(full_text_search_string(&ctl->q.fts, q, ri->id)) ||
209 + (ri->name != ri->id && full_text_search_string(&ctl->q.fts, q, ri->name)))) {
210 + if(!results->matched_instances)
211 + results->matched_instances = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
212 + dictionary_set(results->matched_instances, string2str(ri->name), NULL, 0);
213 + results->matched_types |= SEARCH_MATCH_INSTANCE;
214 }
215
107 - RRDMETRIC *rm;
108 - dfe_start_read(ri->rrdmetrics, rm) {
109 - if(ctl->window.enabled && !query_matches_retention(ctl->window.after, ctl->window.before, rm->first_time_s, (rm->flags & RRD_FLAG_COLLECTED) ? ctl->now : rm->last_time_s, 0))
110 - continue;
111 -
112 - if(unlikely(full_text_search_string(&ctl->q.fts, q, rm->id)) ||
113 - (rm->name != rm->id && full_text_search_string(&ctl->q.fts, q, rm->name))) {
114 - matched = FTS_MATCHED_DIMENSION;
115 - break;
216 + // Check dimensions only if dimensions option is enabled
217 + if(ctl->options & CONTEXTS_OPTION_DIMENSIONS) {
218 + RRDMETRIC *rm;
219 + dfe_start_read(ri->rrdmetrics, rm) {
220 + if(ctl->window.enabled && !query_matches_retention(ctl->window.after, ctl->window.before, rm->first_time_s, (rm->flags & RRD_FLAG_COLLECTED) ? ctl->now : rm->last_time_s, 0))
221 + continue;
222 +
223 + if(unlikely(full_text_search_string(&ctl->q.fts, q, rm->id)) ||
224 + (rm->name != rm->id && full_text_search_string(&ctl->q.fts, q, rm->name))) {
225 + if(!results->matched_dimensions)
226 + results->matched_dimensions = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
227 + dictionary_set(results->matched_dimensions, string2str(rm->name), NULL, 0);
228 + results->matched_types |= SEARCH_MATCH_DIMENSION;
229 + }
230 }
231 + dfe_done(rm);
232 }
118 - dfe_done(rm);
233
120 - size_t label_searches = 0;
121 - RRDLABELS *labels = rrdinstance_labels(ri);
122 - if(unlikely(rrdlabels_entries(labels) &&
123 - rrdlabels_match_simple_pattern_parsed(labels, q, ':', &label_searches) == SP_MATCHED_POSITIVE)) {
124 - ctl->q.fts.searches += label_searches;
125 - ctl->q.fts.char_searches += label_searches;
126 - matched = FTS_MATCHED_LABEL;
127 - break;
128 - }
129 - ctl->q.fts.searches += label_searches;
130 - ctl->q.fts.char_searches += label_searches;
131 -
132 - if(ri->rrdset) {
133 - RRDSET *st = ri->rrdset;
134 - rw_spinlock_read_lock(&st->alerts.spinlock);
135 - for (RRDCALC *rcl = st->alerts.base; rcl; rcl = rcl->next) {
136 - if(unlikely(full_text_search_string(&ctl->q.fts, q, rcl->config.name))) {
137 - matched = FTS_MATCHED_ALERT;
138 - break;
139 - }
140 -
141 - if(unlikely(full_text_search_string(&ctl->q.fts, q, rcl->config.info))) {
142 - matched = FTS_MATCHED_ALERT_INFO;
143 - break;
234 + // Check labels only if labels option is enabled
235 + if(ctl->options & CONTEXTS_OPTION_LABELS) {
236 + size_t label_searches = 0;
237 + RRDLABELS *labels = rrdinstance_labels(ri);
238 + if(unlikely(rrdlabels_entries(labels))) {
239 + results->matched_labels = rrdlabels_full_text_search(labels, q, results->matched_labels, &label_searches);
240 +
241 + if(results->matched_labels) {
242 + results->matched_types |= SEARCH_MATCH_LABEL;
243 }
244 +
245 + ctl->q.fts.searches += label_searches;
246 + ctl->q.fts.char_searches += label_searches;
247 }
146 - rw_spinlock_read_unlock(&st->alerts.spinlock);
248 }
249 +
250 + // We don't check alerts anymore as they are less relevant for the search
251 }
252 dfe_done(ri);
150 - return matched;
253 }
254
255 static ssize_t rrdcontext_to_json_v2_add_context(void *data, RRDCONTEXT_ACQUIRED *rca, bool queryable_context __maybe_unused) {
@@ -158,11 +260,12 @@ static ssize_t rrdcontext_to_json_v2_add_context(void *data, RRDCONTEXT_ACQUIRED
260 if(ctl->window.enabled && !query_matches_retention(ctl->window.after, ctl->window.before, rc->first_time_s, (rc->flags & RRD_FLAG_COLLECTED) ? ctl->now : rc->last_time_s, 0))
261 return 0; // continue to next context
262
161 - FTS_MATCH match = ctl->q.host_match;
263 + struct fts_search_results search_results = {0};
264 +
265 if((ctl->mode & CONTEXTS_V2_SEARCH) && ctl->q.pattern) {
163 - match = rrdcontext_to_json_v2_full_text_search(ctl, rc, ctl->q.pattern);
266 + rrdcontext_to_json_v2_full_text_search(ctl, rc, ctl->q.pattern, &search_results);
267
165 - if(match == FTS_MATCHED_NONE)
268 + if(search_results.matched_types == SEARCH_MATCH_NONE)
269 return 0; // continue to next context
270 }
271
@@ -173,55 +276,57 @@ static ssize_t rrdcontext_to_json_v2_add_context(void *data, RRDCONTEXT_ACQUIRED
276
277 if(ctl->contexts.dict) {
278 struct context_v2_entry t = {
176 - .count = 1,
177 - .id = rc->id,
178 - .family = string_dup(rc->family),
179 - .units = string_dup(rc->units),
180 - .priority = rc->priority,
181 - .first_time_s = rc->first_time_s,
182 - .last_time_s = rc->last_time_s,
183 - .flags = rc->flags,
184 - .match = match,
279 + .count = 1,
280 + .id = rc->id,
281 + .title = string_dup(rc->title),
282 + .family = string_dup(rc->family),
283 + .units = string_dup(rc->units),
284 + .priority = rc->priority,
285 + .first_time_s = rc->first_time_s,
286 + .last_time_s = rc->last_time_s,
287 + .flags = rc->flags,
288 + .nodes = 1,
289 + .instances = dictionary_entries(rc->rrdinstances),
290 + .instances_dict = NULL,
291 + .dimensions_dict = NULL,
292 + .rc = rc,
293 + // Store search results
294 + .matched_types = search_results.matched_types,
295 + .matched_instances = search_results.matched_instances,
296 + .matched_dimensions = search_results.matched_dimensions,
297 + .matched_labels = search_results.matched_labels,
298 };
299
300 +
301 dictionary_set(ctl->contexts.dict, string2str(rc->id), &t, sizeof(struct context_v2_entry));
302 }
303
304 return 1;
305 }
306
193 -void buffer_json_agent_status_id(BUFFER *wb, size_t ai, usec_t duration_ut) {
194 - buffer_json_member_add_object(wb, "st");
195 - {
196 - buffer_json_member_add_uint64(wb, "ai", ai);
197 - buffer_json_member_add_uint64(wb, "code", 200);
198 - buffer_json_member_add_string(wb, "msg", "");
199 - if (duration_ut)
200 - buffer_json_member_add_double(wb, "ms", (NETDATA_DOUBLE) duration_ut / 1000.0);
201 - }
202 - buffer_json_object_close(wb);
203 -}
204 -
205 -void buffer_json_node_add_v2(BUFFER *wb, RRDHOST *host, size_t ni, usec_t duration_ut, bool status) {
206 - buffer_json_member_add_string(wb, "mg", host->machine_guid);
307 +void buffer_json_node_add_v2_mcp(BUFFER *wb, RRDHOST *host, size_t ni __maybe_unused) {
308 + buffer_json_member_add_string(wb, "machine_guid", host->machine_guid);
309
310 if(!UUIDiszero(host->node_id))
209 - buffer_json_member_add_uuid(wb, "nd", host->node_id.uuid);
210 - buffer_json_member_add_string(wb, "nm", rrdhost_hostname(host));
211 - buffer_json_member_add_uint64(wb, "ni", ni);
311 + buffer_json_member_add_uuid(wb, "node_id", host->node_id.uuid);
312
213 - if(status)
214 - buffer_json_agent_status_id(wb, 0, duration_ut);
313 + buffer_json_member_add_string(wb, "hostname", rrdhost_hostname(host));
314 +
315 + buffer_json_member_add_string(wb, "relationship",
316 + host == localhost ? "localhost" :
317 + (rrdhost_is_virtual(host) ? "virtual" : "child"));
318 +
319 + buffer_json_member_add_boolean(wb, "connected", rrdhost_is_online(host));
320 }
321
217 -static void rrdhost_receiver_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *key) {
322 +static void rrdhost_receiver_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *key, CONTEXTS_OPTIONS options) {
323 buffer_json_member_add_object(wb, key);
324 {
325 buffer_json_member_add_uint64(wb, "id", s->ingest.id);
326 buffer_json_member_add_int64(wb, "hops", s->ingest.hops);
327 buffer_json_member_add_string(wb, "type", rrdhost_ingest_type_to_string(s->ingest.type));
328 buffer_json_member_add_string(wb, "status", rrdhost_ingest_status_to_string(s->ingest.status));
224 - buffer_json_member_add_time_t(wb, "since", s->ingest.since);
329 + buffer_json_member_add_time_t_formatted(wb, "since", s->ingest.since, options & CONTEXTS_OPTION_RFC3339);
330 buffer_json_member_add_time_t(wb, "age", s->now - s->ingest.since);
331 buffer_json_member_add_uint64(wb, "metrics", s->ingest.collected.metrics);
332 buffer_json_member_add_uint64(wb, "instances", s->ingest.collected.instances);
@@ -260,7 +365,7 @@ static void rrdhost_receiver_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *
365 buffer_json_object_close(wb); // collection
366 }
367
263 -static void rrdhost_sender_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *key) {
368 +static void rrdhost_sender_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *key, CONTEXTS_OPTIONS options) {
369 if(s->stream.status == RRDHOST_STREAM_STATUS_DISABLED)
370 return;
371
@@ -269,7 +374,7 @@ static void rrdhost_sender_to_json(BUFFER *wb, RRDHOST_STATUS *s, const char *ke
374 buffer_json_member_add_uint64(wb, "id", s->stream.id);
375 buffer_json_member_add_uint64(wb, "hops", s->stream.hops);
376 buffer_json_member_add_string(wb, "status", rrdhost_streaming_status_to_string(s->stream.status));
272 - buffer_json_member_add_time_t(wb, "since", s->stream.since);
377 + buffer_json_member_add_time_t_formatted(wb, "since", s->stream.since, options & CONTEXTS_OPTION_RFC3339);
378 buffer_json_member_add_time_t(wb, "age", s->now - s->stream.since);
379
380 if (s->stream.status == RRDHOST_STREAM_STATUS_OFFLINE)
@@ -362,7 +467,11 @@ static inline void rrdhost_health_to_json_v2(BUFFER *wb, const char *key, RRDHOS
467
468 static void rrdcontext_to_json_v2_rrdhost(BUFFER *wb, RRDHOST *host, struct rrdcontext_to_json_v2_data *ctl, size_t node_id) {
469 buffer_json_add_array_item_object(wb); // this node
365 - buffer_json_node_add_v2(wb, host, node_id, 0,
470 +
471 + if(ctl->options & CONTEXTS_OPTION_MCP)
472 + buffer_json_node_add_v2_mcp(wb, host, node_id);
473 + else
474 + buffer_json_node_add_v2(wb, host, node_id, 0,
475 (ctl->mode & CONTEXTS_V2_AGENTS) && !(ctl->mode & CONTEXTS_V2_NODE_INSTANCES));
476
477 if(ctl->mode & (CONTEXTS_V2_NODES_INFO | CONTEXTS_V2_NODES_STREAM_PATH | CONTEXTS_V2_NODE_INSTANCES)) {
@@ -403,8 +512,8 @@ static void rrdcontext_to_json_v2_rrdhost(BUFFER *wb, RRDHOST *host, struct rrdc
512 buffer_json_member_add_string(wb, "status", rrdhost_db_status_to_string(s.db.status));
513 buffer_json_member_add_string(wb, "liveness", rrdhost_db_liveness_to_string(s.db.liveness));
514 buffer_json_member_add_string(wb, "mode", rrd_memory_mode_name(s.db.mode));
406 - buffer_json_member_add_time_t(wb, "first_time", s.db.first_time_s);
407 - buffer_json_member_add_time_t(wb, "last_time", s.db.last_time_s);
515 + buffer_json_member_add_time_t_formatted(wb, "first_time", s.db.first_time_s, ctl->options & CONTEXTS_OPTION_RFC3339);
516 + buffer_json_member_add_time_t_formatted(wb, "last_time", s.db.last_time_s, ctl->options & CONTEXTS_OPTION_RFC3339);
517
518 buffer_json_member_add_uint64(wb, "metrics", s.db.metrics);
519 buffer_json_member_add_uint64(wb, "instances", s.db.instances);
@@ -412,8 +521,8 @@ static void rrdcontext_to_json_v2_rrdhost(BUFFER *wb, RRDHOST *host, struct rrdc
521 }
522 buffer_json_object_close(wb);
523
415 - rrdhost_receiver_to_json(wb, &s, "ingest");
416 - rrdhost_sender_to_json(wb, &s, "stream");
524 + rrdhost_receiver_to_json(wb, &s, "ingest", ctl->options);
525 + rrdhost_sender_to_json(wb, &s, "stream", ctl->options);
526
527 buffer_json_member_add_object(wb, "ml");
528 buffer_json_member_add_string(wb, "status", rrdhost_ml_status_to_string(s.ml.status));
@@ -465,37 +574,15 @@ static ssize_t rrdcontext_to_json_v2_add_host(void *data, RRDHOST *host, bool qu
574 // interrupted
575 return -1; // stop the query
576
468 - bool host_matched = (ctl->mode & CONTEXTS_V2_NODES);
469 - bool do_contexts = (ctl->mode & (CONTEXTS_V2_CONTEXTS | CONTEXTS_V2_ALERTS));
470 -
471 - ctl->q.host_match = FTS_MATCHED_NONE;
472 - if((ctl->mode & CONTEXTS_V2_SEARCH)) {
473 - // check if we match the host itself
474 - if(ctl->q.pattern && (
475 - full_text_search_string(&ctl->q.fts, ctl->q.pattern, host->hostname) ||
476 - full_text_search_char(&ctl->q.fts, ctl->q.pattern, host->machine_guid) ||
477 - (ctl->q.pattern && full_text_search_char(&ctl->q.fts, ctl->q.pattern, ctl->q.host_node_id_str)))) {
478 - ctl->q.host_match = FTS_MATCHED_HOST;
479 - do_contexts = true;
480 - }
481 - }
577 + bool host_matched = (ctl->mode & (CONTEXTS_V2_NODES | CONTEXTS_V2_FUNCTIONS | CONTEXTS_V2_ALERTS));
578 + bool do_contexts = (ctl->mode & (CONTEXTS_V2_CONTEXTS | CONTEXTS_V2_SEARCH | CONTEXTS_V2_ALERTS));
579
580 if(do_contexts) {
484 - // save it
485 - SIMPLE_PATTERN *old_q = ctl->q.pattern;
486 -
487 - if(ctl->q.host_match == FTS_MATCHED_HOST)
488 - // do not do pattern matching on contexts - we matched the host itself
489 - ctl->q.pattern = NULL;
490 -
581 ssize_t added = query_scope_foreach_context(
582 host, ctl->request->scope_contexts,
583 ctl->contexts.scope_pattern, ctl->contexts.pattern,
584 rrdcontext_to_json_v2_add_context, queryable_host, ctl);
585
496 - // restore it
497 - ctl->q.pattern = old_q;
498 -
586 if(unlikely(added < 0))
587 return -1; // stop the query
588
@@ -520,7 +607,7 @@ static ssize_t rrdcontext_to_json_v2_add_host(void *data, RRDHOST *host, bool qu
607 host_functions_to_dict(host, ctl->functions.dict, &t, sizeof(t), &t.help, &t.tags, &t.access, &t.priority, &t.version);
608 }
609
523 - if(ctl->mode & CONTEXTS_V2_NODES) {
610 + if(ctl->mode & (CONTEXTS_V2_NODES | CONTEXTS_V2_FUNCTIONS | CONTEXTS_V2_ALERTS)) {
611 struct contexts_v2_node t = {
612 .ni = ctl->nodes.ni++,
613 .host = host,
@@ -627,105 +714,498 @@ static void functions_delete_callback(const DICTIONARY_ITEM *item __maybe_unused
714 freez(t->node_ids);
715 }
716
717 +static void contexts_cleanup(struct context_v2_entry *n) {
718 + string_freez(n->title);
719 + string_freez(n->units);
720 + string_freez(n->family);
721 +
722 + dictionary_destroy(n->instances_dict);
723 + dictionary_destroy(n->dimensions_dict);
724 + rrdlabels_aggregated_destroy(n->labels_aggregated);
725 +
726 + // Clean up n's search results
727 + dictionary_destroy(n->matched_instances);
728 + dictionary_destroy(n->matched_dimensions);
729 + rrdlabels_aggregated_destroy(n->matched_labels);
730 +}
731 +
732 static bool contexts_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value, void *data __maybe_unused) {
733 + struct rrdcontext_to_json_v2_data *ctl = data;
734 struct context_v2_entry *o = old_value;
735 struct context_v2_entry *n = new_value;
736
737 o->count++;
738
636 - if(o->family != n->family) {
637 - if((o->flags & RRD_FLAG_COLLECTED) && !(n->flags & RRD_FLAG_COLLECTED))
638 - // keep old
639 - ;
640 - else if(!(o->flags & RRD_FLAG_COLLECTED) && (n->flags & RRD_FLAG_COLLECTED)) {
641 - // keep new
642 - SWAP(o->family, n->family);
739 + o->flags |= n->flags;
740 + o->nodes += n->nodes;
741 + o->instances += n->instances;
742 +
743 + if (ctl->options & CONTEXTS_OPTION_TITLES) {
744 + if(o->title != n->title) {
745 + if((o->flags & RRD_FLAG_COLLECTED) && !(n->flags & RRD_FLAG_COLLECTED))
746 + // keep old
747 + ;
748 + else if(!(o->flags & RRD_FLAG_COLLECTED) && (n->flags & RRD_FLAG_COLLECTED)) {
749 + // keep new
750 + SWAP(o->title, n->title);
751 + }
752 + else {
753 + // merge
754 + STRING *old_title = o->title;
755 + o->title = string_2way_merge(o->title, n->title);
756 + string_freez(old_title);
757 + // n->title will be freed below
758 + }
759 }
644 - else {
645 - // merge
646 - STRING *old_family = o->family;
647 - o->family = string_2way_merge(o->family, n->family);
648 - string_freez(old_family);
649 - // n->family will be freed below
760 + }
761 +
762 + if (ctl->options & CONTEXTS_OPTION_FAMILY) {
763 + if(o->family != n->family) {
764 + if((o->flags & RRD_FLAG_COLLECTED) && !(n->flags & RRD_FLAG_COLLECTED))
765 + // keep old
766 + ;
767 + else if(!(o->flags & RRD_FLAG_COLLECTED) && (n->flags & RRD_FLAG_COLLECTED)) {
768 + // keep new
769 + SWAP(o->family, n->family);
770 + }
771 + else {
772 + // merge
773 + STRING *old_family = o->family;
774 + o->family = string_2way_merge(o->family, n->family);
775 + string_freez(old_family);
776 + // n->family will be freed below
777 + }
778 }
779 }
780
653 - if(o->units != n->units) {
654 - if((o->flags & RRD_FLAG_COLLECTED) && !(n->flags & RRD_FLAG_COLLECTED))
655 - // keep old
656 - ;
657 - else if(!(o->flags & RRD_FLAG_COLLECTED) && (n->flags & RRD_FLAG_COLLECTED)) {
658 - // keep new
659 - SWAP(o->units, n->units);
781 + if (ctl->options & CONTEXTS_OPTION_UNITS) {
782 + if(o->units != n->units) {
783 + if((o->flags & RRD_FLAG_COLLECTED) && !(n->flags & RRD_FLAG_COLLECTED))
784 + // keep old
785 + ;
786 + else if(!(o->flags & RRD_FLAG_COLLECTED) && (n->flags & RRD_FLAG_COLLECTED)) {
787 + // keep new
788 + SWAP(o->units, n->units);
789 + }
790 + else {
791 + // keep old
792 + ;
793 + }
794 }
661 - else {
662 - // keep old
663 - ;
795 + }
796 +
797 + if (ctl->options & CONTEXTS_OPTION_PRIORITIES) {
798 + if(o->priority != n->priority) {
799 + if((o->flags & RRD_FLAG_COLLECTED) && !(n->flags & RRD_FLAG_COLLECTED))
800 + // keep o
801 + ;
802 + else if(!(o->flags & RRD_FLAG_COLLECTED) && (n->flags & RRD_FLAG_COLLECTED))
803 + // keep n
804 + o->priority = n->priority;
805 + else
806 + // keep the min
807 + o->priority = MIN(o->priority, n->priority);
808 }
809 }
810
667 - if(o->priority != n->priority) {
668 - if((o->flags & RRD_FLAG_COLLECTED) && !(n->flags & RRD_FLAG_COLLECTED))
669 - // keep o
670 - ;
671 - else if(!(o->flags & RRD_FLAG_COLLECTED) && (n->flags & RRD_FLAG_COLLECTED))
672 - // keep n
673 - o->priority = n->priority;
674 - else
675 - // keep the min
676 - o->priority = MIN(o->priority, n->priority);
811 + if (ctl->options & CONTEXTS_OPTION_RETENTION) {
812 + if(o->first_time_s && n->first_time_s)
813 + o->first_time_s = MIN(o->first_time_s, n->first_time_s);
814 + else if(!o->first_time_s)
815 + o->first_time_s = n->first_time_s;
816 +
817 + if(o->last_time_s && n->last_time_s)
818 + o->last_time_s = MAX(o->last_time_s, n->last_time_s);
819 + else if(!o->last_time_s)
820 + o->last_time_s = n->last_time_s;
821 }
822
679 - if(o->first_time_s && n->first_time_s)
680 - o->first_time_s = MIN(o->first_time_s, n->first_time_s);
681 - else if(!o->first_time_s)
682 - o->first_time_s = n->first_time_s;
823 + if (ctl->mode & CONTEXTS_V2_SEARCH) {
824 + // Merge search results
825 + o->matched_types |= n->matched_types;
826
684 - if(o->last_time_s && n->last_time_s)
685 - o->last_time_s = MAX(o->last_time_s, n->last_time_s);
686 - else if(!o->last_time_s)
687 - o->last_time_s = n->last_time_s;
827 + // For search result dictionaries, we need to merge them
828 + if(n->matched_instances && !o->matched_instances) {
829 + SWAP(o->matched_instances, n->matched_instances);
830 + }
831 + else if(n->matched_instances && o->matched_instances) {
832 + // Merge entries from n to o
833 + void *entry;
834 + dfe_start_read(n->matched_instances, entry) {
835 + dictionary_set(o->matched_instances, entry_dfe.name, NULL, 0);
836 + }
837 + dfe_done(entry);
838 + }
839
689 - o->flags |= n->flags;
690 - o->match = MIN(o->match, n->match);
840 + if(n->matched_dimensions && !o->matched_dimensions) {
841 + SWAP(o->matched_dimensions, n->matched_dimensions);
842 + }
843 + else if(n->matched_dimensions && o->matched_dimensions) {
844 + // Merge entries from n to o
845 + void *entry;
846 + dfe_start_read(n->matched_dimensions, entry) {
847 + dictionary_set(o->matched_dimensions, entry_dfe.name, NULL, 0);
848 + }
849 + dfe_done(entry);
850 + }
851
692 - string_freez(n->units);
693 - string_freez(n->family);
852 + if(n->matched_labels && !o->matched_labels) {
853 + SWAP(o->matched_labels, n->matched_labels);
854 + }
855 + else if(n->matched_labels && o->matched_labels) {
856 + // Merge n into o
857 + rrdlabels_aggregated_merge(o->matched_labels, n->matched_labels);
858 + }
859 + }
860 +
861 + contexts_cleanup(n);
862 +
863 + // for the react callback to use
864 + o->rc = n->rc;
865
866 return true;
867 }
868
869 +static void contexts_react_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
870 + struct context_v2_entry *t = value;
871 + struct rrdcontext_to_json_v2_data *ctl = data;
872 +
873 + // Only populate dictionaries if they exist (meaning the options are enabled)
874 + if(!(ctl->mode & CONTEXTS_V2_CONTEXTS) || !(ctl->options & (CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_DIMENSIONS | CONTEXTS_OPTION_LABELS)))
875 + return;
876 +
877 + // Initialize dictionaries for the new features if the corresponding options are set
878 + if(ctl->options & CONTEXTS_OPTION_INSTANCES && !t->instances_dict) {
879 + t->instances_dict = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
880 + }
881 +
882 + if(ctl->options & CONTEXTS_OPTION_DIMENSIONS && !t->dimensions_dict) {
883 + t->dimensions_dict = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
884 + }
885 +
886 + if(ctl->options & CONTEXTS_OPTION_LABELS && !t->labels_aggregated) {
887 + t->labels_aggregated = rrdlabels_aggregated_create();
888 + }
889 +
890 + RRDCONTEXT *rc = t->rc;
891 +
892 + // Collect instances, dimensions, and labels if requested
893 + RRDINSTANCE *ri;
894 + dfe_start_read(rc->rrdinstances, ri) {
895 + if(ctl->window.enabled && !query_matches_retention(ctl->window.after, ctl->window.before, ri->first_time_s, (ri->flags & RRD_FLAG_COLLECTED) ? ctl->now : ri->last_time_s, (time_t)ri->update_every_s))
896 + continue;
897 +
898 + // Add instance name to instances dictionary
899 + if(t->instances_dict) {
900 + dictionary_set(t->instances_dict, string2str(ri->name), NULL, 0);
901 + }
902 +
903 + // Collect dimensions from this instance
904 + if(t->dimensions_dict) {
905 + RRDMETRIC *rm;
906 + dfe_start_read(ri->rrdmetrics, rm) {
907 + if(ctl->window.enabled && !query_matches_retention(ctl->window.after, ctl->window.before, rm->first_time_s, (rm->flags & RRD_FLAG_COLLECTED) ? ctl->now : rm->last_time_s, (time_t)ri->update_every_s))
908 + continue;
909 +
910 + dictionary_set(t->dimensions_dict, string2str(rm->name), NULL, 0);
911 + }
912 + dfe_done(rm);
913 + }
914 +
915 + // Collect labels from this instance
916 + if(t->labels_aggregated) {
917 + RRDLABELS *labels = rrdinstance_labels(ri);
918 + rrdlabels_aggregated_add_from_rrdlabels(t->labels_aggregated, labels);
919 + }
920 + }
921 + dfe_done(ri);
922 +}
923 +
924 static void contexts_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
925 struct context_v2_entry *z = value;
700 - string_freez(z->family);
701 - string_freez((z->units));
926 + contexts_cleanup(z);
927 }
928
704 -int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTEXTS_V2_MODE mode) {
705 - int resp = HTTP_RESP_OK;
706 - bool run = true;
929 +static void contexts_v2_search_results_to_json(BUFFER *wb, struct rrdcontext_to_json_v2_data *ctl) {
930 + size_t contexts_count = 0;
931 + size_t contexts_limit = ctl->request->cardinality_limit;
932 + size_t total_contexts = dictionary_entries(ctl->contexts.dict);
933 +
934 + // Calculate per-context item limit: MIN(cardinality_limit/total_contexts, 3)
935 + // But use the contexts that will be shown, not total
936 + size_t contexts_to_show = (contexts_limit && total_contexts > contexts_limit) ? contexts_limit : total_contexts;
937 + size_t per_context_limit = 3; // Default minimum
938 + if (contexts_limit && contexts_to_show > 0) {
939 + size_t calculated_limit = contexts_limit / contexts_to_show;
940 + if (calculated_limit > per_context_limit)
941 + per_context_limit = calculated_limit;
942 + }
943
708 - if(mode & CONTEXTS_V2_SEARCH)
709 - mode |= CONTEXTS_V2_CONTEXTS;
944 + buffer_json_member_add_object(wb, "contexts");
945 +
946 + struct context_v2_entry *z;
947 + dfe_start_read(ctl->contexts.dict, z) {
948 + // Check if we've reached the limit
949 + if (contexts_limit && contexts_count >= contexts_limit) {
950 + // Add a special entry indicating truncation
951 + buffer_json_member_add_object(wb, "__truncated__");
952 + buffer_json_member_add_uint64(wb, "total_contexts", total_contexts);
953 + buffer_json_member_add_uint64(wb, "returned", contexts_count);
954 + buffer_json_member_add_uint64(wb, "remaining", total_contexts - contexts_count);
955 + buffer_json_object_close(wb);
956 + break;
957 + }
958
711 - if(mode & (CONTEXTS_V2_AGENTS_INFO))
712 - mode |= CONTEXTS_V2_AGENTS;
959 + buffer_json_member_add_object(wb, string2str(z->id));
960 + {
961 + // Always show title, family, units in search results for context
962 + if (z->matched_types & SEARCH_MATCH_CONTEXT_TITLE)
963 + buffer_json_member_add_string(wb, "title", string2str(z->title));
964 +
965 + if (z->matched_types & SEARCH_MATCH_CONTEXT_FAMILY)
966 + buffer_json_member_add_string(wb, "family", string2str(z->family));
967 +
968 + if (z->matched_types & SEARCH_MATCH_CONTEXT_UNITS)
969 + buffer_json_member_add_string(wb, "units", string2str(z->units));
970 +
971 + // Output what matched as an array
972 + if (!(ctl->options & CONTEXTS_OPTION_MCP)) {
973 + buffer_json_member_add_array(wb, "matched");
974 + if (z->matched_types & SEARCH_MATCH_CONTEXT_ID)
975 + buffer_json_add_array_item_string(wb, "id");
976 + if (z->matched_types & SEARCH_MATCH_CONTEXT_TITLE)
977 + buffer_json_add_array_item_string(wb, "title");
978 + if (z->matched_types & SEARCH_MATCH_CONTEXT_UNITS)
979 + buffer_json_add_array_item_string(wb, "units");
980 + if (z->matched_types & SEARCH_MATCH_CONTEXT_FAMILY)
981 + buffer_json_add_array_item_string(wb, "families");
982 + if (z->matched_types & SEARCH_MATCH_INSTANCE)
983 + buffer_json_add_array_item_string(wb, "instances");
984 + if (z->matched_types & SEARCH_MATCH_DIMENSION)
985 + buffer_json_add_array_item_string(wb, "dimensions");
986 + if (z->matched_types & SEARCH_MATCH_LABEL)
987 + buffer_json_add_array_item_string(wb, "labels");
988 + buffer_json_array_close(wb);
989 + }
990
714 - if(mode & (CONTEXTS_V2_FUNCTIONS | CONTEXTS_V2_CONTEXTS | CONTEXTS_V2_SEARCH | CONTEXTS_V2_NODES_INFO | CONTEXTS_V2_NODES_STREAM_PATH | CONTEXTS_V2_NODE_INSTANCES))
715 - mode |= CONTEXTS_V2_NODES;
991 + // Add instances array if any matched
992 + if(z->matched_instances && dictionary_entries(z->matched_instances) > 0) {
993 + buffer_json_member_add_array(wb, "instances");
994 + void *entry;
995 + size_t count = 0;
996 + size_t total = dictionary_entries(z->matched_instances);
997 +
998 + dfe_start_read(z->matched_instances, entry) {
999 + if (per_context_limit && total > per_context_limit && count >= per_context_limit - 1) {
1000 + char msg[100];
1001 + snprintf(msg, sizeof(msg), "... %zu instances more", total - count);
1002 + buffer_json_add_array_item_string(wb, msg);
1003 + break;
1004 + }
1005 + buffer_json_add_array_item_string(wb, entry_dfe.name);
1006 + count++;
1007 + }
1008 + dfe_done(entry);
1009 + buffer_json_array_close(wb);
1010 + }
1011 +
1012 + // Add dimensions array if any matched
1013 + if(z->matched_dimensions && dictionary_entries(z->matched_dimensions) > 0) {
1014 + buffer_json_member_add_array(wb, "dimensions");
1015 + void *entry;
1016 + size_t count = 0;
1017 + size_t total = dictionary_entries(z->matched_dimensions);
1018 +
1019 + dfe_start_read(z->matched_dimensions, entry) {
1020 + if (per_context_limit && total > per_context_limit && count >= per_context_limit - 1) {
1021 + char msg[100];
1022 + snprintf(msg, sizeof(msg), "... %zu dimensions more", total - count);
1023 + buffer_json_add_array_item_string(wb, msg);
1024 + break;
1025 + }
1026 + buffer_json_add_array_item_string(wb, entry_dfe.name);
1027 + count++;
1028 + }
1029 + dfe_done(entry);
1030 + buffer_json_array_close(wb);
1031 + }
1032 +
1033 + // Add labels if any matched
1034 + if(z->matched_labels) {
1035 + rrdlabels_aggregated_to_buffer_json(
1036 + z->matched_labels, wb, "labels", per_context_limit);
1037 + }
1038 + }
1039 + buffer_json_object_close(wb);
1040 +
1041 + contexts_count++;
1042 + }
1043 + dfe_done(z);
1044
717 - if(mode & CONTEXTS_V2_ALERTS) {
718 - mode |= CONTEXTS_V2_NODES;
719 - req->options &= ~CONTEXTS_OPTION_ALERTS_WITH_CONFIGURATIONS;
1045 + buffer_json_object_close(wb); // contexts
1046 +
1047 + // Add info about cardinality limit if it was reached
1048 + if (contexts_limit && total_contexts > contexts_limit && (ctl->options & CONTEXTS_OPTION_MCP)) {
1049 + buffer_json_member_add_string(wb, "info", "Cardinality limit reached. Use cardinality_limit parameter to see more results.");
1050 + }
1051 +}
1052 +
1053 +static void contexts_v2_contexts_to_json(BUFFER *wb, struct rrdcontext_to_json_v2_data *ctl) {
1054 + size_t contexts_count = 0;
1055 + size_t contexts_limit = ctl->request->cardinality_limit;
1056 + size_t total_contexts = dictionary_entries(ctl->contexts.dict);
1057 +
1058 + bool contexts_is_object = false;
1059 +
1060 + // If we have more contexts than the limit and MCP option is set, use categorized output
1061 + if (contexts_limit && total_contexts > contexts_limit && (ctl->options & CONTEXTS_OPTION_MCP)) {
1062 + buffer_json_member_add_object(wb, "contexts");
1063 + rrdcontext_categorize_and_output(wb, ctl->contexts.dict, contexts_limit);
1064 + buffer_json_object_close(wb);
1065 + if (ctl->options & CONTEXTS_OPTION_MCP) {
1066 + buffer_json_member_add_string(wb, "info", MCP_INFO_TOO_MANY_CONTEXTS_GROUPED_IN_CATEGORIES);
1067 + }
1068 + } else {
1069 + if (ctl->options & (CONTEXTS_OPTION_TITLES | CONTEXTS_OPTION_FAMILY | CONTEXTS_OPTION_UNITS |
1070 + CONTEXTS_OPTION_PRIORITIES | CONTEXTS_OPTION_RETENTION | CONTEXTS_OPTION_LIVENESS |
1071 + CONTEXTS_OPTION_DIMENSIONS | CONTEXTS_OPTION_LABELS | CONTEXTS_OPTION_INSTANCES)) {
1072 + contexts_is_object = true;
1073 + buffer_json_member_add_object(wb, "contexts");
1074 + } else
1075 + buffer_json_member_add_array(wb, "contexts");
1076 +
1077 + struct context_v2_entry *z;
1078 + dfe_start_read(ctl->contexts.dict, z) {
1079 + // Check if we've reached the limit
1080 + if (contexts_limit && contexts_count >= contexts_limit) {
1081 + // Add a special entry indicating truncation
1082 + if (contexts_is_object) {
1083 + buffer_json_member_add_object(wb, "__truncated__");
1084 + buffer_json_member_add_uint64(wb, "total_contexts", total_contexts);
1085 + buffer_json_member_add_uint64(wb, "returned", contexts_count);
1086 + buffer_json_member_add_uint64(wb, "remaining", total_contexts - contexts_count);
1087 + buffer_json_object_close(wb);
1088 + } else {
1089 + char msg[100];
1090 + snprintf(msg, sizeof(msg), "... %zu contexts more", total_contexts - contexts_count);
1091 + buffer_json_add_array_item_string(wb, msg);
1092 + }
1093 + break;
1094 + }
1095 +
1096 + bool collected = z->flags & RRD_FLAG_COLLECTED;
1097 +
1098 + if (contexts_is_object) {
1099 + buffer_json_member_add_object(wb, string2str(z->id));
1100 + {
1101 + if (ctl->options & CONTEXTS_OPTION_TITLES)
1102 + buffer_json_member_add_string(wb, "title", string2str(z->title));
1103 +
1104 + if (ctl->options & CONTEXTS_OPTION_FAMILY)
1105 + buffer_json_member_add_string(wb, "family", string2str(z->family));
1106 +
1107 + if (ctl->options & CONTEXTS_OPTION_UNITS)
1108 + buffer_json_member_add_string(wb, "units", string2str(z->units));
1109 +
1110 + if (ctl->options & CONTEXTS_OPTION_PRIORITIES)
1111 + buffer_json_member_add_uint64(wb, "priority", z->priority);
1112
721 - if(!(req->options & (CONTEXTS_OPTION_ALERTS_WITH_SUMMARY | CONTEXTS_OPTION_ALERTS_WITH_INSTANCES |
722 - CONTEXTS_OPTION_ALERTS_WITH_VALUES)))
723 - req->options |= CONTEXTS_OPTION_ALERTS_WITH_SUMMARY;
1113 + if (ctl->options & CONTEXTS_OPTION_RETENTION) {
1114 + buffer_json_member_add_time_t_formatted(wb, "first_entry", z->first_time_s, ctl->options & CONTEXTS_OPTION_RFC3339);
1115 + buffer_json_member_add_time_t_formatted(wb, "last_entry", collected ? ctl->now : z->last_time_s, ctl->options & CONTEXTS_OPTION_RFC3339);
1116 + }
1117 +
1118 + if (ctl->options & CONTEXTS_OPTION_LIVENESS)
1119 + buffer_json_member_add_boolean(wb, "live", collected);
1120 +
1121 + // Add dimensions sub-object if requested
1122 + if ((ctl->options & CONTEXTS_OPTION_DIMENSIONS) && z->dimensions_dict) {
1123 + buffer_json_member_add_array(wb, "dimensions");
1124 + void *entry;
1125 + size_t count = 0;
1126 + size_t total = dictionary_entries(z->dimensions_dict);
1127 + size_t limit = ctl->request->cardinality_limit;
1128 +
1129 + dfe_start_read(z->dimensions_dict, entry) {
1130 + if (limit && count >= limit - 1 && total > limit) {
1131 + // Add remaining count message
1132 + char msg[100];
1133 + snprintf(msg, sizeof(msg), "... %zu dimensions more", total - count);
1134 + buffer_json_add_array_item_string(wb, msg);
1135 + break;
1136 + }
1137 + buffer_json_add_array_item_string(wb, entry_dfe.name);
1138 + count++;
1139 + }
1140 + dfe_done(entry);
1141 + buffer_json_array_close(wb);
1142 + }
1143 +
1144 + // Add labels sub-object if requested
1145 + if ((ctl->options & CONTEXTS_OPTION_LABELS) && z->labels_aggregated) {
1146 + rrdlabels_aggregated_to_buffer_json(
1147 + z->labels_aggregated, wb, "labels", ctl->request->cardinality_limit);
1148 + }
1149 +
1150 + // Add instances sub-object if requested
1151 + if ((ctl->options & CONTEXTS_OPTION_INSTANCES) && z->instances_dict) {
1152 + buffer_json_member_add_array(wb, "instances");
1153 + void *entry;
1154 + size_t count = 0;
1155 + size_t total = dictionary_entries(z->instances_dict);
1156 + size_t limit = ctl->request->cardinality_limit;
1157 +
1158 + dfe_start_read(z->instances_dict, entry) {
1159 + if (limit && count >= limit - 1 && total > limit) {
1160 + // Add remaining count message
1161 + char msg[100];
1162 + snprintf(msg, sizeof(msg), "... %zu instances more", total - count);
1163 + buffer_json_add_array_item_string(wb, msg);
1164 + break;
1165 + }
1166 + buffer_json_add_array_item_string(wb, entry_dfe.name);
1167 + count++;
1168 + }
1169 + dfe_done(entry);
1170 + buffer_json_array_close(wb);
1171 + }
1172 + }
1173 + buffer_json_object_close(wb);
1174 + } else {
1175 + buffer_json_add_array_item_string(wb, string2str(z->id));
1176 + }
1177 +
1178 + contexts_count++;
1179 + }
1180 + dfe_done(z);
1181 +
1182 + if(contexts_is_object) {
1183 + buffer_json_object_close(wb); // contexts
1184 +
1185 + if (ctl->options & CONTEXTS_OPTION_MCP) {
1186 + buffer_json_member_add_string(wb, "info", MCP_INFO_CONTEXT_NEXT_STEPS);
1187 + }
1188 + }
1189 + else {
1190 + buffer_json_array_close(wb);
1191 +
1192 + if (ctl->options & CONTEXTS_OPTION_MCP) {
1193 + buffer_json_member_add_string(wb, "info", MCP_INFO_CONTEXT_ARRAY_RESPONSE);
1194 + }
1195 + }
1196 + }
1197 +}
1198 +
1199 +int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTEXTS_V2_MODE mode) {
1200 + int resp = HTTP_RESP_OK;
1201 + bool run = true;
1202 +
1203 + if(mode & CONTEXTS_V2_ALERTS) {
1204 + req->options &= ~CONTEXTS_OPTION_CONFIGURATIONS;
1205 }
1206
1207 if(mode & CONTEXTS_V2_ALERT_TRANSITIONS) {
727 - mode |= CONTEXTS_V2_NODES;
728 - req->options &= ~CONTEXTS_OPTION_ALERTS_WITH_INSTANCES;
1208 + req->options &= ~CONTEXTS_OPTION_INSTANCES;
1209 }
1210
1211 struct rrdcontext_to_json_v2_data ctl = {
@@ -738,7 +1218,7 @@ int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTE
1218 .nodes.pattern = string_to_simple_pattern(req->nodes),
1219 .contexts.pattern = string_to_simple_pattern(req->contexts),
1220 .contexts.scope_pattern = string_to_simple_pattern(req->scope_contexts),
741 - .q.pattern = string_to_simple_pattern_nocase(req->q),
1221 + .q.pattern = string_to_simple_pattern_nocase_substring(req->q),
1222 .alerts.alert_name_pattern = string_to_simple_pattern(req->alerts.alert),
1223 .window = {
1224 .enabled = false,
@@ -752,18 +1232,22 @@ int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTE
1232 };
1233
1234 bool debug = ctl.options & CONTEXTS_OPTION_DEBUG;
1235 +
1236 + // Initialize JSON keys based on options
1237 + json_keys_init((ctl.options & CONTEXTS_OPTION_JSON_LONG_KEYS) ? JSON_KEYS_OPTION_LONG_KEYS : 0);
1238
756 - if(mode & CONTEXTS_V2_NODES) {
1239 + if(mode & (CONTEXTS_V2_NODES | CONTEXTS_V2_FUNCTIONS | CONTEXTS_V2_ALERTS)) {
1240 ctl.nodes.dict = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
1241 NULL, sizeof(struct contexts_v2_node));
1242 }
1243
761 - if(mode & CONTEXTS_V2_CONTEXTS) {
1244 + if(mode & (CONTEXTS_V2_CONTEXTS | CONTEXTS_V2_SEARCH)) {
1245 ctl.contexts.dict = dictionary_create_advanced(
1246 DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE, NULL,
1247 sizeof(struct context_v2_entry));
1248
1249 dictionary_register_conflict_callback(ctl.contexts.dict, contexts_conflict_callback, &ctl);
1250 + dictionary_register_react_callback(ctl.contexts.dict, contexts_react_callback, &ctl);
1251 dictionary_register_delete_callback(ctl.contexts.dict, contexts_delete_callback, &ctl);
1252 }
1253
@@ -796,7 +1280,8 @@ int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTE
1280 buffer_json_initialize(wb, "\"", "\"", 0, true,
1281 ((req->options & CONTEXTS_OPTION_MINIFY) && !(req->options & CONTEXTS_OPTION_DEBUG)) ? BUFFER_JSON_OPTIONS_MINIFY : BUFFER_JSON_OPTIONS_DEFAULT);
1282
799 - buffer_json_member_add_uint64(wb, "api", 2);
1283 + if(!(req->options & CONTEXTS_OPTION_MCP))
1284 + buffer_json_member_add_uint64(wb, "api", 2);
1285
1286 if(req->options & CONTEXTS_OPTION_DEBUG) {
1287 buffer_json_member_add_object(wb, "request");
@@ -843,8 +1328,8 @@ int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTE
1328 if (mode & CONTEXTS_V2_SEARCH)
1329 buffer_json_member_add_string(wb, "q", req->q);
1330
846 - buffer_json_member_add_time_t(wb, "after", req->after);
847 - buffer_json_member_add_time_t(wb, "before", req->before);
1331 + buffer_json_member_add_time_t_formatted(wb, "after", req->after, ctl.options & CONTEXTS_OPTION_RFC3339);
1332 + buffer_json_member_add_time_t_formatted(wb, "before", req->before, ctl.options & CONTEXTS_OPTION_RFC3339);
1333 }
1334 buffer_json_object_close(wb); // filters
1335
@@ -912,16 +1397,20 @@ int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTE
1397
1398 buffer_json_member_add_string(wb, "name", name);
1399 buffer_json_member_add_string(wb, "help", string2str(t->help));
915 - buffer_json_member_add_array(wb, "ni");
916 - {
917 - for (size_t i = 0; i < t->used; i++)
918 - buffer_json_add_array_item_uint64(wb, t->node_ids[i]);
1400 +
1401 + if (!(ctl.options & CONTEXTS_OPTION_MCP)) {
1402 + buffer_json_member_add_array(wb, "ni");
1403 + {
1404 + for (size_t i = 0; i < t->used; i++)
1405 + buffer_json_add_array_item_uint64(wb, t->node_ids[i]);
1406 + }
1407 + buffer_json_array_close(wb);
1408 +
1409 + buffer_json_member_add_uint64(wb, "priority", t->priority);
1410 + buffer_json_member_add_uint64(wb, "version", t->version);
1411 }
920 - buffer_json_array_close(wb);
1412 buffer_json_member_add_string(wb, "tags", string2str(t->tags));
1413 http_access2buffer_json_array(wb, "access", t->access);
923 - buffer_json_member_add_uint64(wb, "priority", t->priority);
924 - buffer_json_member_add_uint64(wb, "version", t->version);
1414 }
1415 buffer_json_object_close(wb);
1416 }
@@ -930,29 +1419,11 @@ int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTE
1419 buffer_json_array_close(wb);
1420 }
1421
933 - if (mode & CONTEXTS_V2_CONTEXTS) {
934 - buffer_json_member_add_object(wb, "contexts");
935 - {
936 - struct context_v2_entry *z;
937 - dfe_start_read(ctl.contexts.dict, z) {
938 - bool collected = z->flags & RRD_FLAG_COLLECTED;
939 -
940 - buffer_json_member_add_object(wb, string2str(z->id));
941 - {
942 - buffer_json_member_add_string(wb, "family", string2str(z->family));
943 - buffer_json_member_add_string(wb, "units", string2str(z->units));
944 - buffer_json_member_add_uint64(wb, "priority", z->priority);
945 - buffer_json_member_add_time_t(wb, "first_entry", z->first_time_s);
946 - buffer_json_member_add_time_t(wb, "last_entry", collected ? ctl.now : z->last_time_s);
947 - buffer_json_member_add_boolean(wb, "live", collected);
948 - if (mode & CONTEXTS_V2_SEARCH)
949 - buffer_json_member_add_string(wb, "match", fts_match_to_string(z->match));
950 - }
951 - buffer_json_object_close(wb);
952 - }
953 - dfe_done(z);
954 - }
955 - buffer_json_object_close(wb); // contexts
1422 + if (mode & CONTEXTS_V2_SEARCH) {
1423 + contexts_v2_search_results_to_json(wb, &ctl);
1424 + }
1425 + else if (mode & CONTEXTS_V2_CONTEXTS) {
1426 + contexts_v2_contexts_to_json(wb, &ctl);
1427 }
1428
1429 if (mode & CONTEXTS_V2_ALERTS)
@@ -968,14 +1439,15 @@ int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTE
1439 buffer_json_object_close(wb);
1440 }
1441
971 - if (mode & (CONTEXTS_V2_VERSIONS))
1442 + if (mode & CONTEXTS_V2_VERSIONS)
1443 version_hashes_api_v2(wb, &ctl.versions);
1444
1445 if (mode & CONTEXTS_V2_AGENTS)
975 - buffer_json_agents_v2(wb, &ctl.timings, ctl.now, mode & (CONTEXTS_V2_AGENTS_INFO), true);
1446 + buffer_json_agents_v2(wb, &ctl.timings, ctl.now, mode & (CONTEXTS_V2_AGENTS_INFO), true, ctl.options);
1447 }
1448
978 - buffer_json_cloud_timings(wb, "timings", &ctl.timings);
1449 + if(!(ctl.options & CONTEXTS_OPTION_MCP))
1450 + buffer_json_cloud_timings(wb, "timings", &ctl.timings);
1451
1452 buffer_json_finalize(wb);
1453
@@ -991,5 +1463,6 @@ cleanup:
1463 simple_pattern_free(ctl.q.pattern);
1464 simple_pattern_free(ctl.alerts.alert_name_pattern);
1465
1466 + json_keys_reset();
1467 return resp;
1468 }
src/database/contexts/api_v2_contexts.h
-2
@@ -7,7 +7,6 @@
7
8 typedef enum __attribute__ ((__packed__)) {
9 FTS_MATCHED_NONE = 0,
10 - FTS_MATCHED_HOST,
10 FTS_MATCHED_CONTEXT,
11 FTS_MATCHED_INSTANCE,
12 FTS_MATCHED_DIMENSION,
@@ -71,7 +70,6 @@ struct rrdcontext_to_json_v2_data {
70 } alerts;
71
72 struct {
74 - FTS_MATCH host_match;
73 char host_node_id_str[UUID_STR_LEN];
74 SIMPLE_PATTERN *pattern;
75 FTS_INDEX fts;
src/database/contexts/api_v2_contexts_agents.c
+4 -4
@@ -7,7 +7,7 @@
7
8 void build_info_to_json_object(BUFFER *b);
9
10 -void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now_s, bool info, bool array) {
10 +void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now_s, bool info, bool array, CONTEXTS_OPTIONS options) {
11 if(!now_s)
12 now_s = now_realtime_sec();
13
@@ -21,7 +21,7 @@ void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now
21 buffer_json_member_add_string(wb, "mg", localhost->machine_guid);
22 buffer_json_member_add_uuid(wb, "nd", localhost->node_id.uuid);
23 buffer_json_member_add_string(wb, "nm", rrdhost_hostname(localhost));
24 - buffer_json_member_add_time_t(wb, "now", now_s);
24 + buffer_json_member_add_time_t_formatted(wb, "now", now_s, options & CONTEXTS_OPTION_RFC3339);
25
26 if(array)
27 buffer_json_member_add_uint64(wb, "ai", 0);
@@ -100,8 +100,8 @@ void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now
100 }
101
102 if(tier_info->first_time_s < tier_info->last_time_s) {
103 - buffer_json_member_add_time_t(wb, "from", tier_info->first_time_s);
104 - buffer_json_member_add_time_t(wb, "to", tier_info->last_time_s);
103 + buffer_json_member_add_time_t_formatted(wb, "from", tier_info->first_time_s, options & CONTEXTS_OPTION_RFC3339);
104 + buffer_json_member_add_time_t_formatted(wb, "to", tier_info->last_time_s, options & CONTEXTS_OPTION_RFC3339);
105 buffer_json_member_add_time_t(wb, "retention", tier_info->retention);
106 buffer_json_member_add_string(wb, "retention_human", tier_info->retention_human);
107
src/database/contexts/api_v2_contexts_alert_config.c
+9 -5
@@ -8,11 +8,15 @@ void contexts_v2_alert_config_to_json_from_sql_alert_config_data(struct sql_aler
8 bool debug = d->debug;
9 d->configs_added++;
10
11 + bool is_mcp = d->ctl && (d->ctl->options & CONTEXTS_OPTION_MCP);
12 +
13 if(d->only_one_config)
14 buffer_json_add_array_item_object(wb); // alert config
15
16 {
15 - buffer_json_member_add_string(wb, "name", t->name);
17 + if(!is_mcp)
18 + buffer_json_member_add_string(wb, "name", t->name);
19 +
20 buffer_json_member_add_uuid_ptr(wb, "config_hash_id", t->config_hash_id);
21
22 buffer_json_member_add_object(wb, "selectors");
@@ -38,8 +42,8 @@ void contexts_v2_alert_config_to_json_from_sql_alert_config_data(struct sql_aler
42 {
43 // buffer_json_member_add_string(wb, "lookup", t->value.db.lookup); // does not exist in Netdata Cloud
44
41 - buffer_json_member_add_time_t(wb, "after", t->value.db.after);
42 - buffer_json_member_add_time_t(wb, "before", t->value.db.before);
45 + buffer_json_member_add_time_t_formatted(wb, "after", t->value.db.after, d->ctl && (d->ctl->options & CONTEXTS_OPTION_RFC3339));
46 + buffer_json_member_add_time_t_formatted(wb, "before", t->value.db.before, d->ctl && (d->ctl->options & CONTEXTS_OPTION_RFC3339));
47 buffer_json_member_add_string(wb, "time_group_condition", alerts_group_conditions_id2txt(t->value.db.time_group_condition));
48 buffer_json_member_add_double(wb, "time_group_value", t->value.db.time_group_value);
49 buffer_json_member_add_string(wb, "dims_group", alerts_dims_grouping_id2group(t->value.db.dims_group));
@@ -62,10 +66,10 @@ void contexts_v2_alert_config_to_json_from_sql_alert_config_data(struct sql_aler
66 NETDATA_DOUBLE green = t->status.green ? str2ndd(t->status.green, NULL) : NAN;
67 NETDATA_DOUBLE red = t->status.red ? str2ndd(t->status.red, NULL) : NAN;
68
65 - if (!isnan(green) || debug)
69 + if (!is_mcp && (!isnan(green) || debug))
70 buffer_json_member_add_double(wb, "green", green);
71
68 - if (!isnan(red) || debug)
72 + if (!is_mcp && (!isnan(red) || debug))
73 buffer_json_member_add_double(wb, "red", red);
74
75 if (t->status.warn || debug)
src/database/contexts/api_v2_contexts_alert_transitions.c
+53 -38
@@ -1,6 +1,7 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 #include "api_v2_contexts_alerts.h"
4 +#include "libnetdata/json/json-keys.h"
5
6 struct alert_transitions_facets alert_transition_facets[] = {
7 [ATF_STATUS] = {
@@ -311,7 +312,7 @@ void contexts_v2_alert_transitions_to_json(BUFFER *wb, struct rrdcontext_to_json
312 for(size_t i = 0; i < ATF_TOTAL_ENTRIES ;i++) {
313 data.facets[i].dict = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_FIXED_SIZE | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, sizeof(struct facet_entry));
314 if(ctl->request->alerts.facets[i])
314 - data.facets[i].pattern = simple_pattern_create(ctl->request->alerts.facets[i], ",|", SIMPLE_PATTERN_EXACT, false);
315 + data.facets[i].pattern = simple_pattern_create(ctl->request->alerts.facets[i], SIMPLE_PATTERN_DEFAULT_WEB_SEPARATORS, SIMPLE_PATTERN_EXACT, false);
316 }
317
318 sql_alert_transitions(
@@ -325,39 +326,42 @@ void contexts_v2_alert_transitions_to_json(BUFFER *wb, struct rrdcontext_to_json
326 &data,
327 debug);
328
328 - buffer_json_member_add_array(wb, "facets");
329 - for (size_t i = 0; i < ATF_TOTAL_ENTRIES; i++) {
330 - buffer_json_add_array_item_object(wb);
331 - {
332 - buffer_json_member_add_string(wb, "id", alert_transition_facets[i].id);
333 - buffer_json_member_add_string(wb, "name", alert_transition_facets[i].name);
334 - buffer_json_member_add_uint64(wb, "order", alert_transition_facets[i].order);
335 - buffer_json_member_add_array(wb, "options");
329 + if(!(ctl->request->options & CONTEXTS_OPTION_MCP)) {
330 + buffer_json_member_add_array(wb, "facets");
331 + for (size_t i = 0; i < ATF_TOTAL_ENTRIES; i++) {
332 + buffer_json_add_array_item_object(wb);
333 {
337 - struct facet_entry *x;
338 - dfe_start_read(data.facets[i].dict, x) {
339 - buffer_json_add_array_item_object(wb);
334 + buffer_json_member_add_string(wb, "id", alert_transition_facets[i].id);
335 + buffer_json_member_add_string(wb, "name", alert_transition_facets[i].name);
336 + buffer_json_member_add_uint64(wb, "order", alert_transition_facets[i].order);
337 + buffer_json_member_add_array(wb, "options");
338 + {
339 + struct facet_entry *x;
340 + dfe_start_read(data.facets[i].dict, x)
341 {
341 - buffer_json_member_add_string(wb, "id", x_dfe.name);
342 - if (i == ATF_NODE) {
343 - RRDHOST *host = rrdhost_find_by_guid(x_dfe.name);
344 - if (host)
345 - buffer_json_member_add_string(wb, "name", rrdhost_hostname(host));
346 - else
342 + buffer_json_add_array_item_object(wb);
343 + {
344 + buffer_json_member_add_string(wb, "id", x_dfe.name);
345 + if (i == ATF_NODE) {
346 + RRDHOST *host = rrdhost_find_by_guid(x_dfe.name);
347 + if (host)
348 + buffer_json_member_add_string(wb, "name", rrdhost_hostname(host));
349 + else
350 + buffer_json_member_add_string(wb, "name", x_dfe.name);
351 + } else
352 buffer_json_member_add_string(wb, "name", x_dfe.name);
348 - } else
349 - buffer_json_member_add_string(wb, "name", x_dfe.name);
350 - buffer_json_member_add_uint64(wb, "count", x->count);
353 + buffer_json_member_add_uint64(wb, "count", x->count);
354 + }
355 + buffer_json_object_close(wb);
356 }
352 - buffer_json_object_close(wb);
357 + dfe_done(x);
358 }
354 - dfe_done(x);
359 + buffer_json_array_close(wb); // options
360 }
356 - buffer_json_array_close(wb); // options
361 + buffer_json_object_close(wb); // facet
362 }
358 - buffer_json_object_close(wb); // facet
363 + buffer_json_array_close(wb); // facets
364 }
360 - buffer_json_array_close(wb); // facets
365
366 buffer_json_member_add_array(wb, "transitions");
367 for(struct sql_alert_transition_fixed_size *t = data.base; t ; t = t->next) {
@@ -365,37 +369,48 @@ void contexts_v2_alert_transitions_to_json(BUFFER *wb, struct rrdcontext_to_json
369 {
370 RRDHOST *host = rrdhost_find_by_guid(t->machine_guid);
371
368 - buffer_json_member_add_uint64(wb, "gi", t->global_id);
369 - buffer_json_member_add_uuid(wb, "transition_id", t->transition_id);
372 + buffer_json_member_add_uint64(wb, JSKEY(alert_global_id), t->global_id);
373 + buffer_json_member_add_string(wb, "alert", *t->alert_name ? t->alert_name : NULL);
374 +
375 + if(!(ctl->options & CONTEXTS_OPTION_MCP)) {
376 + buffer_json_member_add_uuid(wb, "transition_id", t->transition_id);
377 + buffer_json_member_add_string(wb, "machine_guid", t->machine_guid);
378 +
379 + if(host && !UUIDiszero(host->node_id))
380 + buffer_json_member_add_uuid(wb, "node_id", host->node_id.uuid);
381 + }
382 +
383 buffer_json_member_add_uuid(wb, "config_hash_id", t->config_hash_id);
371 - buffer_json_member_add_string(wb, "machine_guid", t->machine_guid);
384
385 if(host) {
386 buffer_json_member_add_string(wb, "hostname", rrdhost_hostname(host));
387 + }
388
376 - if(!UUIDiszero(host->node_id))
377 - buffer_json_member_add_uuid(wb, "node_id", host->node_id.uuid);
389 + if(!(ctl->options & CONTEXTS_OPTION_MCP)) {
390 + buffer_json_member_add_string(wb, "instance", *t->chart ? t->chart : NULL);
391 + buffer_json_member_add_string(wb, "instance_n", *t->chart_name ? t->chart_name : NULL);
392 }
393 + else
394 + buffer_json_member_add_string(wb, "instance", *t->chart_name ? t->chart_name : NULL);
395
380 - buffer_json_member_add_string(wb, "alert", *t->alert_name ? t->alert_name : NULL);
381 - buffer_json_member_add_string(wb, "instance", *t->chart ? t->chart : NULL);
382 - buffer_json_member_add_string(wb, "instance_n", *t->chart_name ? t->chart_name : NULL);
396 buffer_json_member_add_string(wb, "context", *t->chart_context ? t->chart_context : NULL);
397 // buffer_json_member_add_string(wb, "family", *t->family ? t->family : NULL);
398 buffer_json_member_add_string(wb, "component", *t->component ? t->component : NULL);
399 buffer_json_member_add_string(wb, "classification", *t->classification ? t->classification : NULL);
400 buffer_json_member_add_string(wb, "type", *t->type ? t->type : NULL);
401
389 - buffer_json_member_add_time_t(wb, "when", t->when_key);
402 + buffer_json_member_add_time_t_formatted(wb, "when", t->when_key, ctl->options & CONTEXTS_OPTION_RFC3339);
403 buffer_json_member_add_string(wb, "info", *t->info ? t->info : "");
404 buffer_json_member_add_string(wb, "summary", *t->summary ? t->summary : "");
405 buffer_json_member_add_string(wb, "units", *t->units ? t->units : NULL);
406 +
407 buffer_json_member_add_object(wb, "new");
408 {
409 buffer_json_member_add_string(wb, "status", rrdcalc_status2string(t->new_status));
410 buffer_json_member_add_double(wb, "value", t->new_value);
411 }
412 buffer_json_object_close(wb); // new
413 +
414 buffer_json_member_add_object(wb, "old");
415 {
416 buffer_json_member_add_string(wb, "status", rrdcalc_status2string(t->old_status));
@@ -407,9 +422,9 @@ void contexts_v2_alert_transitions_to_json(BUFFER *wb, struct rrdcontext_to_json
422
423 buffer_json_member_add_object(wb, "notification");
424 {
410 - buffer_json_member_add_time_t(wb, "when", t->exec_run_timestamp);
425 + buffer_json_member_add_time_t_formatted(wb, "when", t->exec_run_timestamp, ctl->options & CONTEXTS_OPTION_RFC3339);
426 buffer_json_member_add_time_t(wb, "delay", t->delay);
412 - buffer_json_member_add_time_t(wb, "delay_up_to_time", t->delay_up_to_timestamp);
427 + buffer_json_member_add_time_t_formatted(wb, "delay_up_to_time", t->delay_up_to_timestamp, ctl->options & CONTEXTS_OPTION_RFC3339);
428 health_entry_flags_to_json_array(wb, "flags", t->flags);
429 buffer_json_member_add_string(wb, "exec", *t->exec ? t->exec : string2str(localhost->health.default_exec));
430 buffer_json_member_add_uint64(wb, "exec_code", t->exec_code);
@@ -421,7 +436,7 @@ void contexts_v2_alert_transitions_to_json(BUFFER *wb, struct rrdcontext_to_json
436 }
437 buffer_json_array_close(wb); // all transitions
438
424 - if(ctl->options & CONTEXTS_OPTION_ALERTS_WITH_CONFIGURATIONS) {
439 + if(ctl->options & CONTEXTS_OPTION_CONFIGURATIONS) {
440 DICTIONARY *configs = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
441
442 for(struct sql_alert_transition_fixed_size *t = data.base; t ; t = t->next) {
src/database/contexts/api_v2_contexts_alerts.c
+264 -59
@@ -1,6 +1,7 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 #include "api_v2_contexts.h"
4 +#include "libnetdata/json/json-keys.h"
5
6 struct alert_counts {
7 size_t critical;
@@ -135,7 +136,7 @@ bool rrdcontext_matches_alert(struct rrdcontext_to_json_v2_data *ctl, RRDCONTEXT
136 sizeof(struct alert_by_x_entry),
137 rcl);
138
138 - if (ctl->options & (CONTEXTS_OPTION_ALERTS_WITH_INSTANCES | CONTEXTS_OPTION_ALERTS_WITH_VALUES)) {
139 + if (ctl->options & (CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_VALUES)) {
140 char key[20 + 1];
141 snprintfz(key, sizeof(key) - 1, "%p", rcl);
142
@@ -298,42 +299,54 @@ static int contexts_v2_alert_instance_to_json_callback(const DICTIONARY_ITEM *it
299
300 buffer_json_add_array_item_object(wb);
301 {
301 - buffer_json_member_add_uint64(wb, "ni", t->ni);
302 + if(ctl->request->options & CONTEXTS_OPTION_SUMMARY)
303 + buffer_json_member_add_uint64(wb, JSKEY(alerts_index_id), t->ati);
304
303 - buffer_json_member_add_string(wb, "nm", string2str(t->name));
304 - buffer_json_member_add_string(wb, "ch", string2str(t->chart_id));
305 - buffer_json_member_add_string(wb, "ch_n", string2str(t->chart_name));
305 + if(!(ctl->request->options & CONTEXTS_OPTION_MCP))
306 + buffer_json_member_add_uint64(wb, JSKEY(node_index), t->ni);
307
307 - if(ctl->request->options & CONTEXTS_OPTION_ALERTS_WITH_SUMMARY)
308 - buffer_json_member_add_uint64(wb, "ati", t->ati);
308 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES)
309 + buffer_json_member_add_uint64(wb, JSKEY(alert_global_id), t->global_id);
310
310 - if(ctl->request->options & CONTEXTS_OPTION_ALERTS_WITH_INSTANCES) {
311 - buffer_json_member_add_string(wb, "units", string2str(t->units));
312 - buffer_json_member_add_string(wb, "fami", string2str(t->family));
311 + buffer_json_member_add_string(wb, JSKEY(alert_name), string2str(t->name));
312 +
313 + if(ctl->request->options & CONTEXTS_OPTION_MCP)
314 + buffer_json_member_add_string(wb, JSKEY(hostname), string2str(t->host->hostname));
315 +
316 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES)
317 + buffer_json_member_add_string(wb, JSKEY(context), string2str(t->context));
318 +
319 + if(!(ctl->request->options & CONTEXTS_OPTION_MCP))
320 + buffer_json_member_add_string(wb, JSKEY(instance_id), string2str(t->chart_id));
321 +
322 + buffer_json_member_add_string(wb, JSKEY(instance_name), string2str(t->chart_name));
323 +
324 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES)
325 + buffer_json_member_add_string(wb, JSKEY(status), rrdcalc_status2string(t->status));
326 +
327 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES) {
328 + buffer_json_member_add_string(wb, JSKEY(family), string2str(t->family));
329 buffer_json_member_add_string(wb, "info", string2str(t->info));
314 - buffer_json_member_add_string(wb, "sum", string2str(t->summary));
315 - buffer_json_member_add_string(wb, "ctx", string2str(t->context));
316 - buffer_json_member_add_string(wb, "st", rrdcalc_status2string(t->status));
317 - buffer_json_member_add_uuid(wb, "tr_i", t->last_transition_id);
318 - buffer_json_member_add_double(wb, "tr_v", t->last_status_change_value);
319 - buffer_json_member_add_time_t(wb, "tr_t", t->last_status_change);
320 - buffer_json_member_add_uuid(wb, "cfg", t->config_hash_id);
321 - buffer_json_member_add_string(wb, "src", string2str(t->source));
322 -
323 - buffer_json_member_add_string(wb, "to", string2str(t->recipient));
324 - buffer_json_member_add_string(wb, "tp", string2str(t->type));
325 - buffer_json_member_add_string(wb, "cm", string2str(t->component));
326 - buffer_json_member_add_string(wb, "cl", string2str(t->classification));
327 -
328 - // Agent specific fields
329 - buffer_json_member_add_uint64(wb, "gi", t->global_id);
330 + buffer_json_member_add_string(wb, JSKEY(summary), string2str(t->summary));
331 + buffer_json_member_add_string(wb, "units", string2str(t->units));
332 + buffer_json_member_add_uuid(wb, JSKEY(last_transition_id), t->last_transition_id);
333 + buffer_json_member_add_double(wb, JSKEY(last_transition_value), t->last_status_change_value);
334 + buffer_json_member_add_time_t_formatted(wb, JSKEY(last_transition_timestamp), t->last_status_change, ctl->options & CONTEXTS_OPTION_RFC3339);
335 + buffer_json_member_add_uuid(wb, JSKEY(config_hash_id), t->config_hash_id);
336 + buffer_json_member_add_string(wb, JSKEY(source), string2str(t->source));
337 +
338 + buffer_json_member_add_string(wb, JSKEY(recipients), string2str(t->recipient));
339 + buffer_json_member_add_string(wb, JSKEY(type), string2str(t->type));
340 + buffer_json_member_add_string(wb, JSKEY(component), string2str(t->component));
341 + buffer_json_member_add_string(wb, JSKEY(classification), string2str(t->classification));
342 +
343 // rrdcalc_flags_to_json_array (wb, "flags", t->flags);
344 }
345
333 - if(ctl->request->options & CONTEXTS_OPTION_ALERTS_WITH_VALUES) {
346 + if(ctl->request->options & CONTEXTS_OPTION_VALUES) {
347 // Netdata Cloud fetched these by querying the agents
335 - buffer_json_member_add_double(wb, "v", t->value);
336 - buffer_json_member_add_time_t(wb, "t", t->last_updated);
348 + buffer_json_member_add_double(wb, JSKEY(last_updated_value), t->value);
349 + buffer_json_member_add_time_t_formatted(wb, JSKEY(last_updated_timestamp), t->last_updated, ctl->options & CONTEXTS_OPTION_RFC3339);
350 }
351 }
352 buffer_json_object_close(wb); // alert instance
@@ -358,10 +371,10 @@ static void contexts_v2_alerts_by_x_to_json(BUFFER *wb, DICTIONARY *dict, const
371 buffer_json_add_array_item_object(wb);
372 {
373 buffer_json_member_add_string(wb, "name", b_dfe.name);
361 - buffer_json_member_add_uint64(wb, "cr", b->running.counts.critical);
362 - buffer_json_member_add_uint64(wb, "wr", b->running.counts.warning);
363 - buffer_json_member_add_uint64(wb, "cl", b->running.counts.clear);
364 - buffer_json_member_add_uint64(wb, "er", b->running.counts.error);
374 + buffer_json_member_add_uint64(wb, JSKEY(critical), b->running.counts.critical);
375 + buffer_json_member_add_uint64(wb, JSKEY(warning), b->running.counts.warning);
376 + buffer_json_member_add_uint64(wb, JSKEY(clear), b->running.counts.clear);
377 + buffer_json_member_add_uint64(wb, JSKEY(error), b->running.counts.error);
378 buffer_json_member_add_uint64(wb, "running", b->running.total);
379
380 buffer_json_member_add_uint64(wb, "running_silent", b->running.silent);
@@ -390,8 +403,200 @@ static void contexts_v2_alert_instances_to_json(BUFFER *wb, const char *key, str
403 buffer_json_array_close(wb); // alerts_instances
404 }
405
406 +void contexts_v2_alerts_to_json_mcp(BUFFER *wb, struct rrdcontext_to_json_v2_data *ctl, bool debug __maybe_unused) {
407 + size_t cardinality_limit = ctl->request->cardinality_limit;
408 +
409 + if(ctl->request->options & CONTEXTS_OPTION_SUMMARY) {
410 + size_t alerts_count = 0;
411 + size_t total_alerts = dictionary_entries(ctl->alerts.summary);
412 +
413 + // Add header array
414 + buffer_json_member_add_array(wb, "all_alerts_header");
415 + {
416 + buffer_json_add_array_item_string(wb, "Alert Name");
417 + buffer_json_add_array_item_string(wb, "Alert Summary");
418 +
419 + buffer_json_add_array_item_string(wb, "Metrics Contexts");
420 + buffer_json_add_array_item_string(wb, "Alert Classifications");
421 + buffer_json_add_array_item_string(wb, "Alert Components");
422 + buffer_json_add_array_item_string(wb, "Alert Types");
423 + buffer_json_add_array_item_string(wb, "Notification Recipients");
424 +
425 + buffer_json_add_array_item_string(wb, "# of Critical Instances");
426 + buffer_json_add_array_item_string(wb, "# of Warning Instances");
427 + buffer_json_add_array_item_string(wb, "# of Clear Instances");
428 + buffer_json_add_array_item_string(wb, "# of Error Instances");
429 + buffer_json_add_array_item_string(wb, "# of Instances Watched");
430 + buffer_json_add_array_item_string(wb, "# of Nodes Watched");
431 + buffer_json_add_array_item_string(wb, "# of Alert Configurations");
432 + }
433 + buffer_json_array_close(wb); // header
434 +
435 + // Add data array
436 + buffer_json_member_add_array(wb, "all_alerts");
437 + struct alert_v2_entry *t;
438 + dfe_start_read(ctl->alerts.summary, t)
439 + {
440 + // Apply cardinality limit for summary
441 + if(cardinality_limit && alerts_count >= cardinality_limit)
442 + break;
443 +
444 + // Start row array
445 + buffer_json_add_array_item_array(wb);
446 + {
447 + // Add values in same order as header
448 + buffer_json_add_array_item_string(wb, string2str(t->name));
449 + buffer_json_add_array_item_string(wb, string2str(t->summary));
450 +
451 + rrdlabels_key_to_buffer_array_or_string_or_null(t->context, wb);
452 + rrdlabels_key_to_buffer_array_or_string_or_null(t->classification, wb);
453 + rrdlabels_key_to_buffer_array_or_string_or_null(t->component, wb);
454 + rrdlabels_key_to_buffer_array_or_string_or_null(t->type, wb);
455 + rrdlabels_key_to_buffer_array_or_string_or_null(t->recipient, wb);
456 +
457 + buffer_json_add_array_item_uint64(wb, t->counts.critical);
458 + buffer_json_add_array_item_uint64(wb, t->counts.warning);
459 + buffer_json_add_array_item_uint64(wb, t->counts.clear);
460 + buffer_json_add_array_item_uint64(wb, t->counts.error);
461 + buffer_json_add_array_item_uint64(wb, t->instances);
462 + buffer_json_add_array_item_uint64(wb, dictionary_entries(t->nodes));
463 + buffer_json_add_array_item_uint64(wb, dictionary_entries(t->configs));
464 + }
465 + buffer_json_array_close(wb);
466 + alerts_count++;
467 + }
468 + dfe_done(t);
469 + buffer_json_array_close(wb); // all_alerts
470 +
471 + // Add info about truncation if needed
472 + if(cardinality_limit && total_alerts > cardinality_limit) {
473 + buffer_json_member_add_object(wb, "__all_alerts_info__");
474 + buffer_json_member_add_string(wb, "status", "truncated");
475 + buffer_json_member_add_uint64(wb, "total_alerts", total_alerts);
476 + buffer_json_member_add_uint64(wb, "shown_alerts", alerts_count);
477 + buffer_json_member_add_uint64(wb, "cardinality_limit", cardinality_limit);
478 + buffer_json_object_close(wb);
479 + }
480 + }
481 +
482 + if(ctl->request->options & (CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_VALUES)) {
483 + size_t instances_count = 0;
484 + size_t total_instances = dictionary_entries(ctl->alerts.alert_instances);
485 +
486 + // Add header array for alert instances
487 + buffer_json_member_add_array(wb, "alert_instances_header");
488 + {
489 + buffer_json_add_array_item_string(wb, "Alert Name");
490 + buffer_json_add_array_item_string(wb, "Hostname");
491 +
492 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES)
493 + buffer_json_add_array_item_string(wb, "Context");
494 +
495 + buffer_json_add_array_item_string(wb, "Instance Name");
496 +
497 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES) {
498 + buffer_json_add_array_item_string(wb, "Status");
499 + buffer_json_add_array_item_string(wb, "Family");
500 + buffer_json_add_array_item_string(wb, "Info");
501 + buffer_json_add_array_item_string(wb, "Summary");
502 + buffer_json_add_array_item_string(wb, "Units");
503 + buffer_json_add_array_item_string(wb, "Last Transition ID");
504 + buffer_json_add_array_item_string(wb, "Last Transition Value");
505 + buffer_json_add_array_item_string(wb, "Last Transition Timestamp");
506 + buffer_json_add_array_item_string(wb, "Configuration Hash");
507 + buffer_json_add_array_item_string(wb, "Source");
508 + buffer_json_add_array_item_string(wb, "Recipients");
509 + buffer_json_add_array_item_string(wb, "Type");
510 + buffer_json_add_array_item_string(wb, "Component");
511 + buffer_json_add_array_item_string(wb, "Classification");
512 + }
513 +
514 + if(ctl->request->options & CONTEXTS_OPTION_VALUES) {
515 + buffer_json_add_array_item_string(wb, "Last Updated Value");
516 + buffer_json_add_array_item_string(wb, "Last Updated Timestamp");
517 + }
518 + }
519 + buffer_json_array_close(wb); // alert_instances_header
520 +
521 + // Add data array for alert instances
522 + buffer_json_member_add_array(wb, "alert_instances");
523 + {
524 + struct sql_alert_instance_v2_entry *t;
525 + dfe_start_read(ctl->alerts.alert_instances, t)
526 + {
527 + // Apply cardinality limit for instances
528 + if(cardinality_limit && instances_count >= cardinality_limit)
529 + break;
530 +
531 + // Start row array
532 + buffer_json_add_array_item_array(wb);
533 + {
534 + buffer_json_add_array_item_string(wb, string2str(t->name));
535 + buffer_json_add_array_item_string(wb, string2str(t->host->hostname));
536 +
537 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES)
538 + buffer_json_add_array_item_string(wb, string2str(t->context));
539 +
540 + buffer_json_add_array_item_string(wb, string2str(t->chart_name));
541 +
542 + if(ctl->request->options & CONTEXTS_OPTION_INSTANCES) {
543 + buffer_json_add_array_item_string(wb, rrdcalc_status2string(t->status));
544 + buffer_json_add_array_item_string(wb, string2str(t->family));
545 + buffer_json_add_array_item_string(wb, string2str(t->info));
546 + buffer_json_add_array_item_string(wb, string2str(t->summary));
547 + buffer_json_add_array_item_string(wb, string2str(t->units));
548 +
549 + // Add UUID as string
550 + char uuid_str[UUID_STR_LEN];
551 + uuid_unparse_lower(t->last_transition_id, uuid_str);
552 + buffer_json_add_array_item_string(wb, uuid_str);
553 +
554 + buffer_json_add_array_item_double(wb, t->last_status_change_value);
555 + buffer_json_add_array_item_time_t_formatted(wb, t->last_status_change, ctl->options & CONTEXTS_OPTION_RFC3339);
556 +
557 + // Add config hash UUID as string
558 + uuid_unparse_lower(t->config_hash_id, uuid_str);
559 + buffer_json_add_array_item_string(wb, uuid_str);
560 +
561 + buffer_json_add_array_item_string(wb, string2str(t->source));
562 + buffer_json_add_array_item_string(wb, string2str(t->recipient));
563 + buffer_json_add_array_item_string(wb, string2str(t->type));
564 + buffer_json_add_array_item_string(wb, string2str(t->component));
565 + buffer_json_add_array_item_string(wb, string2str(t->classification));
566 + }
567 +
568 + if(ctl->request->options & CONTEXTS_OPTION_VALUES) {
569 + buffer_json_add_array_item_double(wb, t->value);
570 + buffer_json_add_array_item_time_t_formatted(wb, t->last_updated, ctl->options & CONTEXTS_OPTION_RFC3339);
571 + }
572 + }
573 + buffer_json_array_close(wb); // row
574 + instances_count++;
575 + }
576 + dfe_done(t);
577 + }
578 + buffer_json_array_close(wb); // alert_instances
579 +
580 + // Add info about truncation if needed
581 + if(cardinality_limit && total_instances > cardinality_limit) {
582 + buffer_json_member_add_object(wb, "__alert_instances_info__");
583 + buffer_json_member_add_string(wb, "status", "truncated");
584 + buffer_json_member_add_uint64(wb, "total_instances", total_instances);
585 + buffer_json_member_add_uint64(wb, "shown_instances", instances_count);
586 + buffer_json_member_add_uint64(wb, "cardinality_limit", cardinality_limit);
587 + buffer_json_object_close(wb);
588 + }
589 + }
590 +
591 +}
592 +
593 void contexts_v2_alerts_to_json(BUFFER *wb, struct rrdcontext_to_json_v2_data *ctl, bool debug) {
394 - if(ctl->request->options & CONTEXTS_OPTION_ALERTS_WITH_SUMMARY) {
594 + if(ctl->request->options & CONTEXTS_OPTION_MCP) {
595 + contexts_v2_alerts_to_json_mcp(wb, ctl, debug);
596 + return;
597 + }
598 +
599 + if(ctl->request->options & CONTEXTS_OPTION_SUMMARY) {
600 buffer_json_member_add_array(wb, "alerts");
601 {
602 struct alert_v2_entry *t;
@@ -399,49 +604,49 @@ void contexts_v2_alerts_to_json(BUFFER *wb, struct rrdcontext_to_json_v2_data *c
604 {
605 buffer_json_add_array_item_object(wb);
606 {
402 - buffer_json_member_add_uint64(wb, "ati", t->ati);
607 + buffer_json_member_add_uint64(wb, JSKEY(alerts_index_id), t->ati);
608
404 - buffer_json_member_add_array(wb, "ni");
609 + buffer_json_member_add_array(wb, JSKEY(node_index));
610 void *host_guid;
611 dfe_start_read(t->nodes, host_guid) {
407 - struct contexts_v2_node *cn = dictionary_get(ctl->nodes.dict,host_guid_dfe.name);
612 + struct contexts_v2_node *cn = dictionary_get(ctl->nodes.dict, host_guid_dfe.name);
613 buffer_json_add_array_item_int64(wb, (int64_t) cn->ni);
614 }
615 dfe_done(host_guid);
616 buffer_json_array_close(wb);
617
413 - buffer_json_member_add_string(wb, "nm", string2str(t->name));
414 - buffer_json_member_add_string(wb, "sum", string2str(t->summary));
618 + buffer_json_member_add_string(wb, JSKEY(alert_name), string2str(t->name));
619 + buffer_json_member_add_string(wb, JSKEY(summary), string2str(t->summary));
620
416 - buffer_json_member_add_uint64(wb, "cr", t->counts.critical);
417 - buffer_json_member_add_uint64(wb, "wr", t->counts.warning);
418 - buffer_json_member_add_uint64(wb, "cl", t->counts.clear);
419 - buffer_json_member_add_uint64(wb, "er", t->counts.error);
621 + buffer_json_member_add_uint64(wb, JSKEY(critical), t->counts.critical);
622 + buffer_json_member_add_uint64(wb, JSKEY(warning), t->counts.warning);
623 + buffer_json_member_add_uint64(wb, JSKEY(clear), t->counts.clear);
624 + buffer_json_member_add_uint64(wb, JSKEY(error), t->counts.error);
625
421 - buffer_json_member_add_uint64(wb, "in", t->instances);
422 - buffer_json_member_add_uint64(wb, "nd", dictionary_entries(t->nodes));
423 - buffer_json_member_add_uint64(wb, "cfg", dictionary_entries(t->configs));
626 + buffer_json_member_add_uint64(wb, JSKEY(instances_count), t->instances);
627 + buffer_json_member_add_uint64(wb, JSKEY(nodes_count), dictionary_entries(t->nodes));
628 + buffer_json_member_add_uint64(wb, JSKEY(configurations_count), dictionary_entries(t->configs));
629
425 - buffer_json_member_add_array(wb, "ctx");
630 + buffer_json_member_add_array(wb, JSKEY(contexts));
631 rrdlabels_key_to_buffer_array_item(t->context, wb);
427 - buffer_json_array_close(wb); // ctx
632 + buffer_json_array_close(wb); // contexts
633
429 - buffer_json_member_add_array(wb, "cls");
634 + buffer_json_member_add_array(wb, JSKEY(classifications));
635 rrdlabels_key_to_buffer_array_item(t->classification, wb);
431 - buffer_json_array_close(wb); // classification
636 + buffer_json_array_close(wb); // classifications
637
638
434 - buffer_json_member_add_array(wb, "cp");
639 + buffer_json_member_add_array(wb, JSKEY(components));
640 rrdlabels_key_to_buffer_array_item(t->component, wb);
436 - buffer_json_array_close(wb); // component
641 + buffer_json_array_close(wb); // components
642
438 - buffer_json_member_add_array(wb, "ty");
643 + buffer_json_member_add_array(wb, JSKEY(types));
644 rrdlabels_key_to_buffer_array_item(t->type, wb);
440 - buffer_json_array_close(wb); // type
645 + buffer_json_array_close(wb); // types
646
442 - buffer_json_member_add_array(wb, "to");
647 + buffer_json_member_add_array(wb, JSKEY(recipients));
648 rrdlabels_key_to_buffer_array_item(t->recipient, wb);
444 - buffer_json_array_close(wb); // recipient
649 + buffer_json_array_close(wb); // recipients
650 }
651 buffer_json_object_close(wb); // alert name
652 }
@@ -457,7 +662,7 @@ void contexts_v2_alerts_to_json(BUFFER *wb, struct rrdcontext_to_json_v2_data *c
662 contexts_v2_alerts_by_x_to_json(wb, ctl->alerts.by_module, "alerts_by_module");
663 }
664
460 - if(ctl->request->options & (CONTEXTS_OPTION_ALERTS_WITH_INSTANCES | CONTEXTS_OPTION_ALERTS_WITH_VALUES)) {
665 + if(ctl->request->options & (CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_VALUES)) {
666 contexts_v2_alert_instances_to_json(wb, "alert_instances", ctl, debug);
667 }
668 }
@@ -533,7 +738,7 @@ static void rrdcontext_v2_set_transition_filter(const char *machine_guid, const
738
739 bool rrdcontexts_v2_init_alert_dictionaries(struct rrdcontext_to_json_v2_data *ctl, struct api_v2_contexts_request *req) {
740 if(req->alerts.transition) {
536 - ctl->options |= CONTEXTS_OPTION_ALERTS_WITH_INSTANCES | CONTEXTS_OPTION_ALERTS_WITH_VALUES;
741 + ctl->options |= CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_VALUES;
742 if(!sql_find_alert_transition(req->alerts.transition, rrdcontext_v2_set_transition_filter, ctl))
743 return false;
744 }
@@ -587,7 +792,7 @@ bool rrdcontexts_v2_init_alert_dictionaries(struct rrdcontext_to_json_v2_data *c
792 dictionary_register_insert_callback(ctl->alerts.by_module, alerts_by_x_insert_callback, NULL);
793 dictionary_register_conflict_callback(ctl->alerts.by_module, alerts_by_x_conflict_callback, NULL);
794
590 - if(ctl->options & (CONTEXTS_OPTION_ALERTS_WITH_INSTANCES | CONTEXTS_OPTION_ALERTS_WITH_VALUES)) {
795 + if(ctl->options & (CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_VALUES)) {
796 ctl->alerts.alert_instances = dictionary_create_advanced(
797 DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
798 NULL, sizeof(struct sql_alert_instance_v2_entry));
src/database/contexts/query_target.c
+206 -31
@@ -1,6 +1,7 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 #include "rrdcontext-internal.h"
4 +#include "database/pattern-array.h"
5
6 #define QUERY_TARGET_MAX_REALLOC_INCREASE 500
7 #define query_target_realloc_size(size, start) \
@@ -77,13 +78,32 @@ void query_target_release(QUERY_TARGET *qt) {
78 simple_pattern_free(qt->instances.pattern);
79 qt->instances.pattern = NULL;
80
81 + simple_pattern_free(qt->instances.scope_pattern);
82 + qt->instances.scope_pattern = NULL;
83 +
84 simple_pattern_free(qt->instances.chart_label_key_pattern);
85 qt->instances.chart_label_key_pattern = NULL;
86
87 + simple_pattern_free(qt->instances.scope_chart_label_key_pattern);
88 + qt->instances.scope_chart_label_key_pattern = NULL;
89 +
90 simple_pattern_free(qt->instances.labels_pattern);
91 qt->instances.labels_pattern = NULL;
92 +
93 + simple_pattern_free(qt->instances.scope_labels_pattern);
94 + qt->instances.scope_labels_pattern = NULL;
95 +
96 + pattern_array_free(qt->instances.labels_pa);
97 + qt->instances.labels_pa = NULL;
98 +
99 + pattern_array_free(qt->instances.scope_labels_pa);
100 + qt->instances.scope_labels_pa = NULL;
101 +
102 simple_pattern_free(qt->query.pattern);
103 qt->query.pattern = NULL;
104 +
105 + simple_pattern_free(qt->dimensions.scope_pattern);
106 + qt->dimensions.scope_pattern = NULL;
107
108 // release the query
109 for(size_t i = 0, used = qt->query.used; i < used ;i++) {
@@ -190,6 +210,9 @@ typedef struct query_target_locals {
210
211 const char *scope_nodes;
212 const char *scope_contexts;
213 + const char *scope_instances;
214 + const char *scope_labels;
215 + const char *scope_dimensions;
216
217 const char *nodes;
218 const char *contexts;
@@ -389,6 +412,20 @@ static bool query_dimension_add(QUERY_TARGET_LOCALS *qtl, QUERY_NODE *qn, QUERY_
412 RRDMETRIC *rm = rrdmetric_acquired_value(rma);
413 if(rrd_flag_is_deleted(rm))
414 return false;
415 +
416 + // Check scope_dimensions first - if it doesn't match, skip entirely
417 + if(qt->dimensions.scope_pattern) {
418 + SIMPLE_PATTERN_RESULT ret = SP_NOT_MATCHED;
419 +
420 + if(qtl->match_ids)
421 + ret = simple_pattern_matches_string_extract(qt->dimensions.scope_pattern, rm->id, NULL, 0);
422 +
423 + if(ret == SP_NOT_MATCHED && qtl->match_names && (rm->name != rm->id || !qtl->match_ids))
424 + ret = simple_pattern_matches_string_extract(qt->dimensions.scope_pattern, rm->name, NULL, 0);
425 +
426 + if(ret != SP_MATCHED_POSITIVE)
427 + return false; // Skip this dimension entirely - not in scope
428 + }
429
430 QUERY_STATUS status = QUERY_STATUS_NONE;
431
@@ -662,17 +699,21 @@ static bool query_target_match_alert_pattern(RRDINSTANCE_ACQUIRED *ria, SIMPLE_P
699 return matched;
700 }
701
702 +static inline void query_instance_strings_free(QUERY_INSTANCE *qi) {
703 + string_freez(qi->id_fqdn);
704 + qi->id_fqdn = NULL;
705 +
706 + string_freez(qi->name_fqdn);
707 + qi->name_fqdn = NULL;
708 +}
709 +
710 static inline void query_instance_release(QUERY_INSTANCE *qi) {
711 if(qi->ria) {
712 rrdinstance_release(qi->ria);
713 qi->ria = NULL;
714 }
715
671 - string_freez(qi->id_fqdn);
672 - qi->id_fqdn = NULL;
673 -
674 - string_freez(qi->name_fqdn);
675 - qi->name_fqdn = NULL;
716 + query_instance_strings_free(qi);
717 }
718
719 static inline QUERY_INSTANCE *query_instance_allocate(QUERY_TARGET *qt, RRDINSTANCE_ACQUIRED *ria, size_t qn_slot) {
@@ -729,18 +770,14 @@ static inline SIMPLE_PATTERN_RESULT query_instance_matches(QUERY_INSTANCE *qi,
770 static inline bool query_instance_matches_labels(
771 RRDINSTANCE *ri,
772 SIMPLE_PATTERN *chart_label_key_sp,
732 - SIMPLE_PATTERN *labels_sp)
773 + struct pattern_array *labels_pa)
774 {
734 -
775 RRDLABELS *labels = rrdinstance_labels(ri);
776 if (chart_label_key_sp && rrdlabels_match_simple_pattern_parsed(labels, chart_label_key_sp, '\0', NULL) != SP_MATCHED_POSITIVE)
777 return false;
778
739 - if (labels_sp) {
740 - struct pattern_array *pa = pattern_array_add_simple_pattern(NULL, labels_sp, ':');
741 - bool found = pattern_array_label_match(pa, labels, ':', NULL);
742 - pattern_array_free(pa);
743 - return found;
779 + if (labels_pa) {
780 + return pattern_array_label_match(labels_pa, labels, ':', NULL);
781 }
782
783 return true;
@@ -766,7 +803,7 @@ static bool query_instance_add(QUERY_TARGET_LOCALS *qtl, QUERY_NODE *qn, QUERY_C
803 queryable_instance = query_instance_matches_labels(
804 ri,
805 qt->instances.chart_label_key_pattern,
769 - qt->instances.labels_pattern);
806 + qt->instances.labels_pa);
807
808 if(queryable_instance) {
809 if(qt->instances.alerts_pattern && !query_target_match_alert_pattern(ria, qt->instances.alerts_pattern))
@@ -833,34 +870,113 @@ static inline QUERY_CONTEXT *query_context_allocate(QUERY_TARGET *qt, RRDCONTEXT
870 return qc;
871 }
872
836 -static ssize_t query_context_add(void *data, RRDCONTEXT_ACQUIRED *rca, bool queryable_context) {
837 - QUERY_TARGET_LOCALS *qtl = data;
838 -
839 - RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
840 - if(rrd_flag_is_deleted(rc))
841 - return 0;
842 -
843 - QUERY_NODE *qn = qtl->qn;
873 +static ssize_t query_scope_foreach_instance(QUERY_TARGET_LOCALS *qtl, QUERY_NODE *qn, QUERY_CONTEXT *qc,
874 + RRDCONTEXT_ACQUIRED *rca, bool queryable_context) {
875 QUERY_TARGET *qt = qtl->qt;
845 - QUERY_CONTEXT *qc = query_context_allocate(qt, rca);
846 -
876 + RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
877 ssize_t added = 0;
878 +
879 if(unlikely(qt->request.ria)) {
880 + // Single instance requested
881 + RRDINSTANCE *ri = rrdinstance_acquired_value(qt->request.ria);
882 +
883 + // Check scope_instances
884 + if(qt->instances.scope_pattern) {
885 + QUERY_INSTANCE qi = { .ria = qt->request.ria };
886 + SIMPLE_PATTERN_RESULT ret = query_instance_matches(&qi, ri,
887 + qt->instances.scope_pattern, qtl->match_ids, qtl->match_names,
888 + qt->request.version, qtl->host_node_id_str);
889 + query_instance_strings_free(&qi);
890 + if(ret != SP_MATCHED_POSITIVE)
891 + return 0;
892 + }
893 +
894 + // Check scope_labels
895 + if(qt->instances.scope_labels_pa || qt->instances.scope_chart_label_key_pattern) {
896 + if(!query_instance_matches_labels(ri,
897 + qt->instances.scope_chart_label_key_pattern,
898 + qt->instances.scope_labels_pa))
899 + return 0;
900 + }
901 +
902 if(query_instance_add(qtl, qn, qc, qt->request.ria, queryable_context, false))
903 added++;
904 }
905 else if(unlikely(qtl->st && qtl->st->rrdcontexts.rrdcontext == rca && qtl->st->rrdcontexts.rrdinstance)) {
906 + // Single chart requested
907 + RRDINSTANCE *ri = rrdinstance_acquired_value(qtl->st->rrdcontexts.rrdinstance);
908 +
909 + // Check scope_instances
910 + if(qt->instances.scope_pattern) {
911 + QUERY_INSTANCE qi = { .ria = qtl->st->rrdcontexts.rrdinstance };
912 + SIMPLE_PATTERN_RESULT ret = query_instance_matches(&qi, ri,
913 + qt->instances.scope_pattern, qtl->match_ids, qtl->match_names,
914 + qt->request.version, qtl->host_node_id_str);
915 + query_instance_strings_free(&qi);
916 + if(ret != SP_MATCHED_POSITIVE)
917 + return 0;
918 + }
919 +
920 + // Check scope_labels
921 + if(qt->instances.scope_labels_pa || qt->instances.scope_chart_label_key_pattern) {
922 + if(!query_instance_matches_labels(ri,
923 + qt->instances.scope_chart_label_key_pattern,
924 + qt->instances.scope_labels_pa))
925 + return 0;
926 + }
927 +
928 if(query_instance_add(qtl, qn, qc, qtl->st->rrdcontexts.rrdinstance, queryable_context, false))
929 added++;
930 }
931 else {
932 + // Pattern query - iterate through all instances
933 RRDINSTANCE *ri;
934 dfe_start_read(rc->rrdinstances, ri) {
859 - if(query_instance_add(qtl, qn, qc, (RRDINSTANCE_ACQUIRED *) ri_dfe.item, queryable_context, true))
860 - added++;
861 - }
935 + if(rrd_flag_is_deleted(ri))
936 + continue;
937 +
938 + RRDINSTANCE_ACQUIRED *ria = (RRDINSTANCE_ACQUIRED *) ri_dfe.item;
939 +
940 + // Check scope_instances
941 + if(qt->instances.scope_pattern) {
942 + QUERY_INSTANCE qi = { .ria = ria };
943 + SIMPLE_PATTERN_RESULT ret = query_instance_matches(&qi, ri,
944 + qt->instances.scope_pattern, qtl->match_ids, qtl->match_names,
945 + qt->request.version, qtl->host_node_id_str);
946 + query_instance_strings_free(&qi);
947 + if(ret != SP_MATCHED_POSITIVE)
948 + continue;
949 + }
950 +
951 + // Check scope_labels
952 + if(qt->instances.scope_labels_pa || qt->instances.scope_chart_label_key_pattern) {
953 + if(!query_instance_matches_labels(ri,
954 + qt->instances.scope_chart_label_key_pattern,
955 + qt->instances.scope_labels_pa))
956 + continue;
957 + }
958 +
959 + if(query_instance_add(qtl, qn, qc, ria, queryable_context, true))
960 + added++;
961 + }
962 dfe_done(ri);
963 }
964 +
965 + return added;
966 +}
967 +
968 +static ssize_t query_context_add(void *data, RRDCONTEXT_ACQUIRED *rca, bool queryable_context) {
969 + QUERY_TARGET_LOCALS *qtl = data;
970 +
971 + RRDCONTEXT *rc = rrdcontext_acquired_value(rca);
972 + if(rrd_flag_is_deleted(rc))
973 + return 0;
974 +
975 + QUERY_NODE *qn = qtl->qn;
976 + QUERY_TARGET *qt = qtl->qt;
977 + QUERY_CONTEXT *qc = query_context_allocate(qt, rca);
978 +
979 + ssize_t added = query_scope_foreach_instance(qtl, qn, qc, rca, queryable_context);
980
981 if(!added) {
982 query_context_release(qc);
@@ -999,9 +1115,12 @@ void query_target_generate_name(QUERY_TARGET *qt) {
1115 , tier_buffer
1116 );
1117 else if(qt->request.version >= 2)
1002 - snprintfz(qt->id, MAX_QUERY_TARGET_ID_LENGTH, "data_v2://scope_nodes:%s/scope_contexts:%s/nodes:%s/contexts:%s/instances:%s/labels:%s/dimensions:%s/after:%lld/before:%lld/points:%zu/time_group:%s%s/options:%s%s%s"
1118 + snprintfz(qt->id, MAX_QUERY_TARGET_ID_LENGTH, "data_v2://scope_nodes:%s/scope_contexts:%s/scope_instances:%s/scope_labels:%s/scope_dimensions:%s/nodes:%s/contexts:%s/instances:%s/labels:%s/dimensions:%s/after:%lld/before:%lld/points:%zu/time_group:%s%s/options:%s%s%s"
1119 , qt->request.scope_nodes ? qt->request.scope_nodes : "*"
1120 , qt->request.scope_contexts ? qt->request.scope_contexts : "*"
1121 + , qt->request.scope_instances ? qt->request.scope_instances : "*"
1122 + , qt->request.scope_labels ? qt->request.scope_labels : "*"
1123 + , qt->request.scope_dimensions ? qt->request.scope_dimensions : "*"
1124 , qt->request.nodes ? qt->request.nodes : "*"
1125 , (qt->request.contexts) ? qt->request.contexts : "*"
1126 , (qt->request.instances) ? qt->request.instances : "*"
@@ -1052,6 +1171,16 @@ QUERY_TARGET *query_target_create(QUERY_TARGET_REQUEST *qtr) {
1171 if(qtr->contexts && !qtr->scope_contexts)
1172 qtr->scope_contexts = qtr->contexts;
1173
1174 + // IMPORTANT: old dashboards do not know about scope_instances and scope_labels
1175 + // so this code makes non-scope instances and labels to be used as scope.
1176 + // Leave it commented!
1177 +
1178 + // if(qtr->instances && !qtr->scope_instances)
1179 + // qtr->scope_instances = qtr->instances;
1180 + //
1181 + // if(qtr->labels && !qtr->scope_labels)
1182 + // qtr->scope_labels = qtr->labels;
1183 +
1184 memset(&qt->db, 0, sizeof(qt->db));
1185 qt->query_points = STORAGE_POINT_UNSET;
1186
@@ -1077,6 +1206,9 @@ QUERY_TARGET *query_target_create(QUERY_TARGET_REQUEST *qtr) {
1206 .st = qt->request.st,
1207 .scope_nodes = qt->request.scope_nodes,
1208 .scope_contexts = qt->request.scope_contexts,
1209 + .scope_instances = qt->request.scope_instances,
1210 + .scope_labels = qt->request.scope_labels,
1211 + .scope_dimensions = qt->request.scope_dimensions,
1212 .nodes = qt->request.nodes,
1213 .contexts = qt->request.contexts,
1214 .instances = qt->request.instances,
@@ -1096,10 +1228,20 @@ QUERY_TARGET *query_target_create(QUERY_TARGET_REQUEST *qtr) {
1228 qt->contexts.scope_pattern = string_to_simple_pattern(qtl.scope_contexts);
1229
1230 qt->instances.pattern = string_to_simple_pattern(qtl.instances);
1231 + qt->instances.scope_pattern = string_to_simple_pattern(qtl.scope_instances);
1232 qt->query.pattern = string_to_simple_pattern(qtl.dimensions);
1233 + qt->dimensions.scope_pattern = string_to_simple_pattern(qtl.scope_dimensions);
1234 qt->instances.chart_label_key_pattern = string_to_simple_pattern(qtl.chart_label_key);
1235 + qt->instances.scope_chart_label_key_pattern = string_to_simple_pattern(qtl.chart_label_key); // For now, using same as non-scope
1236 qt->instances.labels_pattern = string_to_simple_pattern(qtl.labels);
1237 + qt->instances.scope_labels_pattern = string_to_simple_pattern(qtl.scope_labels);
1238 qt->instances.alerts_pattern = string_to_simple_pattern(qtl.alerts);
1239 +
1240 + // Pre-compile pattern arrays for labels
1241 + if(qt->instances.labels_pattern)
1242 + qt->instances.labels_pa = pattern_array_add_simple_pattern(NULL, qt->instances.labels_pattern, ':');
1243 + if(qt->instances.scope_labels_pattern)
1244 + qt->instances.scope_labels_pa = pattern_array_add_simple_pattern(NULL, qt->instances.scope_labels_pattern, ':');
1245
1246 qtl.match_ids = qt->request.options & RRDR_OPTION_MATCH_IDS;
1247 qtl.match_names = qt->request.options & RRDR_OPTION_MATCH_NAMES;
@@ -1150,9 +1292,12 @@ QUERY_TARGET *query_target_create(QUERY_TARGET_REQUEST *qtr) {
1292 }
1293
1294 ssize_t weights_foreach_rrdmetric_in_context(RRDCONTEXT_ACQUIRED *rca,
1295 + SIMPLE_PATTERN *scope_instances_sp,
1296 + struct pattern_array *scope_labels_pa,
1297 + SIMPLE_PATTERN *scope_dimensions_sp,
1298 SIMPLE_PATTERN *instances_sp,
1299 SIMPLE_PATTERN *chart_label_key_sp,
1155 - SIMPLE_PATTERN *labels_sp,
1300 + struct pattern_array *labels_pa,
1301 SIMPLE_PATTERN *alerts_sp,
1302 SIMPLE_PATTERN *dimensions_sp,
1303 bool match_ids, bool match_names,
@@ -1175,18 +1320,33 @@ ssize_t weights_foreach_rrdmetric_in_context(RRDCONTEXT_ACQUIRED *rca,
1320 continue;
1321
1322 RRDINSTANCE_ACQUIRED *ria = (RRDINSTANCE_ACQUIRED *) ri_dfe.item;
1323 +
1324 + // Check scope_instances first - if it doesn't match, skip entirely
1325 + if(scope_instances_sp) {
1326 + QUERY_INSTANCE qi = { .ria = ria, };
1327 + SIMPLE_PATTERN_RESULT ret = query_instance_matches(&qi, ri, scope_instances_sp, match_ids, match_names, version, host_node_id_str);
1328 + query_instance_strings_free(&qi);
1329 +
1330 + if (ret != SP_MATCHED_POSITIVE)
1331 + continue;
1332 + }
1333 +
1334 + // Check scope_labels - if it doesn't match, skip entirely
1335 + if(scope_labels_pa) {
1336 + if(!query_instance_matches_labels(ri, NULL, scope_labels_pa))
1337 + continue;
1338 + }
1339
1340 if(instances_sp) {
1341 QUERY_INSTANCE qi = { .ria = ria, };
1342 SIMPLE_PATTERN_RESULT ret = query_instance_matches(&qi, ri, instances_sp, match_ids, match_names, version, host_node_id_str);
1182 - qi.ria = NULL;
1183 - query_instance_release(&qi);
1343 + query_instance_strings_free(&qi);
1344
1345 if (ret != SP_MATCHED_POSITIVE)
1346 continue;
1347 }
1348
1189 - if(!query_instance_matches_labels(ri, chart_label_key_sp, labels_sp))
1349 + if(!query_instance_matches_labels(ri, chart_label_key_sp, labels_pa))
1350 continue;
1351
1352 if(alerts_sp && !query_target_match_alert_pattern(ria, alerts_sp))
@@ -1199,6 +1359,20 @@ ssize_t weights_foreach_rrdmetric_in_context(RRDCONTEXT_ACQUIRED *rca,
1359 if(rrd_flag_is_deleted(rm))
1360 continue;
1361
1362 + // Check scope_dimensions first - if it doesn't match, skip entirely
1363 + if(scope_dimensions_sp) {
1364 + SIMPLE_PATTERN_RESULT ret = SP_NOT_MATCHED;
1365 +
1366 + if (match_ids)
1367 + ret = simple_pattern_matches_string_extract(scope_dimensions_sp, rm->id, NULL, 0);
1368 +
1369 + if (ret == SP_NOT_MATCHED && match_names && (rm->name != rm->id || !match_ids))
1370 + ret = simple_pattern_matches_string_extract(scope_dimensions_sp, rm->name, NULL, 0);
1371 +
1372 + if(ret != SP_MATCHED_POSITIVE)
1373 + continue;
1374 + }
1375 +
1376 if(dimensions_sp) {
1377 SIMPLE_PATTERN_RESULT ret = SP_NOT_MATCHED;
1378
@@ -1230,5 +1404,6 @@ ssize_t weights_foreach_rrdmetric_in_context(RRDCONTEXT_ACQUIRED *rca,
1404 break;
1405 }
1406 dfe_done(ri);
1407 +
1408 return count;
1409 }
src/database/contexts/rrdcontext-context-registry.c
+37 -9
@@ -3,6 +3,7 @@
3 #include "rrdcontext.h"
4 #include "rrdcontext-internal.h"
5 #include "rrdcontext-context-registry.h"
6 +#include "web/mcp/mcp.h"
7
8 // The registry - using a raw JudyL array
9 // Key: STRING pointer
@@ -126,6 +127,16 @@ size_t rrdcontext_context_registry_unique_count(void) {
127 void rrdcontext_context_registry_json_mcp_array(BUFFER *wb, SIMPLE_PATTERN *pattern) {
128 spinlock_lock(&context_registry_spinlock);
129
130 + buffer_json_member_add_object(wb, "info");
131 + {
132 + buffer_json_member_add_string(wb, "instructions",
133 + "The following is the list of contexts.\n"
134 + "You can get additional information for any context by calling,\n"
135 + "the tool " MCP_TOOL_GET_METRICS_DETAILS " with params:\n"
136 + "`metrics=context1|context2` to get more information about context1 and context2.\n");
137 + }
138 + buffer_json_object_close(wb); // info
139 +
140 buffer_json_member_add_array(wb, "header");
141 buffer_json_add_array_item_string(wb, "context");
142 buffer_json_add_array_item_string(wb, "number_of_nodes_having_it");
@@ -163,15 +174,8 @@ void rrdcontext_context_registry_json_mcp_categories_array(BUFFER *wb, SIMPLE_PA
174 // JudyL array to store unique category STRINGs as keys and counts as values
175 Pvoid_t categories_judyl = NULL;
176
166 - // Header information
167 - buffer_json_member_add_array(wb, "header");
168 - buffer_json_add_array_item_string(wb, "category");
169 - buffer_json_add_array_item_string(wb, "number_of_contexts");
170 - buffer_json_array_close(wb);
171 -
172 - buffer_json_member_add_array(wb, "categories");
173 -
177 // First pass: count occurrences of each category
178 + size_t contexts_count = 0, contexts_size = 0;
179 Word_t index = 0;
180 bool first = true;
181 Pvoid_t *PValue;
@@ -179,7 +183,9 @@ void rrdcontext_context_registry_json_mcp_categories_array(BUFFER *wb, SIMPLE_PA
183 if (!index || !*PValue) continue;
184
185 const char *context_name = string2str((STRING *)index);
182 -
186 + contexts_size += string_strlen((STRING *)index) + 10;
187 + contexts_count++;
188 +
189 // Find the last dot in the context name
190 const char *first_dot = strchr(context_name, '.');
191
@@ -214,6 +220,28 @@ void rrdcontext_context_registry_json_mcp_categories_array(BUFFER *wb, SIMPLE_PA
220 }
221
222 // Second pass: output the unique categories and their counts
223 +
224 + // Header information
225 + buffer_json_member_add_object(wb, "info");
226 + {
227 + buffer_json_member_add_uint64(wb, "original_contexts_count", contexts_count);
228 + buffer_json_member_add_uint64(wb, "original_contexts_size", contexts_size);
229 + buffer_json_member_add_string(wb, "instructions",
230 + "The following list groups metric contexts by prefix.\n"
231 + "In case the original list of contexts is too big to be processed at once,\n"
232 + "use the `q` parameter to fetch the contexts in smaller batches.\n"
233 + "Example: call the " MCP_TOOL_LIST_METRICS " with params:\n"
234 + "`q=system.*|net.*` to get all system.* and net.* contexts\n");
235 + }
236 + buffer_json_object_close(wb); // info
237 +
238 + buffer_json_member_add_array(wb, "header");
239 + buffer_json_add_array_item_string(wb, "category");
240 + buffer_json_add_array_item_string(wb, "number_of_contexts");
241 + buffer_json_array_close(wb);
242 +
243 + buffer_json_member_add_array(wb, "categories");
244 +
245 index = 0;
246 first = true;
247 while ((PValue = JudyLFirstThenNext(categories_judyl, &index, &first))) {
src/database/contexts/rrdcontext.h
+30 -9
@@ -3,6 +3,9 @@
3 #ifndef NETDATA_RRDCONTEXT_H
4 #define NETDATA_RRDCONTEXT_H 1
5
6 +// Forward declaration for pattern_array
7 +struct pattern_array;
8 +
9 // ----------------------------------------------------------------------------
10 // RRDMETRIC
11
@@ -75,6 +78,7 @@ typedef enum {
78 RRDCONTEXT_OPTION_DEEPSCAN = (1 << 6),
79 RRDCONTEXT_OPTION_SHOW_UUIDS = (1 << 7),
80 RRDCONTEXT_OPTION_SHOW_HIDDEN = (1 << 8),
81 + RRDCONTEXT_OPTION_RFC3339 = (1 << 9), // Return timestamps in RFC3339 format
82 RRDCONTEXT_OPTION_SKIP_ID = (1 << 31), // internal use
83 } RRDCONTEXT_TO_JSON_OPTIONS;
84
@@ -258,6 +262,9 @@ typedef struct query_target_request {
262
263 const char *scope_nodes;
264 const char *scope_contexts;
265 + const char *scope_instances;
266 + const char *scope_labels;
267 + const char *scope_dimensions;
268
269 // selecting / filtering metrics to be queried
270 RRDHOST *host; // the host to be queried (can be NULL, hosts will be used)
@@ -294,6 +301,9 @@ typedef struct query_target_request {
301
302 // group by across multiple time-series
303 struct group_by_pass group_by[MAX_QUERY_GROUP_BY_PASSES];
304 +
305 + // limiting cardinality of summary lists
306 + size_t cardinality_limit; // limit the number of items per category in summary
307
308 usec_t received_ut;
309
@@ -371,6 +381,7 @@ typedef struct query_target {
381 QUERY_DIMENSION *array;
382 uint32_t used; // how many items of the array are used
383 uint32_t size; // the size of the array
384 + SIMPLE_PATTERN *scope_pattern;
385 } dimensions;
386
387 struct {
@@ -378,9 +389,14 @@ typedef struct query_target {
389 uint32_t used; // how many items of the array are used
390 uint32_t size; // the size of the array
391 SIMPLE_PATTERN *pattern;
392 + SIMPLE_PATTERN *scope_pattern;
393 SIMPLE_PATTERN *labels_pattern;
394 + SIMPLE_PATTERN *scope_labels_pattern;
395 SIMPLE_PATTERN *alerts_pattern;
396 SIMPLE_PATTERN *chart_label_key_pattern;
397 + SIMPLE_PATTERN *scope_chart_label_key_pattern;
398 + struct pattern_array *labels_pa;
399 + struct pattern_array *scope_labels_pa;
400 } instances;
401
402 struct {
@@ -617,18 +633,18 @@ struct alert_transitions_facets {
633 extern struct alert_transitions_facets alert_transition_facets[];
634
635 struct api_v2_contexts_request {
620 - char *scope_nodes;
621 - char *scope_contexts;
622 - char *nodes;
623 - char *contexts;
624 - char *q;
636 + const char *scope_nodes;
637 + const char *scope_contexts;
638 + const char *nodes;
639 + const char *contexts;
640 + const char *q;
641
642 CONTEXTS_OPTIONS options;
643
644 struct {
645 CONTEXTS_ALERT_STATUS status;
630 - char *alert;
631 - char *transition;
646 + const char *alert;
647 + const char *transition;
648 uint32_t last;
649
650 const char *facets[ATF_TOTAL_ENTRIES];
@@ -638,6 +654,7 @@ struct api_v2_contexts_request {
654 time_t after;
655 time_t before;
656 time_t timeout_ms;
657 + size_t cardinality_limit;
658
659 qt_interrupt_callback_t interrupt_callback;
660 void *interrupt_callback_data;
@@ -661,10 +678,11 @@ typedef enum __attribute__ ((__packed__)) {
678 int rrdcontext_to_json_v2(BUFFER *wb, struct api_v2_contexts_request *req, CONTEXTS_V2_MODE mode);
679
680 RRDCONTEXT_TO_JSON_OPTIONS rrdcontext_to_json_parse_options(char *o);
664 -void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now_s, bool info, bool array);
681 +void buffer_json_agents_v2(BUFFER *wb, struct query_timings *timings, time_t now_s, bool info, bool array, CONTEXTS_OPTIONS options);
682 void buffer_json_node_add_v2(BUFFER *wb, RRDHOST *host, size_t ni, usec_t duration_ut, bool status);
683 void buffer_json_query_timings(BUFFER *wb, const char *key, struct query_timings *timings);
684 void buffer_json_cloud_timings(BUFFER *wb, const char *key, struct query_timings *timings);
685 +void buffer_json_agent_status_id(BUFFER *wb, size_t ai, usec_t duration_ut);
686
687 // ----------------------------------------------------------------------------
688 // scope
@@ -683,9 +701,12 @@ ssize_t query_scope_foreach_context(RRDHOST *host, const char *scope_contexts, S
701
702 typedef ssize_t (*weights_add_metric_t)(void *data, RRDHOST *host, RRDCONTEXT_ACQUIRED *rca, RRDINSTANCE_ACQUIRED *ria, RRDMETRIC_ACQUIRED *rma);
703 ssize_t weights_foreach_rrdmetric_in_context(RRDCONTEXT_ACQUIRED *rca,
704 + SIMPLE_PATTERN *scope_instances_sp,
705 + struct pattern_array *scope_labels_pa,
706 + SIMPLE_PATTERN *scope_dimensions_sp,
707 SIMPLE_PATTERN *instances_sp,
708 SIMPLE_PATTERN *chart_label_key_sp,
688 - SIMPLE_PATTERN *labels_sp,
709 + struct pattern_array *labels_pa,
710 SIMPLE_PATTERN *alerts_sp,
711 SIMPLE_PATTERN *dimensions_sp,
712 bool match_ids, bool match_names,
src/database/rrdhost.c
+1 -1
@@ -9,7 +9,7 @@
9 RRDHOST *localhost = NULL;
10 netdata_rwlock_t rrd_rwlock = NETDATA_RWLOCK_INITIALIZER;
11
12 -RRDHOST *rrdhost_find_by_node_id(char *node_id) {
12 +RRDHOST *rrdhost_find_by_node_id(const char *node_id) {
13
14 ND_UUID node_uuid;
15 if (unlikely(!node_id || uuid_parse(node_id, node_uuid.uuid)))
src/database/rrdhost.h
+5 -2
@@ -357,9 +357,12 @@ extern RRDHOST *localhost;
357 #define rrdhost_sender_replicating_charts_minus_one(host) (__atomic_sub_fetch(&((host)->stream.snd.status.replication.charts), 1, __ATOMIC_RELAXED))
358 #define rrdhost_sender_replicating_charts_zero(host) (__atomic_store_n(&((host)->stream.snd.status.replication.charts), 0, __ATOMIC_RELAXED))
359
360 +#define rrdhost_is_virtual(host) \
361 + rrdhost_option_check(host, RRDHOST_OPTION_VIRTUAL_HOST)
362 +
363 #define rrdhost_is_local(host) ( \
364 (host) == localhost || \
362 - rrdhost_option_check(host, RRDHOST_OPTION_VIRTUAL_HOST) \
365 + rrdhost_is_virtual(host) \
366 )
367
368 #define rrdhost_is_online_flags(flags) ((flags & RRDHOST_FLAG_COLLECTOR_ONLINE) && !(flags & RRDHOST_FLAG_ORPHAN))
@@ -389,7 +392,7 @@ void rrdhost_acquired_release(RRDHOST_ACQUIRED *rha);
392
393 RRDHOST *rrdhost_find_by_hostname(const char *hostname);
394 RRDHOST *rrdhost_find_by_guid(const char *guid);
392 -RRDHOST *rrdhost_find_by_node_id(char *node_id);
395 +RRDHOST *rrdhost_find_by_node_id(const char *node_id);
396
397 #ifdef RRDHOST_INTERNALS
398 RRDHOST *rrdhost_create(
src/database/rrdlabels-aggregated.c new
+237
@@ -0,0 +1,237 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "rrdlabels-aggregated.h"
4 +
5 +// Internal structure for aggregated labels
6 +// Uses JudyL for efficiency: key = STRING* (with reference), value = Pvoid_t to values JudyL
7 +struct rrdlabels_aggregated {
8 + Pvoid_t keys_judy; // JudyL: key=STRING* of label key, value=Pvoid_t to values JudyL
9 +};
10 +
11 +// Create a new aggregated labels structure
12 +RRDLABELS_AGGREGATED *rrdlabels_aggregated_create(void) {
13 + RRDLABELS_AGGREGATED *agg = callocz(1, sizeof(RRDLABELS_AGGREGATED));
14 + agg->keys_judy = (Pvoid_t) NULL;
15 + return agg;
16 +}
17 +
18 +// Destroy aggregated labels structure and free all memory
19 +void rrdlabels_aggregated_destroy(RRDLABELS_AGGREGATED *agg) {
20 + if (!agg) return;
21 +
22 + // Free all nested JudyL arrays and their STRING references
23 + Pvoid_t *PValue;
24 + Word_t key_index = 0;
25 + bool first_then_next = true;
26 +
27 + while ((PValue = JudyLFirstThenNext(agg->keys_judy, &key_index, &first_then_next))) {
28 + STRING *key_string = (STRING *)key_index;
29 + Pvoid_t values_judy = *PValue;
30 +
31 + // Free all value STRING references in the nested JudyL
32 + Word_t value_index = 0;
33 + bool value_first_then_next = true;
34 + Pvoid_t *PValueInner;
35 +
36 + while ((PValueInner = JudyLFirstThenNext(values_judy, &value_index, &value_first_then_next))) {
37 + STRING *value_string = (STRING *)value_index;
38 + string_freez(value_string);
39 + }
40 +
41 + // Free the values JudyL
42 + JudyLFreeArray(&values_judy, PJE0);
43 +
44 + // Free the key STRING reference
45 + string_freez(key_string);
46 + }
47 +
48 + // Free the main JudyL
49 + JudyLFreeArray(&agg->keys_judy, PJE0);
50 +
51 + freez(agg);
52 +}
53 +
54 +// Callback function for adding labels
55 +static int rrdlabels_aggregated_add_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data) {
56 + (void)ls; // unused
57 +
58 + struct { RRDLABELS_AGGREGATED *agg; } *callback_data = data;
59 + RRDLABELS_AGGREGATED *agg = callback_data->agg;
60 +
61 + // Create STRING references with dup for safety
62 + STRING *key_string = string_strdupz(name);
63 + STRING *value_string = string_strdupz(value);
64 +
65 + // Find or create the values JudyL for this key
66 + Pvoid_t *PValue = JudyLIns(&agg->keys_judy, (Word_t)key_string, PJE0);
67 + if (!PValue || PValue == PJERR) {
68 + string_freez(key_string);
69 + string_freez(value_string);
70 + return -1; // Error
71 + }
72 +
73 + Pvoid_t values_judy;
74 + if (!*PValue) {
75 + // New key - create new values JudyL
76 + values_judy = (Pvoid_t) NULL;
77 + *PValue = values_judy;
78 + } else {
79 + // Existing key - free the duplicate key string since we don't need it
80 + string_freez(key_string);
81 + values_judy = *PValue;
82 + }
83 +
84 + // Add the value to the values JudyL
85 + Pvoid_t *PValueInner = JudyLIns(&values_judy, (Word_t)value_string, PJE0);
86 + if (!PValueInner || PValueInner == PJERR) {
87 + string_freez(value_string);
88 + return -1; // Error
89 + }
90 +
91 + if (*PValueInner) {
92 + // Value already exists - free the duplicate
93 + string_freez(value_string);
94 + } else {
95 + // New value - store it
96 + *PValueInner = (Pvoid_t)1; // Just mark as present
97 + }
98 +
99 + // Update the main JudyL with the potentially modified values_judy
100 + *PValue = values_judy;
101 +
102 + return 0; // Continue
103 +}
104 +
105 +// Add all labels from an RRDLABELS instance to the aggregated structure
106 +void rrdlabels_aggregated_add_from_rrdlabels(RRDLABELS_AGGREGATED *agg, RRDLABELS *labels) {
107 + if (!agg || !labels) return;
108 +
109 + // Use the rrdlabels iteration macros from rrdlabels.c
110 + // We need access to the internal structure, so we'll use walkthrough instead
111 +
112 + // Helper structure for the callback
113 + struct {
114 + RRDLABELS_AGGREGATED *agg;
115 + } callback_data = { .agg = agg };
116 +
117 + // Use the existing walkthrough function
118 + rrdlabels_walkthrough_read(labels, rrdlabels_aggregated_add_callback, &callback_data);
119 +}
120 +
121 +// Add a single label key-value pair to the aggregated structure
122 +void rrdlabels_aggregated_add_label(RRDLABELS_AGGREGATED *agg, const char *key, const char *value) {
123 + if (!agg || !key || !value) return;
124 +
125 + // This is essentially the same logic as the callback, but exposed as a public function
126 + rrdlabels_aggregated_add_callback(key, value, RRDLABEL_SRC_AUTO, &(struct { RRDLABELS_AGGREGATED *agg; }){ .agg = agg });
127 +}
128 +
129 +// Output aggregated labels as JSON object with keys and their value arrays
130 +void rrdlabels_aggregated_to_buffer_json(RRDLABELS_AGGREGATED *agg, BUFFER *wb, const char *key, size_t cardinality_limit) {
131 + if (!agg || !wb) return;
132 +
133 + buffer_json_member_add_object(wb, key);
134 +
135 + // Iterate through all keys
136 + Pvoid_t *PValue;
137 + Word_t key_index = 0;
138 + bool first_then_next = true;
139 +
140 + while ((PValue = JudyLFirstThenNext(agg->keys_judy, &key_index, &first_then_next))) {
141 + STRING *key_string = (STRING *)key_index;
142 + Pvoid_t values_judy = *PValue;
143 +
144 + // Add key and its values array
145 + buffer_json_member_add_array(wb, string2str(key_string));
146 +
147 + // Count total values for this key
148 + Word_t total_values = JudyLCount(values_judy, 0, -1, PJE0);
149 +
150 + // Iterate through all values for this key
151 + Word_t value_index = 0;
152 + bool value_first_then_next = true;
153 + Pvoid_t *PValueInner;
154 + size_t count = 0;
155 +
156 + while ((PValueInner = JudyLFirstThenNext(values_judy, &value_index, &value_first_then_next))) {
157 + if(cardinality_limit && count >= cardinality_limit - 1 && total_values > cardinality_limit) {
158 + // Add remaining count message
159 + char msg[100];
160 + snprintf(msg, sizeof(msg), "... %zu values more", total_values - count);
161 + buffer_json_add_array_item_string(wb, msg);
162 + break;
163 + }
164 + STRING *value_string = (STRING *)value_index;
165 + buffer_json_add_array_item_string(wb, string2str(value_string));
166 + count++;
167 + }
168 +
169 + buffer_json_array_close(wb);
170 + }
171 +
172 + buffer_json_object_close(wb);
173 +}
174 +
175 +// Merge all labels from source aggregated structure into destination
176 +void rrdlabels_aggregated_merge(RRDLABELS_AGGREGATED *dst, RRDLABELS_AGGREGATED *src) {
177 + if (!dst || !src) return;
178 +
179 + // Iterate through all keys in source
180 + Pvoid_t *PValue;
181 + Word_t key_index = 0;
182 + bool first_then_next = true;
183 +
184 + while ((PValue = JudyLFirstThenNext(src->keys_judy, &key_index, &first_then_next))) {
185 + STRING *src_key_string = (STRING *)key_index;
186 + Pvoid_t src_values_judy = *PValue;
187 +
188 + // Get or create the destination values JudyL for this key
189 + STRING *dst_key_string = string_dup(src_key_string); // Create a new reference
190 + Pvoid_t *PDstValue = JudyLIns(&dst->keys_judy, (Word_t)dst_key_string, PJE0);
191 +
192 + if (!PDstValue || PDstValue == PJERR) {
193 + string_freez(dst_key_string);
194 + continue; // Skip on error
195 + }
196 +
197 + Pvoid_t dst_values_judy;
198 + if (!*PDstValue) {
199 + // New key in destination
200 + dst_values_judy = (Pvoid_t) NULL;
201 + *PDstValue = dst_values_judy;
202 + } else {
203 + // Key already exists in destination - free the duplicate key
204 + string_freez(dst_key_string);
205 + dst_values_judy = *PDstValue;
206 + }
207 +
208 + // Now merge all values from source to destination
209 + Word_t value_index = 0;
210 + bool value_first_then_next = true;
211 + Pvoid_t *PValueInner;
212 +
213 + while ((PValueInner = JudyLFirstThenNext(src_values_judy, &value_index, &value_first_then_next))) {
214 + STRING *src_value_string = (STRING *)value_index;
215 + STRING *dst_value_string = string_dup(src_value_string); // Create a new reference
216 +
217 + // Insert into destination values
218 + Pvoid_t *PDstValueInner = JudyLIns(&dst_values_judy, (Word_t)dst_value_string, PJE0);
219 +
220 + if (!PDstValueInner || PDstValueInner == PJERR) {
221 + string_freez(dst_value_string);
222 + continue; // Skip on error
223 + }
224 +
225 + if (*PDstValueInner) {
226 + // Value already exists - free the duplicate
227 + string_freez(dst_value_string);
228 + } else {
229 + // New value - mark as present
230 + *PDstValueInner = (Pvoid_t)1;
231 + }
232 + }
233 +
234 + // Update the destination JudyL with the potentially modified values_judy
235 + *PDstValue = dst_values_judy;
236 + }
237 +}
\ No newline at end of file
src/database/rrdlabels-aggregated.h new
+32
@@ -0,0 +1,32 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_RRDLABELS_AGGREGATED_H
4 +#define NETDATA_RRDLABELS_AGGREGATED_H
5 +
6 +#include "libnetdata/libnetdata.h"
7 +#include "rrdlabels.h"
8 +
9 +// Aggregated labels structure for collecting unique keys and their values
10 +// across multiple RRDLABELS instances
11 +struct rrdlabels_aggregated;
12 +typedef struct rrdlabels_aggregated RRDLABELS_AGGREGATED;
13 +
14 +// Create a new aggregated labels structure
15 +RRDLABELS_AGGREGATED *rrdlabels_aggregated_create(void);
16 +
17 +// Destroy aggregated labels structure and free all memory
18 +void rrdlabels_aggregated_destroy(RRDLABELS_AGGREGATED *agg);
19 +
20 +// Add all labels from an RRDLABELS instance to the aggregated structure
21 +void rrdlabels_aggregated_add_from_rrdlabels(RRDLABELS_AGGREGATED *agg, RRDLABELS *labels);
22 +
23 +// Add a single label key-value pair to the aggregated structure
24 +void rrdlabels_aggregated_add_label(RRDLABELS_AGGREGATED *agg, const char *key, const char *value);
25 +
26 +// Output aggregated labels as JSON object with keys and their value arrays
27 +void rrdlabels_aggregated_to_buffer_json(RRDLABELS_AGGREGATED *agg, BUFFER *wb, const char *key, size_t cardinality_limit);
28 +
29 +// Merge all labels from source aggregated structure into destination
30 +void rrdlabels_aggregated_merge(RRDLABELS_AGGREGATED *dst, RRDLABELS_AGGREGATED *src);
31 +
32 +#endif /* NETDATA_RRDLABELS_AGGREGATED_H */
\ No newline at end of file
src/database/rrdlabels.c
+152 -8
@@ -5,6 +5,7 @@
5 // Key OF HS ARRRAY
6
7 struct {
8 + int count;
9 Pvoid_t JudyHS;
10 SPINLOCK spinlock;
11 } global_labels = {
@@ -130,6 +131,7 @@ static RRDLABEL *add_label_name_value(const char *name, const char *value)
131 rrdlabel->label.index = label_index;
132 *PValue = rrdlabel;
133 RRDLABELS_MEMORY_DELTA(&dictionary_stats_category_rrdlabels, judy_mem, sizeof(RRDLABEL_IDX));
134 + global_labels.count++;
135 }
136 __atomic_add_fetch(&rrdlabel->refcount, 1, __ATOMIC_RELAXED);
137
@@ -148,7 +150,9 @@ static void delete_label(RRDLABEL *label)
150 if (refcount == 0) {
151 JudyAllocThreadPulseReset();
152
151 - JudyHSDel(&global_labels.JudyHS, (void *)label, sizeof(*label), PJE0);
153 + int rc = JudyHSDel(&global_labels.JudyHS, (void *)&label->index, sizeof(label->index), PJE0);
154 + if(rc)
155 + global_labels.count--;
156
157 int64_t judy_mem = JudyAllocThreadPulseGetAndReset();
158
@@ -161,6 +165,13 @@ static void delete_label(RRDLABEL *label)
165 spinlock_unlock(&global_labels.spinlock);
166 }
167
168 +int rrdlabels_registry_count(void) {
169 + spinlock_lock(&global_labels.spinlock);
170 + int count = global_labels.count;
171 + spinlock_unlock(&global_labels.spinlock);
172 + return count;
173 +}
174 +
175 // ----------------------------------------------------------------------------
176 // rrdlabels_destroy()
177
@@ -249,7 +260,8 @@ static void labels_add_already_sanitized(RRDLABELS *labels, const char *key, con
260
261 RRDLABEL *old_label_with_same_key = rrdlabels_find_label_with_key_unsafe(labels, new_label, false);
262 if (old_label_with_same_key) {
252 - (void) JudyLDel(&labels->JudyL, (Word_t) old_label_with_same_key, PJE0);
263 + int del_result = JudyLDel(&labels->JudyL, (Word_t) old_label_with_same_key, PJE0);
264 + (void)del_result;
265 judy_mem = JudyAllocThreadPulseGetAndReset();
266 RRDLABELS_MEMORY_DELTA(&dictionary_stats_category_rrdlabels, judy_mem, 0);
267 delete_label(old_label_with_same_key);
@@ -389,6 +401,36 @@ void rrdlabels_key_to_buffer_array_item(RRDLABELS *labels, BUFFER *wb)
401 lfe_done(labels);
402 }
403
404 +void rrdlabels_key_to_buffer_array_or_string_or_null(RRDLABELS *labels, BUFFER *wb)
405 +{
406 + if(!labels) {
407 + buffer_json_add_array_item_string(wb, NULL);
408 + return;
409 + }
410 +
411 + size_t items = rrdlabels_entries(labels);
412 + if(!items) {
413 + buffer_json_add_array_item_string(wb, NULL);
414 + return;
415 + }
416 +
417 + if(items > 1)
418 + buffer_json_add_array_item_array(wb);
419 +
420 + RRDLABEL *lb;
421 + RRDLABEL_SRC ls;
422 + lfe_start_read(labels, lb, ls) {
423 + buffer_json_add_array_item_string(wb, string2str(lb->index.key));
424 + if(items == 1)
425 + break;
426 + }
427 + lfe_done(labels);
428 +
429 + if(items > 1)
430 + buffer_json_array_close(wb);
431 +
432 +}
433 +
434 // ----------------------------------------------------------------------------
435
436 void rrdlabels_get_value_strcpyz(RRDLABELS *labels, char *dst, size_t dst_len, const char *key) {
@@ -538,6 +580,25 @@ int rrdlabels_walkthrough_read(RRDLABELS *labels, int (*callback)(const char *na
580 return ret;
581 }
582
583 +int rrdlabels_walkthrough_read_string(RRDLABELS *labels, int (*callback)(STRING *name, STRING *value, RRDLABEL_SRC ls, void *data), void *data)
584 +{
585 + int ret = 0;
586 +
587 + if(unlikely(!labels || !callback)) return 0;
588 +
589 + RRDLABEL *lb;
590 + RRDLABEL_SRC ls;
591 + lfe_start_read(labels, lb, ls)
592 + {
593 + ret = callback(lb->index.key, lb->index.value, ls, data);
594 + if (ret < 0)
595 + break;
596 + }
597 + lfe_done(labels);
598 +
599 + return ret;
600 +}
601 +
602 static SIMPLE_PATTERN_RESULT rrdlabels_walkthrough_read_sp(RRDLABELS *labels, SIMPLE_PATTERN_RESULT (*callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *data)
603 {
604 SIMPLE_PATTERN_RESULT ret = SP_NOT_MATCHED;
@@ -1191,6 +1252,73 @@ static int rrdlabels_unittest_pattern_check()
1252 return rc;
1253 }
1254
1255 +// --------------------------------------------------------------------------------------------------------------------
1256 +// Full text search for labels
1257 +
1258 +#include "rrdlabels-aggregated.h"
1259 +
1260 +struct label_full_text_search_data {
1261 + SIMPLE_PATTERN *pattern;
1262 + RRDLABELS_AGGREGATED *agg;
1263 + size_t searches;
1264 + bool found;
1265 +};
1266 +
1267 +static int label_full_text_search_callback_string(STRING *name, STRING *value, RRDLABEL_SRC ls __maybe_unused, void *data) {
1268 + struct label_full_text_search_data *d = (struct label_full_text_search_data *)data;
1269 +
1270 + // Now we have STRING* directly, no need to create them
1271 + bool key_matches = false;
1272 + bool value_matches = false;
1273 +
1274 + // First try to match the key
1275 + if(simple_pattern_matches_string(d->pattern, name)) {
1276 + key_matches = true;
1277 + d->searches++;
1278 + }
1279 +
1280 + // Then try to match the value
1281 + if(simple_pattern_matches_string(d->pattern, value)) {
1282 + value_matches = true;
1283 + d->searches++;
1284 + }
1285 +
1286 + // If either matched, add this label to the aggregated structure
1287 + if(key_matches || value_matches) {
1288 + if(!d->agg) {
1289 + d->agg = rrdlabels_aggregated_create();
1290 + }
1291 + rrdlabels_aggregated_add_label(d->agg, string2str(name), string2str(value));
1292 + d->found = true;
1293 + }
1294 +
1295 + return 0; // Continue walking through labels
1296 +}
1297 +
1298 +RRDLABELS_AGGREGATED *rrdlabels_full_text_search(RRDLABELS *labels, SIMPLE_PATTERN *pattern, RRDLABELS_AGGREGATED *agg, size_t *searches) {
1299 + if(!labels || !pattern) return agg;
1300 +
1301 + struct label_full_text_search_data data = {
1302 + .pattern = pattern,
1303 + .agg = agg,
1304 + .searches = 0,
1305 + .found = false
1306 + };
1307 +
1308 + rrdlabels_walkthrough_read_string(labels, label_full_text_search_callback_string, &data);
1309 +
1310 + if(searches)
1311 + *searches = data.searches;
1312 +
1313 + // If we found matches and didn't have an agg structure, one was created
1314 + // If we had an agg structure, we added to it (or not if no matches)
1315 + // If no matches and no initial agg, data.agg is still NULL
1316 + return data.agg;
1317 +}
1318 +
1319 +// --------------------------------------------------------------------------------------------------------------------
1320 +// unittest
1321 +
1322 static int rrdlabels_unittest_migrate_check()
1323 {
1324 fprintf(stderr, "\n%s() tests\n", __FUNCTION__);
@@ -1215,8 +1343,11 @@ static int rrdlabels_unittest_migrate_check()
1343
1344 int rc = 0;
1345 rc = rrdlabels_unittest_expect_value(labels1, "key1", "value2", RRDLABEL_FLAG_OLD | RRDLABEL_SRC_CONFIG);
1218 - if (rc)
1346 + if (rc) {
1347 + rrdlabels_destroy(labels1);
1348 + rrdlabels_destroy(labels2);
1349 return rc;
1350 + }
1351
1352 fprintf(stderr, "labels1 (migrated) entries found %zu (should be 3)\n", rrdlabels_entries(labels1));
1353 size_t entries = rrdlabels_entries(labels1);
@@ -1242,26 +1373,38 @@ static int rrdlabels_unittest_migrate_check()
1373 rrdlabels_add(labels2, "key2", "value2", RRDLABEL_SRC_CONFIG);
1374
1375 rc = rrdlabels_unittest_expect_value(labels1, "key1", "value1", RRDLABEL_FLAG_NEW | RRDLABEL_SRC_CONFIG);
1245 - if (rc)
1376 + if (rc) {
1377 + rrdlabels_destroy(labels1);
1378 + rrdlabels_destroy(labels2);
1379 return rc;
1380 + }
1381
1382 rrdlabels_walkthrough_index_read(labels2, unittest_dump_labels, "\nlabels2");
1383
1384 rrdlabels_copy(labels1, labels2); // labels1 should have 5 keys
1385 rc = rrdlabels_unittest_expect_value(labels1, "key1", "value1", RRDLABEL_FLAG_OLD | RRDLABEL_SRC_CONFIG);
1252 - if (rc)
1386 + if (rc) {
1387 + rrdlabels_destroy(labels1);
1388 + rrdlabels_destroy(labels2);
1389 return rc;
1390 + }
1391
1392 rc = rrdlabels_unittest_expect_value(labels1, "key0", "value0", RRDLABEL_FLAG_NEW | RRDLABEL_SRC_CONFIG);
1256 - if (rc)
1393 + if (rc) {
1394 + rrdlabels_destroy(labels1);
1395 + rrdlabels_destroy(labels2);
1396 return rc;
1397 + }
1398
1399 rrdlabels_walkthrough_index_read(labels1, unittest_dump_labels, "\nlabels1 after copy from labels2");
1400 entries = rrdlabels_entries(labels1);
1401
1402 fprintf(stderr, "labels1 (copied) entries found %zu (should be 5)\n", rrdlabels_entries(labels1));
1263 - if (entries != 5)
1403 + if (entries != 5) {
1404 + rrdlabels_destroy(labels1);
1405 + rrdlabels_destroy(labels2);
1406 return 1;
1407 + }
1408
1409 rrdlabels_add(labels1, "key0", "value0", RRDLABEL_SRC_CONFIG);
1410 rc = rrdlabels_unittest_expect_value(labels1, "key0", "value0", RRDLABEL_FLAG_OLD | RRDLABEL_SRC_CONFIG);
@@ -1457,6 +1600,7 @@ int rrdlabels_unittest(void) {
1600 errors += rrdlabels_unittest_pattern_check();
1601
1602 fprintf(stderr, "%d errors found\n", errors);
1603 +
1604 + // string_destroy();
1605 return errors;
1606 }
1462 -
src/database/rrdlabels.h
+12
@@ -33,6 +33,7 @@ void rrdlabels_add(RRDLABELS *labels, const char *name, const char *value, RRDLA
33 void rrdlabels_add_pair(RRDLABELS *labels, const char *string, RRDLABEL_SRC ls);
34 void rrdlabels_value_to_buffer_array_item_or_null(RRDLABELS *labels, BUFFER *wb, const char *key);
35 void rrdlabels_key_to_buffer_array_item(RRDLABELS *labels, BUFFER *wb);
36 +void rrdlabels_key_to_buffer_array_or_string_or_null(RRDLABELS *labels, BUFFER *wb);
37 void rrdlabels_get_value_strdup_or_null(RRDLABELS *labels, char **value, const char *key);
38 void rrdlabels_get_value_to_buffer_or_unset(RRDLABELS *labels, BUFFER *wb, const char *key, const char *unset);
39 bool rrdlabels_exist(RRDLABELS *labels, const char *key);
@@ -44,10 +45,19 @@ void rrdlabels_unmark_all(RRDLABELS *labels);
45 void rrdlabels_remove_all_unmarked(RRDLABELS *labels);
46
47 int rrdlabels_walkthrough_read(RRDLABELS *labels, int (*callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *data);
48 +int rrdlabels_walkthrough_read_string(RRDLABELS *labels, int (*callback)(STRING *name, STRING *value, RRDLABEL_SRC ls, void *data), void *data);
49 void rrdlabels_log_to_buffer(RRDLABELS *labels, BUFFER *wb);
50 bool rrdlabels_match_simple_pattern(RRDLABELS *labels, const char *simple_pattern_txt);
51
52 SIMPLE_PATTERN_RESULT rrdlabels_match_simple_pattern_parsed(RRDLABELS *labels, SIMPLE_PATTERN *pattern, char equal, size_t *searches);
53 +
54 +// Forward declaration for RRDLABELS_AGGREGATED
55 +struct rrdlabels_aggregated;
56 +// Full text search through labels - matches if either key OR value matches the pattern
57 +// If agg is NULL and matches are found, a new aggregated structure is created and returned
58 +// If agg is not NULL, matches are added to it and it is returned
59 +// Returns NULL only if no matches found and agg was NULL
60 +struct rrdlabels_aggregated *rrdlabels_full_text_search(RRDLABELS *labels, SIMPLE_PATTERN *pattern, struct rrdlabels_aggregated *agg, size_t *searches);
61 int rrdlabels_to_buffer(RRDLABELS *labels, BUFFER *wb, const char *before_each, const char *equal, const char *quote, const char *between_them,
62 bool (*filter_callback)(const char *name, const char *value, RRDLABEL_SRC ls, void *data), void *filter_data,
63 void (*name_sanitizer)(char *dst, const char *src, size_t dst_size),
@@ -64,4 +74,6 @@ size_t rrdlabels_sanitize_name(char *dst, const char *src, size_t dst_size);
74 // unfortunately this break when defined in exporting_engine.h
75 bool exporting_labels_filter_callback(const char *name, const char *value, RRDLABEL_SRC ls, void *data);
76
77 +int rrdlabels_registry_count(void);
78 +
79 #endif /* NETDATA_RRDLABELS_H */
src/libnetdata/buffer/buffer.c
+48
@@ -125,6 +125,42 @@ void buffer_sprintf(BUFFER *wb, const char *fmt, ...)
125 va_end(args);
126 }
127
128 +void buffer_json_member_add_sprintf(BUFFER *wb, const char *key, const char *fmt, ...)
129 +{
130 + va_list args;
131 +
132 + // Create a temporary buffer for the formatted string
133 + BUFFER *tmp = buffer_create(0, NULL);
134 +
135 + va_start(args, fmt);
136 + buffer_vsprintf(tmp, fmt, args);
137 + va_end(args);
138 +
139 + // Add as JSON member (which will handle escaping)
140 + buffer_json_member_add_string(wb, key, buffer_tostring(tmp));
141 +
142 + // Free the temporary buffer
143 + buffer_free(tmp);
144 +}
145 +
146 +void buffer_json_add_array_item_sprintf(BUFFER *wb, const char *fmt, ...)
147 +{
148 + va_list args;
149 +
150 + // Create a temporary buffer for the formatted string
151 + BUFFER *tmp = buffer_create(0, NULL);
152 +
153 + va_start(args, fmt);
154 + buffer_vsprintf(tmp, fmt, args);
155 + va_end(args);
156 +
157 + // Add as array item (which will handle escaping)
158 + buffer_json_add_array_item_string(wb, buffer_tostring(tmp));
159 +
160 + // Free the temporary buffer
161 + buffer_free(tmp);
162 +}
163 +
164 // generate a javascript date, the fastest possible way...
165 void buffer_jsdate(BUFFER *wb, int year, int month, int day, int hours, int minutes, int seconds)
166 {
@@ -335,9 +371,15 @@ void buffer_json_finalize(BUFFER *wb) {
371
372 // ----------------------------------------------------------------------------
373
374 +__attribute__((nonstring))
375 const char hex_digits[16] = "0123456789ABCDEF";
376 +
377 +__attribute__((nonstring))
378 const char hex_digits_lower[16] = "0123456789abcdef";
379 +
380 +__attribute__((nonstring))
381 const char base64_digits[64] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
382 +
383 unsigned char hex_value_from_ascii[256];
384 unsigned char base64_value_from_ascii[256];
385
@@ -370,6 +412,12 @@ void buffer_json_member_add_duration_ut(BUFFER *wb, const char *key, int64_t dur
412 buffer_json_member_add_string(wb, key, buf);
413 }
414
415 +void buffer_json_add_array_item_datetime_rfc3339(BUFFER *wb, uint64_t datetime_ut, bool utc) {
416 + char buf[RFC3339_MAX_LENGTH];
417 + rfc3339_datetime_ut(buf, sizeof(buf), datetime_ut, 2, utc);
418 + buffer_json_add_array_item_string(wb, buf);
419 +}
420 +
421 // ----------------------------------------------------------------------------
422 // unit test
423
src/libnetdata/buffer/buffer.h
+89 -281
@@ -5,9 +5,12 @@
5
6 #include "../uuid/uuid.h"
7 #include "../http/content_type.h"
8 +#include "../clocks/clocks.h"
9 #include "../string/utf8.h"
10 #include "../libnetdata.h"
11
12 +#define API_RELATIVE_TIME_MAX (time_t)(3 * 365 * 86400)
13 +
14 #define BUFFER_JSON_MAX_DEPTH 32 // max is 255
15
16 // gcc with libstdc++ may require this,
@@ -120,6 +123,8 @@ static inline void buffer_freep(BUFFER **bp) {
123 void buffer_snprintf(BUFFER *wb, size_t len, const char *fmt, ...) PRINTFLIKE(3, 4);
124 void buffer_vsprintf(BUFFER *wb, const char *fmt, va_list args);
125 void buffer_sprintf(BUFFER *wb, const char *fmt, ...) PRINTFLIKE(2,3);
126 +void buffer_json_member_add_sprintf(BUFFER *wb, const char *key, const char *fmt, ...) PRINTFLIKE(3,4);
127 +void buffer_json_add_array_item_sprintf(BUFFER *wb, const char *fmt, ...) PRINTFLIKE(2,3);
128 void buffer_strcat_htmlescape(BUFFER *wb, const char *txt);
129
130 void buffer_char_replace(BUFFER *wb, char from, char to);
@@ -821,7 +826,8 @@ static void buffer_json_add_quoted_string_value(BUFFER *wb, const char *value) {
826 buffer_fast_strcat(wb, "null", 4);
827 }
828
824 -static inline void buffer_json_member_add_object(BUFFER *wb, const char *key) {
829 +ALWAYS_INLINE
830 +static void buffer_json_member_add_object(BUFFER *wb, const char *key) {
831 buffer_print_json_comma_newline_spacing(wb);
832 buffer_print_json_key(wb, key);
833 buffer_fast_strcat(wb, ":{", 2);
@@ -830,7 +836,8 @@ static inline void buffer_json_member_add_object(BUFFER *wb, const char *key) {
836 _buffer_json_depth_push(wb, BUFFER_JSON_OBJECT);
837 }
838
833 -static inline void buffer_json_object_close(BUFFER *wb) {
839 +ALWAYS_INLINE
840 +static void buffer_json_object_close(BUFFER *wb) {
841 #ifdef NETDATA_INTERNAL_CHECKS
842 assert(wb->json.depth >= 0 && "BUFFER JSON: nothing is open to close it");
843 assert(wb->json.stack[wb->json.depth].type == BUFFER_JSON_OBJECT && "BUFFER JSON: an object is not open to close it");
@@ -843,7 +850,8 @@ static inline void buffer_json_object_close(BUFFER *wb) {
850 _buffer_json_depth_pop(wb);
851 }
852
846 -static inline void buffer_json_member_add_string(BUFFER *wb, const char *key, const char *value) {
853 +ALWAYS_INLINE
854 +static void buffer_json_member_add_string(BUFFER *wb, const char *key, const char *value) {
855 buffer_print_json_comma_newline_spacing(wb);
856 buffer_print_json_key(wb, key);
857 buffer_putc(wb, ':');
@@ -852,12 +860,14 @@ static inline void buffer_json_member_add_string(BUFFER *wb, const char *key, co
860 wb->json.stack[wb->json.depth].count++;
861 }
862
855 -static inline void buffer_json_member_add_string_or_omit(BUFFER *wb, const char *key, const char *value) {
863 +ALWAYS_INLINE
864 +static void buffer_json_member_add_string_or_omit(BUFFER *wb, const char *key, const char *value) {
865 if(value && *value)
866 buffer_json_member_add_string(wb, key, value);
867 }
868
860 -static inline void buffer_json_member_add_string_or_empty(BUFFER *wb, const char *key, const char *value) {
869 +ALWAYS_INLINE
870 +static void buffer_json_member_add_string_or_empty(BUFFER *wb, const char *key, const char *value) {
871 if(!value)
872 value = "";
873
@@ -867,7 +877,8 @@ static inline void buffer_json_member_add_string_or_empty(BUFFER *wb, const char
877 void buffer_json_member_add_datetime_rfc3339(BUFFER *wb, const char *key, uint64_t datetime_ut, bool utc);
878 void buffer_json_member_add_duration_ut(BUFFER *wb, const char *key, int64_t duration_ut);
879
870 -static inline void buffer_json_member_add_quoted_string(BUFFER *wb, const char *key, const char *value) {
880 +ALWAYS_INLINE
881 +static void buffer_json_member_add_quoted_string(BUFFER *wb, const char *key, const char *value) {
882 buffer_print_json_comma_newline_spacing(wb);
883 buffer_print_json_key(wb, key);
884 buffer_putc(wb, ':');
@@ -880,7 +891,8 @@ static inline void buffer_json_member_add_quoted_string(BUFFER *wb, const char *
891 wb->json.stack[wb->json.depth].count++;
892 }
893
883 -static inline void buffer_json_member_add_uuid_ptr(BUFFER *wb, const char *key, nd_uuid_t *value) {
894 +ALWAYS_INLINE
895 +static void buffer_json_member_add_uuid_ptr(BUFFER *wb, const char *key, nd_uuid_t *value) {
896 buffer_print_json_comma_newline_spacing(wb);
897 buffer_print_json_key(wb, key);
898 buffer_putc(wb, ':');
@@ -896,7 +908,8 @@ static inline void buffer_json_member_add_uuid_ptr(BUFFER *wb, const char *key,
908 wb->json.stack[wb->json.depth].count++;
909 }
910
899 -static inline void buffer_json_member_add_uuid(BUFFER *wb, const char *key, nd_uuid_t value) {
911 +ALWAYS_INLINE
912 +static void buffer_json_member_add_uuid(BUFFER *wb, const char *key, nd_uuid_t value) {
913 buffer_print_json_comma_newline_spacing(wb);
914 buffer_print_json_key(wb, key);
915 buffer_putc(wb, ':');
@@ -912,7 +925,8 @@ static inline void buffer_json_member_add_uuid(BUFFER *wb, const char *key, nd_u
925 wb->json.stack[wb->json.depth].count++;
926 }
927
915 -static inline void buffer_json_member_add_uuid_compact(BUFFER *wb, const char *key, nd_uuid_t value) {
928 +ALWAYS_INLINE
929 +static void buffer_json_member_add_uuid_compact(BUFFER *wb, const char *key, nd_uuid_t value) {
930 buffer_print_json_comma_newline_spacing(wb);
931 buffer_print_json_key(wb, key);
932 buffer_putc(wb, ':');
@@ -928,7 +942,8 @@ static inline void buffer_json_member_add_uuid_compact(BUFFER *wb, const char *k
942 wb->json.stack[wb->json.depth].count++;
943 }
944
931 -static inline void buffer_json_member_add_boolean(BUFFER *wb, const char *key, bool value) {
945 +ALWAYS_INLINE
946 +static void buffer_json_member_add_boolean(BUFFER *wb, const char *key, bool value) {
947 buffer_print_json_comma_newline_spacing(wb);
948 buffer_print_json_key(wb, key);
949 buffer_putc(wb, ':');
@@ -937,7 +952,8 @@ static inline void buffer_json_member_add_boolean(BUFFER *wb, const char *key, b
952 wb->json.stack[wb->json.depth].count++;
953 }
954
940 -static inline void buffer_json_member_add_array(BUFFER *wb, const char *key) {
955 +ALWAYS_INLINE
956 +static void buffer_json_member_add_array(BUFFER *wb, const char *key) {
957 buffer_print_json_comma_newline_spacing(wb);
958 if (key) {
959 buffer_print_json_key(wb, key);
@@ -951,7 +967,8 @@ static inline void buffer_json_member_add_array(BUFFER *wb, const char *key) {
967 _buffer_json_depth_push(wb, BUFFER_JSON_ARRAY);
968 }
969
954 -static inline void buffer_json_add_array_item_array(BUFFER *wb) {
970 +ALWAYS_INLINE
971 +static void buffer_json_add_array_item_array(BUFFER *wb) {
972 if(!(wb->json.options & BUFFER_JSON_OPTIONS_MINIFY) && wb->json.stack[wb->json.depth].type == BUFFER_JSON_ARRAY) {
973 // an array inside another array
974 buffer_print_json_comma(wb);
@@ -967,14 +984,16 @@ static inline void buffer_json_add_array_item_array(BUFFER *wb) {
984 _buffer_json_depth_push(wb, BUFFER_JSON_ARRAY);
985 }
986
970 -static inline void buffer_json_add_array_item_string(BUFFER *wb, const char *value) {
987 +ALWAYS_INLINE
988 +static void buffer_json_add_array_item_string(BUFFER *wb, const char *value) {
989 buffer_print_json_comma_newline_spacing(wb);
990
991 buffer_json_add_string_value(wb, value);
992 wb->json.stack[wb->json.depth].count++;
993 }
994
977 -static inline void buffer_json_add_array_item_uuid(BUFFER *wb, nd_uuid_t *value) {
995 +ALWAYS_INLINE
996 +static void buffer_json_add_array_item_uuid(BUFFER *wb, nd_uuid_t *value) {
997 if(value && !uuid_is_null(*value)) {
998 char uuid[GUID_LEN + 1];
999 uuid_unparse_lower(*value, uuid);
@@ -984,7 +1003,8 @@ static inline void buffer_json_add_array_item_uuid(BUFFER *wb, nd_uuid_t *value)
1003 buffer_json_add_array_item_string(wb, NULL);
1004 }
1005
987 -static inline void buffer_json_add_array_item_uuid_compact(BUFFER *wb, nd_uuid_t *value) {
1006 +ALWAYS_INLINE
1007 +static void buffer_json_add_array_item_uuid_compact(BUFFER *wb, nd_uuid_t *value) {
1008 if(value && !uuid_is_null(*value)) {
1009 char uuid[GUID_LEN + 1];
1010 uuid_unparse_lower_compact(*value, uuid);
@@ -994,42 +1014,48 @@ static inline void buffer_json_add_array_item_uuid_compact(BUFFER *wb, nd_uuid_t
1014 buffer_json_add_array_item_string(wb, NULL);
1015 }
1016
997 -static inline void buffer_json_add_array_item_double(BUFFER *wb, NETDATA_DOUBLE value) {
1017 +ALWAYS_INLINE
1018 +static void buffer_json_add_array_item_double(BUFFER *wb, NETDATA_DOUBLE value) {
1019 buffer_print_json_comma_newline_spacing(wb);
1020
1021 buffer_print_netdata_double(wb, value);
1022 wb->json.stack[wb->json.depth].count++;
1023 }
1024
1004 -static inline void buffer_json_add_array_item_int64(BUFFER *wb, int64_t value) {
1025 +ALWAYS_INLINE
1026 +static void buffer_json_add_array_item_int64(BUFFER *wb, int64_t value) {
1027 buffer_print_json_comma_newline_spacing(wb);
1028
1029 buffer_print_int64(wb, value);
1030 wb->json.stack[wb->json.depth].count++;
1031 }
1032
1011 -static inline void buffer_json_add_array_item_uint64(BUFFER *wb, uint64_t value) {
1033 +ALWAYS_INLINE
1034 +static void buffer_json_add_array_item_uint64(BUFFER *wb, uint64_t value) {
1035 buffer_print_json_comma_newline_spacing(wb);
1036
1037 buffer_print_uint64(wb, value);
1038 wb->json.stack[wb->json.depth].count++;
1039 }
1040
1018 -static inline void buffer_json_add_array_item_boolean(BUFFER *wb, bool value) {
1041 +ALWAYS_INLINE
1042 +static void buffer_json_add_array_item_boolean(BUFFER *wb, bool value) {
1043 buffer_print_json_comma_newline_spacing(wb);
1044
1045 buffer_strcat(wb, value ? "true" : "false");
1046 wb->json.stack[wb->json.depth].count++;
1047 }
1048
1025 -static inline void buffer_json_add_array_item_time_t(BUFFER *wb, time_t value) {
1049 +ALWAYS_INLINE
1050 +static void buffer_json_add_array_item_time_t(BUFFER *wb, time_t value) {
1051 buffer_print_json_comma_newline_spacing(wb);
1052
1053 buffer_print_int64(wb, value);
1054 wb->json.stack[wb->json.depth].count++;
1055 }
1056
1032 -static inline void buffer_json_add_array_item_time_ms(BUFFER *wb, time_t value) {
1057 +ALWAYS_INLINE
1058 +static void buffer_json_add_array_item_time_ms(BUFFER *wb, time_t value) {
1059 buffer_print_json_comma_newline_spacing(wb);
1060
1061 buffer_print_int64(wb, value);
@@ -1037,15 +1063,22 @@ static inline void buffer_json_add_array_item_time_ms(BUFFER *wb, time_t value)
1063 wb->json.stack[wb->json.depth].count++;
1064 }
1065
1040 -static inline void buffer_json_add_array_item_time_t2ms(BUFFER *wb, time_t value) {
1041 - buffer_print_json_comma_newline_spacing(wb);
1066 +void buffer_json_add_array_item_datetime_rfc3339(BUFFER *wb, uint64_t datetime_ut, bool utc);
1067
1043 - buffer_print_int64(wb, value);
1044 - buffer_fast_strcat(wb, "000", 3);
1045 - wb->json.stack[wb->json.depth].count++;
1068 +ALWAYS_INLINE
1069 +static void buffer_json_add_array_item_time_t_formatted(BUFFER *wb, time_t value, bool rfc3339) {
1070 + if(unlikely(rfc3339 && (!value || value > API_RELATIVE_TIME_MAX))) {
1071 + if (!value)
1072 + buffer_json_add_array_item_string(wb, NULL);
1073 + else
1074 + buffer_json_add_array_item_datetime_rfc3339(wb, (uint64_t)value * USEC_PER_SEC, true);
1075 + }
1076 + else
1077 + buffer_json_add_array_item_time_t(wb, value);
1078 }
1079
1048 -static inline void buffer_json_add_array_item_object(BUFFER *wb) {
1080 +ALWAYS_INLINE
1081 +static void buffer_json_add_array_item_object(BUFFER *wb) {
1082 buffer_print_json_comma_newline_spacing(wb);
1083
1084 buffer_putc(wb, '{');
@@ -1054,7 +1087,8 @@ static inline void buffer_json_add_array_item_object(BUFFER *wb) {
1087 _buffer_json_depth_push(wb, BUFFER_JSON_OBJECT);
1088 }
1089
1057 -static inline void buffer_json_member_add_time_t(BUFFER *wb, const char *key, time_t value) {
1090 +ALWAYS_INLINE
1091 +static void buffer_json_member_add_time_t(BUFFER *wb, const char *key, time_t value) {
1092 buffer_print_json_comma_newline_spacing(wb);
1093 buffer_print_json_key(wb, key);
1094 buffer_putc(wb, ':');
@@ -1063,17 +1097,20 @@ static inline void buffer_json_member_add_time_t(BUFFER *wb, const char *key, ti
1097 wb->json.stack[wb->json.depth].count++;
1098 }
1099
1066 -static inline void buffer_json_member_add_time_t2ms(BUFFER *wb, const char *key, time_t value) {
1067 - buffer_print_json_comma_newline_spacing(wb);
1068 - buffer_print_json_key(wb, key);
1069 - buffer_putc(wb, ':');
1070 - buffer_print_int64(wb, value);
1071 - buffer_fast_strcat(wb, "000", 3);
1072 -
1073 - wb->json.stack[wb->json.depth].count++;
1100 +ALWAYS_INLINE
1101 +static void buffer_json_member_add_time_t_formatted(BUFFER *wb, const char *key, time_t value, bool rfc3339) {
1102 + if(unlikely(rfc3339 && (!value || value > API_RELATIVE_TIME_MAX))) {
1103 + if(!value)
1104 + buffer_json_member_add_string(wb, key, NULL);
1105 + else
1106 + buffer_json_member_add_datetime_rfc3339(wb, key, (uint64_t)value * USEC_PER_SEC, true);
1107 + }
1108 + else
1109 + buffer_json_member_add_time_t(wb, key, value);
1110 }
1111
1076 -static inline void buffer_json_member_add_uint64(BUFFER *wb, const char *key, uint64_t value) {
1112 +ALWAYS_INLINE
1113 +static void buffer_json_member_add_uint64(BUFFER *wb, const char *key, uint64_t value) {
1114 buffer_print_json_comma_newline_spacing(wb);
1115 buffer_print_json_key(wb, key);
1116 buffer_putc(wb, ':');
@@ -1082,7 +1119,8 @@ static inline void buffer_json_member_add_uint64(BUFFER *wb, const char *key, ui
1119 wb->json.stack[wb->json.depth].count++;
1120 }
1121
1085 -static inline void buffer_json_member_add_int64(BUFFER *wb, const char *key, int64_t value) {
1122 +ALWAYS_INLINE
1123 +static void buffer_json_member_add_int64(BUFFER *wb, const char *key, int64_t value) {
1124 buffer_print_json_comma_newline_spacing(wb);
1125 buffer_print_json_key(wb, key);
1126 buffer_putc(wb, ':');
@@ -1091,7 +1129,8 @@ static inline void buffer_json_member_add_int64(BUFFER *wb, const char *key, int
1129 wb->json.stack[wb->json.depth].count++;
1130 }
1131
1094 -static inline void buffer_json_member_add_double(BUFFER *wb, const char *key, NETDATA_DOUBLE value) {
1132 +ALWAYS_INLINE
1133 +static void buffer_json_member_add_double(BUFFER *wb, const char *key, NETDATA_DOUBLE value) {
1134 buffer_print_json_comma_newline_spacing(wb);
1135 buffer_print_json_key(wb, key);
1136 buffer_putc(wb, ':');
@@ -1100,7 +1139,8 @@ static inline void buffer_json_member_add_double(BUFFER *wb, const char *key, NE
1139 wb->json.stack[wb->json.depth].count++;
1140 }
1141
1103 -static inline void buffer_json_array_close(BUFFER *wb) {
1142 +ALWAYS_INLINE
1143 +static void buffer_json_array_close(BUFFER *wb) {
1144 #ifdef NETDATA_INTERNAL_CHECKS
1145 assert(wb->json.depth >= 0 && "BUFFER JSON: nothing is open to close it");
1146 assert(wb->json.stack[wb->json.depth].type == BUFFER_JSON_ARRAY && "BUFFER JSON: an array is not open to close it");
@@ -1114,245 +1154,8 @@ static inline void buffer_json_array_close(BUFFER *wb) {
1154 _buffer_json_depth_pop(wb);
1155 }
1156
1117 -typedef enum __attribute__((packed)) {
1118 - RRDF_FIELD_OPTS_NONE = 0,
1119 - RRDF_FIELD_OPTS_UNIQUE_KEY = (1 << 0), // the field is the unique key of the row
1120 - RRDF_FIELD_OPTS_VISIBLE = (1 << 1), // the field should be visible by default
1121 - RRDF_FIELD_OPTS_STICKY = (1 << 2), // the field should be sticky
1122 - RRDF_FIELD_OPTS_FULL_WIDTH = (1 << 3), // the field should get full width
1123 - RRDF_FIELD_OPTS_WRAP = (1 << 4), // the field should wrap
1124 - RRDF_FIELD_OPTS_DUMMY = (1 << 5), // not a presentable field
1125 - RRDF_FIELD_OPTS_EXPANDED_FILTER = (1 << 6), // show the filter expanded
1126 -} RRDF_FIELD_OPTIONS;
1127 -
1128 -typedef enum __attribute__((packed)) {
1129 - RRDF_FIELD_TYPE_NONE,
1130 - RRDF_FIELD_TYPE_INTEGER,
1131 - RRDF_FIELD_TYPE_BOOLEAN,
1132 - RRDF_FIELD_TYPE_STRING,
1133 - RRDF_FIELD_TYPE_DETAIL_STRING,
1134 - RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
1135 - RRDF_FIELD_TYPE_DURATION,
1136 - RRDF_FIELD_TYPE_TIMESTAMP,
1137 - RRDF_FIELD_TYPE_ARRAY,
1138 -} RRDF_FIELD_TYPE;
1139 -
1140 -static inline const char *rrdf_field_type_to_string(RRDF_FIELD_TYPE type) {
1141 - switch(type) {
1142 - default:
1143 - case RRDF_FIELD_TYPE_NONE:
1144 - return "none";
1145 -
1146 - case RRDF_FIELD_TYPE_INTEGER:
1147 - return "integer";
1148 -
1149 - case RRDF_FIELD_TYPE_BOOLEAN:
1150 - return "boolean";
1151 -
1152 - case RRDF_FIELD_TYPE_STRING:
1153 - return "string";
1154 -
1155 - case RRDF_FIELD_TYPE_DETAIL_STRING:
1156 - return "detail-string";
1157 -
1158 - case RRDF_FIELD_TYPE_BAR_WITH_INTEGER:
1159 - return "bar-with-integer";
1160 -
1161 - case RRDF_FIELD_TYPE_DURATION:
1162 - return "duration";
1163 -
1164 - case RRDF_FIELD_TYPE_TIMESTAMP:
1165 - return "timestamp";
1166 -
1167 - case RRDF_FIELD_TYPE_ARRAY:
1168 - return "array";
1169 - }
1170 -}
1171 -
1172 -typedef enum __attribute__((packed)) {
1173 - RRDF_FIELD_VISUAL_VALUE, // show the value, possibly applying a transformation
1174 - RRDF_FIELD_VISUAL_BAR, // show the value and a bar, respecting the max field to fill the bar at 100%
1175 - RRDF_FIELD_VISUAL_PILL, //
1176 - RRDF_FIELD_VISUAL_RICH, //
1177 - RRDR_FIELD_VISUAL_ROW_OPTIONS, // this is a dummy column that is used for row options
1178 -} RRDF_FIELD_VISUAL;
1179 -
1180 -static inline const char *rrdf_field_visual_to_string(RRDF_FIELD_VISUAL visual) {
1181 - switch(visual) {
1182 - default:
1183 - case RRDF_FIELD_VISUAL_VALUE:
1184 - return "value";
1185 -
1186 - case RRDF_FIELD_VISUAL_BAR:
1187 - return "bar";
1188 -
1189 - case RRDF_FIELD_VISUAL_PILL:
1190 - return "pill";
1191 -
1192 - case RRDF_FIELD_VISUAL_RICH:
1193 - return "richValue";
1194 -
1195 - case RRDR_FIELD_VISUAL_ROW_OPTIONS:
1196 - return "rowOptions";
1197 - }
1198 -}
1199 -
1200 -typedef enum __attribute__((packed)) {
1201 - RRDF_FIELD_TRANSFORM_NONE, // show the value as-is
1202 - RRDF_FIELD_TRANSFORM_NUMBER, // show the value respecting the decimal_points
1203 - RRDF_FIELD_TRANSFORM_DURATION_S, // transform as duration in second to a human-readable duration
1204 - RRDF_FIELD_TRANSFORM_DATETIME_MS, // UNIX epoch timestamp in ms
1205 - RRDF_FIELD_TRANSFORM_DATETIME_USEC, // UNIX epoch timestamp in usec
1206 - RRDF_FIELD_TRANSFORM_XML, // format the field with an XML prettifier
1207 -} RRDF_FIELD_TRANSFORM;
1208 -
1209 -static inline const char *rrdf_field_transform_to_string(RRDF_FIELD_TRANSFORM transform) {
1210 - switch(transform) {
1211 - default:
1212 - case RRDF_FIELD_TRANSFORM_NONE:
1213 - return "none";
1214 -
1215 - case RRDF_FIELD_TRANSFORM_NUMBER:
1216 - return "number";
1217 -
1218 - case RRDF_FIELD_TRANSFORM_DURATION_S:
1219 - return "duration";
1220 -
1221 - case RRDF_FIELD_TRANSFORM_DATETIME_MS:
1222 - return "datetime";
1223 -
1224 - case RRDF_FIELD_TRANSFORM_DATETIME_USEC:
1225 - return "datetime_usec";
1226 -
1227 - case RRDF_FIELD_TRANSFORM_XML:
1228 - return "xml";
1229 - }
1230 -}
1231 -
1232 -typedef enum __attribute__((packed)) {
1233 - RRDF_FIELD_SORT_ASCENDING = (1 << 0),
1234 - RRDF_FIELD_SORT_DESCENDING = (1 << 1),
1235 -
1236 - RRDF_FIELD_SORT_FIXED = (1 << 7),
1237 -} RRDF_FIELD_SORT;
1238 -
1239 -static inline const char *rrdf_field_sort_to_string(RRDF_FIELD_SORT sort) {
1240 - if(sort & RRDF_FIELD_SORT_DESCENDING)
1241 - return "descending";
1242 -
1243 - else
1244 - return "ascending";
1245 -}
1246 -
1247 -typedef enum __attribute__((packed)) {
1248 - RRDF_FIELD_SUMMARY_UNIQUECOUNT, // Finds the number of unique values of a group of rows
1249 - RRDF_FIELD_SUMMARY_SUM, // Sums the values of a group of rows
1250 - RRDF_FIELD_SUMMARY_MIN, // Finds the minimum value of a group of rows
1251 - RRDF_FIELD_SUMMARY_MAX, // Finds the maximum value of a group of rows
1252 - // RRDF_FIELD_SUMMARY_EXTENT, // Finds the minimum and maximum values of a group of rows
1253 - RRDF_FIELD_SUMMARY_MEAN, // Finds the mean/average value of a group of rows
1254 - RRDF_FIELD_SUMMARY_MEDIAN, // Finds the median value of a group of rows
1255 - // RRDF_FIELD_SUMMARY_UNIQUE, // Finds the unique values of a group of rows
1256 - RRDF_FIELD_SUMMARY_COUNT, // Calculates the number of rows in a group
1257 -} RRDF_FIELD_SUMMARY;
1258 -
1259 -static inline const char *rrdf_field_summary_to_string(RRDF_FIELD_SUMMARY summary) {
1260 - switch(summary) {
1261 - default:
1262 - case RRDF_FIELD_SUMMARY_COUNT:
1263 - return "count";
1264 -
1265 - case RRDF_FIELD_SUMMARY_UNIQUECOUNT:
1266 - return "uniqueCount";
1267 -
1268 - case RRDF_FIELD_SUMMARY_SUM:
1269 - return "sum";
1270 -
1271 - case RRDF_FIELD_SUMMARY_MIN:
1272 - return "min";
1273 -
1274 - case RRDF_FIELD_SUMMARY_MEAN:
1275 - return "mean";
1276 -
1277 - case RRDF_FIELD_SUMMARY_MEDIAN:
1278 - return "median";
1279 -
1280 - case RRDF_FIELD_SUMMARY_MAX:
1281 - return "max";
1282 - }
1283 -}
1284 -
1285 -typedef enum __attribute__((packed)) {
1286 - RRDF_FIELD_FILTER_NONE = 0,
1287 - RRDF_FIELD_FILTER_RANGE,
1288 - RRDF_FIELD_FILTER_MULTISELECT,
1289 - RRDF_FIELD_FILTER_FACET,
1290 -} RRDF_FIELD_FILTER;
1291 -
1292 -static inline const char *rrdf_field_filter_to_string(RRDF_FIELD_FILTER filter) {
1293 - switch(filter) {
1294 - case RRDF_FIELD_FILTER_RANGE:
1295 - return "range";
1296 -
1297 - case RRDF_FIELD_FILTER_MULTISELECT:
1298 - return "multiselect";
1299 -
1300 - case RRDF_FIELD_FILTER_FACET:
1301 - return "facet";
1302 -
1303 - default:
1304 - case RRDF_FIELD_FILTER_NONE:
1305 - return "none";
1306 - }
1307 -}
1308 -
1309 -static inline void
1310 -buffer_rrdf_table_add_field(BUFFER *wb, size_t field_id, const char *key, const char *name, RRDF_FIELD_TYPE type,
1311 - RRDF_FIELD_VISUAL visual, RRDF_FIELD_TRANSFORM transform, size_t decimal_points,
1312 - const char *units, NETDATA_DOUBLE max, RRDF_FIELD_SORT sort, const char *pointer_to,
1313 - RRDF_FIELD_SUMMARY summary, RRDF_FIELD_FILTER filter, RRDF_FIELD_OPTIONS options,
1314 - const char *default_value) {
1315 -
1316 - buffer_json_member_add_object(wb, key);
1317 - {
1318 - buffer_json_member_add_uint64(wb, "index", field_id);
1319 - buffer_json_member_add_boolean(wb, "unique_key", options & RRDF_FIELD_OPTS_UNIQUE_KEY);
1320 - buffer_json_member_add_string(wb, "name", name);
1321 - buffer_json_member_add_boolean(wb, "visible", options & RRDF_FIELD_OPTS_VISIBLE);
1322 - buffer_json_member_add_string(wb, "type", rrdf_field_type_to_string(type));
1323 - buffer_json_member_add_string_or_omit(wb, "units", units);
1324 - buffer_json_member_add_string(wb, "visualization", rrdf_field_visual_to_string(visual));
1325 -
1326 - buffer_json_member_add_object(wb, "value_options");
1327 - {
1328 - buffer_json_member_add_string_or_omit(wb, "units", units);
1329 - buffer_json_member_add_string(wb, "transform", rrdf_field_transform_to_string(transform));
1330 - buffer_json_member_add_uint64(wb, "decimal_points", decimal_points);
1331 - buffer_json_member_add_string(wb, "default_value", default_value);
1332 - }
1333 - buffer_json_object_close(wb);
1334 -
1335 - if (!isnan((NETDATA_DOUBLE) (max)))
1336 - buffer_json_member_add_double(wb, "max", (NETDATA_DOUBLE) (max));
1337 -
1338 - buffer_json_member_add_string_or_omit(wb, "pointer_to", pointer_to);
1339 - buffer_json_member_add_string(wb, "sort", rrdf_field_sort_to_string(sort));
1340 - buffer_json_member_add_boolean(wb, "sortable", !(sort & RRDF_FIELD_SORT_FIXED));
1341 - buffer_json_member_add_boolean(wb, "sticky", options & RRDF_FIELD_OPTS_STICKY);
1342 - buffer_json_member_add_string(wb, "summary", rrdf_field_summary_to_string(summary));
1343 - buffer_json_member_add_string(wb, "filter", rrdf_field_filter_to_string(filter));
1344 -
1345 - buffer_json_member_add_boolean(wb, "full_width", options & RRDF_FIELD_OPTS_FULL_WIDTH);
1346 - buffer_json_member_add_boolean(wb, "wrap", options & RRDF_FIELD_OPTS_WRAP);
1347 - buffer_json_member_add_boolean(wb, "default_expanded_filter", options & RRDF_FIELD_OPTS_EXPANDED_FILTER);
1348 -
1349 - if(options & RRDF_FIELD_OPTS_DUMMY)
1350 - buffer_json_member_add_boolean(wb, "dummy", true);
1351 - }
1352 - buffer_json_object_close(wb);
1353 -}
1354 -
1355 -static inline void buffer_copy(BUFFER *dst, BUFFER *src) {
1157 +ALWAYS_INLINE
1158 +static void buffer_copy(BUFFER *dst, BUFFER *src) {
1159 if(!src || !dst)
1160 return;
1161
@@ -1365,7 +1168,8 @@ static inline void buffer_copy(BUFFER *dst, BUFFER *src) {
1168 dst->json = src->json;
1169 }
1170
1368 -static inline BUFFER *buffer_dup(BUFFER *src) {
1171 +ALWAYS_INLINE
1172 +static BUFFER *buffer_dup(BUFFER *src) {
1173 if(!src)
1174 return NULL;
1175
@@ -1375,7 +1179,8 @@ static inline BUFFER *buffer_dup(BUFFER *src) {
1179 }
1180
1181 char *url_encode(const char *str);
1378 -static inline void buffer_key_value_urlencode(BUFFER *wb, const char *key, const char *value) {
1182 +ALWAYS_INLINE
1183 +static void buffer_key_value_urlencode(BUFFER *wb, const char *key, const char *value) {
1184 char *encoded = NULL;
1185
1186 if(value && *value)
@@ -1386,4 +1191,7 @@ static inline void buffer_key_value_urlencode(BUFFER *wb, const char *key, const
1191 freez(encoded);
1192 }
1193
1194 +// Include functions_fields.h for field-related functionalities
1195 +#include "functions_fields.h"
1196 +
1197 #endif /* NETDATA_WEB_BUFFER_H */
src/libnetdata/buffer/functions_fields.c new
+156
@@ -0,0 +1,156 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "functions_fields.h"
4 +
5 +// Define mapping tables for the enums using the template macros
6 +
7 +// RRDF_FIELD_TYPE mapping
8 +ENUM_STR_MAP_DEFINE(RRDF_FIELD_TYPE) = {
9 + {RRDF_FIELD_TYPE_NONE, "none"},
10 + {RRDF_FIELD_TYPE_INTEGER, "integer"},
11 + {RRDF_FIELD_TYPE_BOOLEAN, "boolean"},
12 + {RRDF_FIELD_TYPE_STRING, "string"},
13 + {RRDF_FIELD_TYPE_DETAIL_STRING, "detail-string"},
14 + {RRDF_FIELD_TYPE_BAR_WITH_INTEGER, "bar-with-integer"},
15 + {RRDF_FIELD_TYPE_DURATION, "duration"},
16 + {RRDF_FIELD_TYPE_TIMESTAMP, "timestamp"},
17 + {RRDF_FIELD_TYPE_ARRAY, "array"},
18 + {0, NULL}
19 +};
20 +
21 +// Define the conversion functions
22 +ENUM_STR_DEFINE_FUNCTIONS(RRDF_FIELD_TYPE, RRDF_FIELD_TYPE_NONE, "none")
23 +
24 +// Convert an RRDF_FIELD_TYPE to a simplified JSON scalar type
25 +const char *field_type_to_json_scalar_type(RRDF_FIELD_TYPE type) {
26 + switch (type) {
27 + case RRDF_FIELD_TYPE_INTEGER:
28 + case RRDF_FIELD_TYPE_BAR_WITH_INTEGER:
29 + case RRDF_FIELD_TYPE_DURATION:
30 + case RRDF_FIELD_TYPE_TIMESTAMP:
31 + return "number";
32 +
33 + case RRDF_FIELD_TYPE_BOOLEAN:
34 + return "boolean";
35 +
36 + case RRDF_FIELD_TYPE_STRING:
37 + case RRDF_FIELD_TYPE_DETAIL_STRING:
38 + return "string";
39 +
40 + case RRDF_FIELD_TYPE_ARRAY:
41 + return "array";
42 +
43 + case RRDF_FIELD_TYPE_NONE:
44 + default:
45 + return "unknown";
46 + }
47 +}
48 +
49 +// RRDF_FIELD_VISUAL mapping
50 +ENUM_STR_MAP_DEFINE(RRDF_FIELD_VISUAL) = {
51 + {RRDF_FIELD_VISUAL_VALUE, "value"},
52 + {RRDF_FIELD_VISUAL_BAR, "bar"},
53 + {RRDF_FIELD_VISUAL_PILL, "pill"},
54 + {RRDF_FIELD_VISUAL_RICH, "richValue"},
55 + {RRDR_FIELD_VISUAL_ROW_OPTIONS, "rowOptions"},
56 + {0, NULL}
57 +};
58 +
59 +// Define the conversion functions
60 +ENUM_STR_DEFINE_FUNCTIONS(RRDF_FIELD_VISUAL, RRDF_FIELD_VISUAL_VALUE, "value")
61 +
62 +// RRDF_FIELD_TRANSFORM mapping
63 +ENUM_STR_MAP_DEFINE(RRDF_FIELD_TRANSFORM) = {
64 + {RRDF_FIELD_TRANSFORM_NONE, "none"},
65 + {RRDF_FIELD_TRANSFORM_NUMBER, "number"},
66 + {RRDF_FIELD_TRANSFORM_DURATION_S, "duration"},
67 + {RRDF_FIELD_TRANSFORM_DATETIME_MS, "datetime"},
68 + {RRDF_FIELD_TRANSFORM_DATETIME_USEC, "datetime_usec"},
69 + {RRDF_FIELD_TRANSFORM_XML, "xml"},
70 + {0, NULL}
71 +};
72 +
73 +// Define the conversion functions
74 +ENUM_STR_DEFINE_FUNCTIONS(RRDF_FIELD_TRANSFORM, RRDF_FIELD_TRANSFORM_NONE, "none")
75 +
76 +// RRDF_FIELD_SORT mapping
77 +ENUM_STR_MAP_DEFINE(RRDF_FIELD_SORT) = {
78 + {RRDF_FIELD_SORT_ASCENDING, "ascending"},
79 + {RRDF_FIELD_SORT_DESCENDING, "descending"},
80 + {0, NULL}
81 +};
82 +
83 +// Define the conversion functions
84 +ENUM_STR_DEFINE_FUNCTIONS(RRDF_FIELD_SORT, RRDF_FIELD_SORT_ASCENDING, "ascending")
85 +
86 +// RRDF_FIELD_SUMMARY mapping
87 +ENUM_STR_MAP_DEFINE(RRDF_FIELD_SUMMARY) = {
88 + {RRDF_FIELD_SUMMARY_COUNT, "count"},
89 + {RRDF_FIELD_SUMMARY_UNIQUECOUNT, "uniqueCount"},
90 + {RRDF_FIELD_SUMMARY_SUM, "sum"},
91 + {RRDF_FIELD_SUMMARY_MIN, "min"},
92 + {RRDF_FIELD_SUMMARY_MAX, "max"},
93 + {RRDF_FIELD_SUMMARY_MEAN, "mean"},
94 + {RRDF_FIELD_SUMMARY_MEDIAN, "median"},
95 + {0, NULL}
96 +};
97 +
98 +// Define the conversion functions
99 +ENUM_STR_DEFINE_FUNCTIONS(RRDF_FIELD_SUMMARY, RRDF_FIELD_SUMMARY_COUNT, "count")
100 +
101 +// RRDF_FIELD_FILTER mapping
102 +ENUM_STR_MAP_DEFINE(RRDF_FIELD_FILTER) = {
103 + {RRDF_FIELD_FILTER_NONE, "none"},
104 + {RRDF_FIELD_FILTER_RANGE, "range"},
105 + {RRDF_FIELD_FILTER_MULTISELECT, "multiselect"},
106 + {RRDF_FIELD_FILTER_FACET, "facet"},
107 + {0, NULL}
108 +};
109 +
110 +// Define the conversion functions
111 +ENUM_STR_DEFINE_FUNCTIONS(RRDF_FIELD_FILTER, RRDF_FIELD_FILTER_NONE, "none")
112 +
113 +void buffer_rrdf_table_add_field(BUFFER *wb, size_t field_id, const char *key, const char *name, RRDF_FIELD_TYPE type,
114 + RRDF_FIELD_VISUAL visual, RRDF_FIELD_TRANSFORM transform, size_t decimal_points,
115 + const char *units, NETDATA_DOUBLE max, RRDF_FIELD_SORT sort, const char *pointer_to,
116 + RRDF_FIELD_SUMMARY summary, RRDF_FIELD_FILTER filter, RRDF_FIELD_OPTIONS options,
117 + const char *default_value) {
118 +
119 + buffer_json_member_add_object(wb, key);
120 + {
121 + buffer_json_member_add_uint64(wb, "index", field_id);
122 + buffer_json_member_add_boolean(wb, "unique_key", options & RRDF_FIELD_OPTS_UNIQUE_KEY);
123 + buffer_json_member_add_string(wb, "name", name);
124 + buffer_json_member_add_boolean(wb, "visible", options & RRDF_FIELD_OPTS_VISIBLE);
125 + buffer_json_member_add_string(wb, "type", RRDF_FIELD_TYPE_2str(type));
126 + buffer_json_member_add_string_or_omit(wb, "units", units);
127 + buffer_json_member_add_string(wb, "visualization", RRDF_FIELD_VISUAL_2str(visual));
128 +
129 + buffer_json_member_add_object(wb, "value_options");
130 + {
131 + buffer_json_member_add_string_or_omit(wb, "units", units);
132 + buffer_json_member_add_string(wb, "transform", RRDF_FIELD_TRANSFORM_2str(transform));
133 + buffer_json_member_add_uint64(wb, "decimal_points", decimal_points);
134 + buffer_json_member_add_string(wb, "default_value", default_value);
135 + }
136 + buffer_json_object_close(wb);
137 +
138 + if (!isnan((NETDATA_DOUBLE) (max)))
139 + buffer_json_member_add_double(wb, "max", (NETDATA_DOUBLE) (max));
140 +
141 + buffer_json_member_add_string_or_omit(wb, "pointer_to", pointer_to);
142 + buffer_json_member_add_string(wb, "sort", RRDF_FIELD_SORT_2str(sort));
143 + buffer_json_member_add_boolean(wb, "sortable", !(sort & RRDF_FIELD_SORT_FIXED));
144 + buffer_json_member_add_boolean(wb, "sticky", options & RRDF_FIELD_OPTS_STICKY);
145 + buffer_json_member_add_string(wb, "summary", RRDF_FIELD_SUMMARY_2str(summary));
146 + buffer_json_member_add_string(wb, "filter", RRDF_FIELD_FILTER_2str(filter));
147 +
148 + buffer_json_member_add_boolean(wb, "full_width", options & RRDF_FIELD_OPTS_FULL_WIDTH);
149 + buffer_json_member_add_boolean(wb, "wrap", options & RRDF_FIELD_OPTS_WRAP);
150 + buffer_json_member_add_boolean(wb, "default_expanded_filter", options & RRDF_FIELD_OPTS_EXPANDED_FILTER);
151 +
152 + if(options & RRDF_FIELD_OPTS_DUMMY)
153 + buffer_json_member_add_boolean(wb, "dummy", true);
154 + }
155 + buffer_json_object_close(wb);
156 +}
\ No newline at end of file
src/libnetdata/buffer/functions_fields.h new
+99
@@ -0,0 +1,99 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_FUNCTIONS_FIELDS_H
4 +#define NETDATA_FUNCTIONS_FIELDS_H 1
5 +
6 +#include "../libnetdata.h"
7 +#include "buffer.h"
8 +#include "../template-enum.h"
9 +
10 +typedef enum __attribute__((packed)) {
11 + RRDF_FIELD_OPTS_NONE = 0,
12 + RRDF_FIELD_OPTS_UNIQUE_KEY = (1 << 0), // the field is the unique key of the row
13 + RRDF_FIELD_OPTS_VISIBLE = (1 << 1), // the field should be visible by default
14 + RRDF_FIELD_OPTS_STICKY = (1 << 2), // the field should be sticky
15 + RRDF_FIELD_OPTS_FULL_WIDTH = (1 << 3), // the field should get full width
16 + RRDF_FIELD_OPTS_WRAP = (1 << 4), // the field should wrap
17 + RRDF_FIELD_OPTS_DUMMY = (1 << 5), // not a presentable field
18 + RRDF_FIELD_OPTS_EXPANDED_FILTER = (1 << 6), // show the filter expanded
19 +} RRDF_FIELD_OPTIONS;
20 +
21 +typedef enum __attribute__((packed)) {
22 + RRDF_FIELD_TYPE_NONE,
23 + RRDF_FIELD_TYPE_INTEGER,
24 + RRDF_FIELD_TYPE_BOOLEAN,
25 + RRDF_FIELD_TYPE_STRING,
26 + RRDF_FIELD_TYPE_DETAIL_STRING,
27 + RRDF_FIELD_TYPE_BAR_WITH_INTEGER,
28 + RRDF_FIELD_TYPE_DURATION,
29 + RRDF_FIELD_TYPE_TIMESTAMP,
30 + RRDF_FIELD_TYPE_ARRAY,
31 +} RRDF_FIELD_TYPE;
32 +
33 +// Declare enum string mapping functions
34 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDF_FIELD_TYPE)
35 +
36 +// Convert an RRDF_FIELD_TYPE to a simplified JSON scalar type string
37 +// for easier LLM comprehension
38 +const char *field_type_to_json_scalar_type(RRDF_FIELD_TYPE type);
39 +
40 +typedef enum __attribute__((packed)) {
41 + RRDF_FIELD_VISUAL_VALUE, // show the value, possibly applying a transformation
42 + RRDF_FIELD_VISUAL_BAR, // show the value and a bar, respecting the max field to fill the bar at 100%
43 + RRDF_FIELD_VISUAL_PILL, //
44 + RRDF_FIELD_VISUAL_RICH, //
45 + RRDR_FIELD_VISUAL_ROW_OPTIONS, // this is a dummy column that is used for row options
46 +} RRDF_FIELD_VISUAL;
47 +
48 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDF_FIELD_VISUAL)
49 +
50 +typedef enum __attribute__((packed)) {
51 + RRDF_FIELD_TRANSFORM_NONE, // show the value as-is
52 + RRDF_FIELD_TRANSFORM_NUMBER, // show the value respecting the decimal_points
53 + RRDF_FIELD_TRANSFORM_DURATION_S, // transform as duration in second to a human-readable duration
54 + RRDF_FIELD_TRANSFORM_DATETIME_MS, // UNIX epoch timestamp in ms
55 + RRDF_FIELD_TRANSFORM_DATETIME_USEC, // UNIX epoch timestamp in usec
56 + RRDF_FIELD_TRANSFORM_XML, // format the field with an XML prettifier
57 +} RRDF_FIELD_TRANSFORM;
58 +
59 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDF_FIELD_TRANSFORM)
60 +
61 +typedef enum __attribute__((packed)) {
62 + RRDF_FIELD_SORT_ASCENDING = (1 << 0),
63 + RRDF_FIELD_SORT_DESCENDING = (1 << 1),
64 +
65 + RRDF_FIELD_SORT_FIXED = (1 << 7),
66 +} RRDF_FIELD_SORT;
67 +
68 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDF_FIELD_SORT)
69 +
70 +typedef enum __attribute__((packed)) {
71 + RRDF_FIELD_SUMMARY_UNIQUECOUNT, // Finds the number of unique values of a group of rows
72 + RRDF_FIELD_SUMMARY_SUM, // Sums the values of a group of rows
73 + RRDF_FIELD_SUMMARY_MIN, // Finds the minimum value of a group of rows
74 + RRDF_FIELD_SUMMARY_MAX, // Finds the maximum value of a group of rows
75 + // RRDF_FIELD_SUMMARY_EXTENT, // Finds the minimum and maximum values of a group of rows
76 + RRDF_FIELD_SUMMARY_MEAN, // Finds the mean/average value of a group of rows
77 + RRDF_FIELD_SUMMARY_MEDIAN, // Finds the median value of a group of rows
78 + // RRDF_FIELD_SUMMARY_UNIQUE, // Finds the unique values of a group of rows
79 + RRDF_FIELD_SUMMARY_COUNT, // Calculates the number of rows in a group
80 +} RRDF_FIELD_SUMMARY;
81 +
82 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDF_FIELD_SUMMARY)
83 +
84 +typedef enum __attribute__((packed)) {
85 + RRDF_FIELD_FILTER_NONE = 0,
86 + RRDF_FIELD_FILTER_RANGE,
87 + RRDF_FIELD_FILTER_MULTISELECT,
88 + RRDF_FIELD_FILTER_FACET,
89 +} RRDF_FIELD_FILTER;
90 +
91 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(RRDF_FIELD_FILTER)
92 +
93 +void buffer_rrdf_table_add_field(BUFFER *wb, size_t field_id, const char *key, const char *name, RRDF_FIELD_TYPE type,
94 + RRDF_FIELD_VISUAL visual, RRDF_FIELD_TRANSFORM transform, size_t decimal_points,
95 + const char *units, NETDATA_DOUBLE max, RRDF_FIELD_SORT sort, const char *pointer_to,
96 + RRDF_FIELD_SUMMARY summary, RRDF_FIELD_FILTER filter, RRDF_FIELD_OPTIONS options,
97 + const char *default_value);
98 +
99 +#endif /* NETDATA_FUNCTIONS_FIELDS_H */
\ No newline at end of file
src/libnetdata/facets/README.md
+384
@@ -0,0 +1,384 @@
1 +# Facets and Logs Query System
2 +
3 +The facets engine provides a unified interface for querying, searching, and analyzing log data from multiple sources including systemd journal (Linux) and Windows Event logs.
4 +
5 +## Overview
6 +
7 +The logs query system uses a two-phase approach:
8 +1. **Discovery Phase**: Get information about available log sources
9 +2. **Query Phase**: Query the actual log data
10 +
11 +## Discovery Phase (info=true)
12 +
13 +When called with `info=true`, the system returns metadata about available log sources and accepted parameters.
14 +
15 +### Response Structure
16 +
17 +```json
18 +{
19 + "_request": {
20 + // Echo of the request parameters
21 + },
22 + "versions": {
23 + "sources": 1748491820000467 // Version timestamp for source list
24 + },
25 + "v": 3, // API version
26 + "accepted_params": [
27 + // List of all parameters the API accepts
28 + ],
29 + "required_params": [
30 + // Array of required parameter definitions
31 + ],
32 + "show_ids": false,
33 + "has_history": true,
34 + "pagination": {
35 + "enabled": true,
36 + "key": "anchor",
37 + "column": "timestamp",
38 + "units": "timestamp_usec"
39 + },
40 + "status": 200,
41 + "type": "table",
42 + "help": "..."
43 +}
44 +```
45 +
46 +### Platform-Specific Sources
47 +
48 +#### Linux (systemd journal)
49 +Sources are hierarchical and represent different views of journal data:
50 +- `all` - All available logs
51 +- `all-local-logs` - All local logs
52 +- `all-local-namespaces` - Namespace-specific logs
53 +- `all-local-system-logs` - System logs only
54 +- `all-local-user-logs` - User logs only
55 +- `namespace-*` - Specific namespace logs
56 +- `remote-*` - Remote logs (if available)
57 +
58 +Each source includes:
59 +- File count
60 +- Total size
61 +- Time coverage (e.g., "1y 6mo 8d 16h 21m 18s")
62 +
63 +#### Windows (Event logs)
64 +Sources are Windows Event channels organized by provider:
65 +- `All` - All event channels
66 +- `All-Admin` - Administrative channels
67 +- `All-Classic` - Classic event logs (Application, Security, System)
68 +- `All-Operational` - Operational channels
69 +- Provider-specific channels (e.g., `Microsoft-Windows-*/Operational`)
70 +- Application channels (e.g., `Netdata/Health`, `Netdata/Daemon`)
71 +
72 +Each source includes:
73 +- Channel count
74 +- Total size
75 +- Time coverage
76 +- **Entry count** (unique to Windows)
77 +- `default_selected` flag (indicates if selected by default)
78 +
79 +**Windows Events Performance Optimization:**
80 +- Windows Events have few native fields (Level, TimeCreated, EventID, etc.)
81 +- Additional data is stored in XML format within each event
82 +- XML parsing is expensive, so windows-events.plugin uses **lazy loading**:
83 + - XML is parsed only for rows that will be returned to the user
84 + - For full-text search, XML is fetched and searched but not parsed
85 + - Field extraction from XML happens only for visible rows
86 +- This approach balances search capability with performance
87 +
88 +## Query Parameters
89 +
90 +### Core Parameters
91 +
92 +#### Required Parameters
93 +- `__logs_sources` - Multiselect field for choosing log sources to query
94 +
95 +#### Time Filtering
96 +- `after` - Start timestamp (Unix seconds)
97 +- `before` - End timestamp (Unix seconds)
98 +
99 +#### Pagination
100 +- `anchor` - Pagination anchor (timestamp in microseconds)
101 +- `direction` - "backward" (newest first) or "forward"
102 +- `last` - Number of entries to return (default: 200)
103 +
104 +#### Search and Analysis
105 +- `query` - Full-text search query
106 +- `facets` - Array of field names to analyze and return facet counts
107 +- `histogram` - Field name for generating time-based histogram (default: `_PRIORITY` for Linux, `Level` for Windows)
108 +
109 +#### Options
110 +- `data_only` - Return only log data without facet analysis
111 +- `delta` - Return incremental updates
112 +- `tail` - Follow mode for real-time updates
113 +- `sampling` - Sample rate for large datasets (default: 1000000)
114 +- `slice` - Time slicing for analysis
115 +- `if_modified_since` - Conditional requests
116 +
117 +### Default Values
118 +When parameters are not specified:
119 +- `source_type`: 1 (platform default)
120 +- `direction`: "backward" (newest entries first)
121 +- `last`: 200 entries
122 +- `sampling`: 1000000
123 +- Time range: Last 15 minutes (900 seconds)
124 +
125 +## Auto-Selection Behavior
126 +
127 +The system is designed to work without user intervention:
128 +
129 +1. **Source Auto-Selection**:
130 + - If no sources are selected, the UI/client should select the first available source
131 + - On Windows, sources with `default_selected: true` should be pre-selected
132 +
133 +2. **Immediate Data Fetch**:
134 + - After the `info` call, clients should immediately fetch data
135 + - Use default or auto-selected sources
136 + - Apply default time range and pagination
137 +
138 +## Query Response Format
139 +
140 +The query response includes:
141 +- Faceted results with counts (unless `data_only=true`)
142 +- Log entries matching the query
143 +- Histogram data with breakdown per facet value (unless `data_only=true`)
144 +- Pagination information for fetching more results
145 +- Source failures return partial data (no explicit error indication)
146 +
147 +## Usage Example Flow
148 +
149 +1. **Get available sources**:
150 + ```
151 + GET /api/v1/logs?info=true
152 + ```
153 +
154 +2. **Parse response and auto-select sources**:
155 + - Use sources with `default_selected: true` (Windows)
156 + - Or select first source (Linux)
157 +
158 +3. **Query logs**:
159 + ```
160 + POST /api/v1/logs
161 + {
162 + "__logs_sources": ["Application", "System"],
163 + "after": 1748527000,
164 + "before": 1748528000,
165 + "query": "error",
166 + "last": 100
167 + }
168 + ```
169 +
170 +## Modes of Operation
171 +
172 +### Fast Data Query (`data_only=true`)
173 +The fastest query mode that seeks directly to the anchor point (or time boundary) and returns the next `last` entries in the specified `direction`:
174 +- Does not scan the entire time window
175 +- No facet calculation unless `delta=true` is specified
176 +- Uses learned out-of-order deltas to minimize data scanning
177 +- Ideal for pagination and real-time updates
178 +
179 +### Full Analysis Query (`data_only=false`) - Default
180 +Scans the entire time window to calculate complete facet counts and histogram data:
181 +- Returns the same `last` entries as data_only mode
182 +- Provides comprehensive statistics for the entire time range
183 +- Required for initial queries to understand data distribution
184 +- Must complete full scan due to potential out-of-order entries
185 +
186 +### Real-time Following (`tail=true`)
187 +Combined with `data_only=true` and `delta=true` for efficient log following:
188 +- Scans only new entries since last anchor
189 +- When used with `if_modified_since`, returns HTTP 304 if no changes
190 +- Change detection:
191 + - Linux: Uses inotify for file watching
192 + - Windows: Polls providers for latest timestamps
193 +- Equivalent to `tail -f` or `journalctl -f`
194 +- Clients typically poll once per second for updates
195 +
196 +## Advanced Parameters
197 +
198 +### Sampling (`sampling`)
199 +Controls when statistical sampling begins:
200 +- Default: 1,000,000 entries (unusually high for accurate results)
201 +- Sampling algorithm (systemd-journal only):
202 + - Estimates volume per journal file based on time window
203 + - Distributes sampling proportionally across files
204 + - Maintains temporal representation across the dataset
205 +- Provides accurate counts up to the sampling threshold
206 +- Above threshold, provides statistically representative estimates
207 +- Sampling stages:
208 + - First stage: Skip facet processing, continue row counting
209 + - Second stage: Skip rows, estimate counts
210 + - Histogram shows additional dimensions: `unsampled` and `estimated`
211 +
212 +### Slicing (`slice`)
213 +Database-level filtering optimization (Linux only):
214 +- `slice=false` (default):
215 + - Facets library reads all data
216 + - Knows counts for all facet values (selected and non-selected)
217 + - Shows all possible filter options to users
218 +- `slice=true` (when backend supports it):
219 + - Database uses indexes to filter data
220 + - Faster queries for filtered datasets
221 + - Non-selected facet values may show as zero
222 + - Backend provides list of all possible values separately
223 +
224 +Windows Events note: Does not support slicing; Netdata maintains internal cache of possible facet values.
225 +
226 +## Performance Characteristics
227 +
228 +### Processing Speed
229 +- Typical: ~200,000 rows/second on modern hardware
230 +- Factors affecting speed:
231 + - Query complexity
232 + - Number of sources
233 + - Time window size
234 + - Filtering and facets
235 +
236 +### Progress Reporting
237 +For long-running queries:
238 +- UI can request progress updates
239 +- Each progress check extends the timeout
240 +- Immediate cancellation on user action
241 +- Prevents timeout during active monitoring
242 +
243 +### Out-of-Order Data Handling
244 +Log databases often contain out-of-order entries:
245 +- Plugins learn maximum out-of-order deltas per source
246 +- Linux: Per journal file
247 +- Windows: Per event provider
248 +- Enables efficient minimal scanning for `data_only=true` queries
249 +
250 +## Implementation Notes
251 +
252 +- The backend automatically detects the platform and returns appropriate sources
253 +- The same query interface works for both systemd journal and Windows Events
254 +- Sources can change over time (tracked by `versions.sources`)
255 +- Pagination uses microsecond timestamps for precise positioning
256 +- The system supports real-time following with `tail=true`
257 +- All caching and optimization is transparent to the caller
258 +- Authentication handled via USER_AUTH structure from Netdata Cloud SSO
259 +- Plugins run with root/Administrator privileges for full data access
260 +- User preferences (filters, time windows, facets) persist at dashboard level
261 +- All queries are logged for audit purposes (standard system logging)
262 +
263 +## Integration with Netdata Architecture
264 +
265 +### Plugin Communication
266 +- The facets library runs within plugins (systemd-journal.plugin, windows-events.plugin)
267 +- Plugins can run anywhere in the Netdata ecosystem (parent nodes, child nodes, etc.)
268 +- Communication with plugins happens via **rrdfunctions** - Netdata's function execution framework
269 +- rrdfunctions provide the transport layer to send requests to plugins and receive responses
270 +
271 +### MCP (Model Context Protocol) Integration
272 +- Netdata has an MCP server implementation for LLM interactions
273 +- Functions that return `has_history=true` (like logs) are currently excluded from MCP's table processing
274 +- Regular functions return simple table format: `{"type": "table", "data": [...], "columns": {...}}`
275 +- Logs functions return a different JSON structure with additional fields for:
276 + - Faceted search results
277 + - Histogram data
278 + - Pagination information (anchors)
279 + - Time-based navigation
280 + - Dynamic field discovery
281 +
282 +#### JSON Structure Differences: Regular Tables vs Logs
283 +
284 +**Regular Table Functions** (e.g., processes, network connections):
285 +```json
286 +{
287 + "status": 200,
288 + "type": "table",
289 + "has_history": false,
290 + "columns": { /* column definitions */ },
291 + "data": [ /* simple arrays of values */ ],
292 + "charts": { /* optional chart configs */ }
293 +}
294 +```
295 +
296 +**Logs Functions** (e.g., systemd-journal, Windows events):
297 +```json
298 +{
299 + "status": 200,
300 + "type": "table",
301 + "has_history": true,
302 + "_request": { /* complete request parameters */ },
303 + "columns": { /* column definitions with facet support */ },
304 + "data": [
305 + /* arrays starting with timestamp, rowOptions, then values */
306 + ],
307 + "facets": { /* available filters with counts */ },
308 + "histogram": { /* time-series visualization */ },
309 + "pagination": { /* anchor-based navigation */ },
310 + "_journal_files": { /* source metadata */ },
311 + "_sampling": { /* sampling statistics */ },
312 + "items": 12345,
313 + "last_modified": 1234567890,
314 + /* many more metadata fields */
315 +}
316 +```
317 +
318 +Key differences:
319 +- Logs have 20+ additional top-level fields
320 +- Data rows include timestamps and metadata
321 +- Built-in support for faceted filtering and time navigation
322 +- Rich metadata about data sources and query performance
323 +
324 +### Dynamic Schema Challenge
325 +- systemd-journal can store any structured data with custom fields
326 +- Plugins discover new fields dynamically as they process data
327 +- LLMs need a way to discover available fields to build intelligent queries
328 +- Current MCP implementation doesn't handle this dynamic schema discovery
329 +
330 +#### Example: Schema Variability in systemd-journal
331 +The same systemd-journal can contain completely different datasets with different schemas:
332 +
333 +1. **Standard System Logs** - Traditional journal fields:
334 + - System fields: `_HOSTNAME`, `_UID`, `_GID`, `_PID`, `_COMM`, `_EXE`
335 + - Message fields: `MESSAGE`, `PRIORITY`, `SYSLOG_FACILITY`, `SYSLOG_IDENTIFIER`
336 + - Systemd fields: `_SYSTEMD_UNIT`, `_SYSTEMD_CGROUP`, `_SYSTEMD_SLICE`
337 + - Boot/runtime fields: `_BOOT_ID`, `_MACHINE_ID`, `_RUNTIME_SCOPE`
338 +
339 +2. **Netdata Agent Events** - Custom application data with `AE_` prefix:
340 + - Agent metadata: `AE_AGENT_ID`, `AE_AGENT_VERSION`, `AE_AGENT_STATUS`
341 + - Hardware info: `AE_HW_BOARD_NAME`, `AE_HW_CHASSIS_TYPE`, `AE_HW_SYS_VENDOR`
342 + - Cloud/container: `AE_HOST_CLOUD_PROVIDER`, `AE_HOST_CONTAINER`, `AE_AGENT_KUBERNETES`
343 + - Crash analytics: `AE_AGENT_CRASHES`, `AE_FATAL_FAULT_ADDRESS`, `AE_FATAL_THREAD`
344 + - Performance: `AE_AGENT_UPTIME`, `AE_AGENT_TIMINGS_INIT`, `AE_AGENT_TIMINGS_EXIT`
345 +
346 +3. **Other Structured Data** - Any application can log structured data:
347 + - Netdata alerts: `ND_ALERT_NAME`, `ND_ALERT_STATUS`, `ND_ALERT_CLASS`
348 + - Custom applications: Arbitrary fields specific to each application
349 + - IoT devices: Sensor readings, device states, telemetry data
350 + - Business applications: Transaction IDs, user actions, audit trails
351 +
352 +This variability means:
353 +- The same logs query API must handle completely different schemas
354 +- Fields available for filtering/faceting vary by dataset
355 +- LLMs need to discover what fields exist before building meaningful queries
356 +- Traditional fixed-schema approaches don't work
357 +
358 +#### Platform-Specific Schema Characteristics
359 +
360 +**Linux (systemd-journal)**:
361 +- All fields are native journal fields - no lazy loading needed
362 +- Can have **thousands** of different fields in a single dataset
363 +- Multiple datasets can be queried in parallel (multiplexed, interleaved)
364 +- Fields are discovered dynamically as data is processed
365 +- Fast field access and filtering
366 +- This massive scalability and flexibility makes systemd-journal extremely powerful as a structured data store
367 +
368 +**Windows (Event logs)**:
369 +- Limited native fields (Level, TimeCreated, EventID, Provider, etc.)
370 +- Rich data stored in XML format within each event
371 +- Lazy XML parsing for performance:
372 + - Full-text search scans XML without parsing
373 + - XML parsing happens only for returned rows
374 + - Balances search capability with performance constraints
375 +- Field discovery requires XML inspection
376 +
377 +### Future Direction
378 +- Evolve MCP's function processing to handle both regular tables and logs uniformly
379 +- Create MCP tools that:
380 + - Support dynamic field discovery
381 + - Enable faceted search and analysis
382 + - Provide intelligent query building based on discovered schema
383 + - Handle pagination and time-based navigation
384 + - Leverage the full power of the facets engine
src/libnetdata/json/json-keys.c new
+160
@@ -0,0 +1,160 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "json-keys.h"
4 +
5 +// Static key name structures
6 +static const JSON_KEY_NAMES json_short_keys = {
7 + .selected = "sl",
8 + .excluded = "ex",
9 + .queried = "qr",
10 + .failed = "fl",
11 + .dimensions = "ds",
12 + .instances = "is",
13 + .contexts = "ctx",
14 + .alerts = "al",
15 + .statistics = "sts",
16 + .name = "nm",
17 + .hostname = "nm",
18 + .node_id = "nd",
19 + .value = "vl",
20 + .label_values = "vl",
21 + .machine_guid = "mg",
22 + .agent_index = "ai",
23 + .clear = "cl",
24 + .warning = "wr",
25 + .critical = "cr",
26 + .error = "er",
27 + .other = "ot",
28 + .count = "cnt",
29 + .volume = "vol",
30 + .anomaly_rate = "arp",
31 + .anomaly_count = "arc",
32 + .contribution = "con",
33 + .point_annotations = "pa",
34 + .point_schema = "point",
35 + .priority = "pri",
36 + .update_every = "ue",
37 + .tier = "tr",
38 + .after = "af",
39 + .before = "bf",
40 + .status = "st",
41 + .first_entry = "fe",
42 + .last_entry = "le",
43 + .node_index = "ni",
44 + .units = "un",
45 + .weight = "wg",
46 + .summary = "sum",
47 + .instances_count = "in",
48 + .nodes_count = "nd",
49 + .configurations_count = "cfg",
50 + .instance_id = "ch",
51 + .instance_name = "ch_n",
52 + .family = "fami",
53 + .context = "ctx",
54 +
55 + .alerts_index_id = "ati",
56 + .alert_global_id = "gi",
57 + .alert_name = "nm",
58 + .last_transition_id = "tr_i",
59 + .last_transition_value = "tr_v",
60 + .last_transition_timestamp = "tr_t",
61 + .last_updated_value = "v",
62 + .last_updated_timestamp = "t",
63 +
64 + .classification = "cl",
65 + .classifications = "cls",
66 + .component = "cm",
67 + .components = "cp",
68 + .type = "tp",
69 + .types = "ty",
70 + .recipients = "to",
71 + .source = "src",
72 + .config_hash_id = "cfg",
73 +};
74 +
75 +static const JSON_KEY_NAMES json_long_keys = {
76 + .selected = "selected",
77 + .excluded = "excluded",
78 + .queried = "queried",
79 + .failed = "failed",
80 + .dimensions = "dimensions",
81 + .instances = "instances",
82 + .contexts = "contexts",
83 + .alerts = "alerts",
84 + .statistics = "statistics",
85 + .name = "name",
86 + .hostname = "hostname",
87 + .node_id = "node_id",
88 + .value = "value",
89 + .label_values = "label_values",
90 + .machine_guid = "machine_guid",
91 + .agent_index = "agents_array_index",
92 + .clear = "clear",
93 + .warning = "warning",
94 + .critical = "critical",
95 + .error = "error",
96 + .other = "other",
97 + .count = "count",
98 + .volume = "volume",
99 + .anomaly_rate = "anomaly_rate_percent",
100 + .anomaly_count = "anomalous_points_count",
101 + .contribution = "contribution_percent",
102 + .point_annotations = "point_annotations_bitmap",
103 + .point_schema = "point_schema",
104 + .priority = "priority",
105 + .update_every = "update_every",
106 + .tier = "tier",
107 + .after = "after",
108 + .before = "before",
109 + .status = "status",
110 + .first_entry = "first_entry",
111 + .last_entry = "last_entry",
112 + .node_index = "nodes_array_index",
113 + .units = "units",
114 + .weight = "weight",
115 + .summary = "summary",
116 + .instances_count = "instances_count",
117 + .nodes_count = "nodes_count",
118 + .configurations_count = "configurations_count",
119 + .instance_id = "instance_id",
120 + .instance_name = "instance",
121 + .family = "family",
122 + .context = "context",
123 +
124 + .alerts_index_id = "alerts_array_index_id",
125 + .alert_global_id = "global_id",
126 + .alert_name = "alert",
127 + .last_transition_id = "last_transition_id",
128 + .last_transition_value = "last_transition_value",
129 + .last_transition_timestamp = "last_transition_timestamp",
130 + .last_updated_value = "last_updated_value",
131 + .last_updated_timestamp = "last_updated_timestamp",
132 +
133 + .classification = "classification",
134 + .classifications = "classifications",
135 + .component = "component",
136 + .components = "components",
137 + .type = "type",
138 + .types = "types",
139 + .recipients = "recipients",
140 + .source = "source",
141 + .config_hash_id = "config_hash_id",
142 +};
143 +
144 +// Thread-local pointer to the current key names
145 +__thread const JSON_KEY_NAMES *json_keys = &json_short_keys;
146 +
147 +void json_keys_init(JSON_KEYS_OPTIONS options) {
148 + if(options & JSON_KEYS_OPTION_LONG_KEYS)
149 + json_keys = &json_long_keys;
150 + else
151 + json_keys = &json_short_keys;
152 +}
153 +
154 +void json_keys_reset(void) {
155 + json_keys = &json_short_keys;
156 +}
157 +
158 +bool json_keys_are_long(void) {
159 + return json_keys == &json_long_keys;
160 +}
\ No newline at end of file
src/libnetdata/json/json-keys.h new
+115
@@ -0,0 +1,115 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_JSON_KEYS_H
4 +#define NETDATA_JSON_KEYS_H
5 +
6 +#include "../libnetdata.h"
7 +
8 +// JSON key names struct for short vs long key support
9 +typedef struct json_key_names {
10 + // Status and statistics
11 + const char *selected; // "sl" / "selected"
12 + const char *excluded; // "ex" / "excluded"
13 + const char *queried; // "qr" / "queried"
14 + const char *failed; // "fl" / "failed"
15 +
16 + // Object types
17 + const char *dimensions; // "ds" / "dimensions"
18 + const char *instances; // "is" / "instances"
19 + const char *contexts; // "ctx" / "contexts"
20 + const char *alerts; // "al" / "alerts"
21 + const char *statistics; // "sts" / "statistics"
22 +
23 + const char *node_id; // "nd" / "node_id"
24 + const char *machine_guid; // "mg" / "machine_guid"
25 +
26 + // metadata by name
27 + const char *name; // "nm" / "name"
28 + const char *hostname; // "nm" / "hostname"
29 + const char *alert_name; // "nm" / "alert_name"
30 + const char *context; // "ctx" / "context"
31 + const char *instance_id; // "ch" / "instance_id"
32 + const char *instance_name; // "ch_n" / "instance"
33 + const char *family; // "fami" / "family"
34 +
35 + // Values
36 + const char *value; // "vl" / "value"
37 + const char *label_values; // "vl" / "label_values"
38 +
39 + // Alert levels
40 + const char *clear; // "cl" / "clear"
41 + const char *warning; // "wr" / "warning"
42 + const char *critical; // "cr" / "critical"
43 + const char *error; // "er" / "error"
44 + const char *other; // "ot" / "other"
45 +
46 + // Storage Points
47 + const char *count; // "cnt" / "count"
48 + const char *volume; // "vol" / "volume"
49 + const char *anomaly_rate; // "arp" / "anomaly_rate_percent"
50 + const char *anomaly_count; // "arc" / "anomalous_points_count"
51 + const char *contribution; // "con" / "contribution_percent"
52 + const char *point_annotations; // "pa" / "point_annotations_bitmap"
53 + const char *point_schema; // "point" / "point_schema"
54 +
55 + // Other fields
56 + const char *priority; // "pri" / "priority"
57 + const char *update_every; // "ue" / "update_every"
58 + const char *tier; // "tr" / "tier"
59 + const char *after; // "af" / "after"
60 + const char *before; // "bf" / "before"
61 + const char *status; // "st" / "status"
62 + const char *first_entry; // "fe" / "first_entry"
63 + const char *last_entry; // "le" / "last_entry"
64 + const char *units; // "un" / "units"
65 + const char *weight; // "wg" / "weight"
66 +
67 + // indexes
68 + const char *agent_index; // "ai" / "agents_array_index"
69 + const char *node_index; // "ni" / "nodes_array_index"
70 + const char *alerts_index_id; // "ati" / "alerts_array_index_id"
71 +
72 + // alerts fields
73 + const char *summary; // "sum" / "summary"
74 + const char *nodes_count; // "nd" / "nodes_count"
75 + const char *instances_count; // "in" / "instances_count"
76 + const char *configurations_count; // "cfg" / "configurations_count"
77 +
78 + const char *alert_global_id; // "gi" / "global_id"
79 + const char *last_transition_id; // "tr_i" / "last_transition_id"
80 + const char *last_transition_value; // "tr_v" / "last_transition_value"
81 + const char *last_transition_timestamp; // "tr_t" / "last_transition_timestamp"
82 + const char *last_updated_value; // "v" / "last_updated_value"
83 + const char *last_updated_timestamp; // "t" / "last_updated_timestamp"
84 +
85 + const char *classification; // "cl" / "classification"
86 + const char *classifications; // "cls" / "classifications"
87 + const char *component; // "cm" / "component"
88 + const char *components; // "cp" / "components"
89 + const char *type; // "tp" / "type"
90 + const char *types; // "ty" / "types"
91 + const char *recipients; // "to" / "recipients"
92 +
93 + const char *source; // "src" / "source"
94 + const char *config_hash_id; // "cfg" / "configuration_hash"
95 +} JSON_KEY_NAMES;
96 +
97 +// Options type for controlling key format
98 +typedef enum {
99 + JSON_KEYS_OPTION_LONG_KEYS = (1 << 0),
100 +} JSON_KEYS_OPTIONS;
101 +
102 +// Thread-local pointer to the current key names
103 +extern __thread const JSON_KEY_NAMES *json_keys;
104 +
105 +// Macro for clean key access
106 +#define JSKEY(member) (json_keys->member)
107 +
108 +// Key name initialization
109 +void json_keys_init(JSON_KEYS_OPTIONS options);
110 +void json_keys_reset(void);
111 +
112 +// Check if long keys are enabled
113 +bool json_keys_are_long(void);
114 +
115 +#endif //NETDATA_JSON_KEYS_H
\ No newline at end of file
src/libnetdata/libnetdata.h
+2 -1
@@ -70,7 +70,8 @@ extern const char *netdata_configured_host_prefix;
70 #include "uuid/uuid.h"
71 #include "uuid/uuidmap.h"
72 #include "http/content_type.h"
73 -#include "http/http_access.h"
73 +#include "user-auth/http-access.h"
74 +#include "user-auth/user-auth.h"
75
76 #include "inlined.h"
77 #include "parsers/parsers.h"
src/libnetdata/log/systemd-journal-helpers.c
+1 -1
@@ -57,7 +57,7 @@ int journal_direct_fd(const char *path) {
57 struct sockaddr_un addr;
58 memset(&addr, 0, sizeof(struct sockaddr_un));
59 addr.sun_family = AF_UNIX;
60 - strncpy(addr.sun_path, path, sizeof(addr.sun_path) - 1);
60 + strncpyz(addr.sun_path, path, sizeof(addr.sun_path) - 1);
61
62 // Connect the socket (optional, but can simplify send operations)
63 if (connect(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
src/libnetdata/parsers/entries.c
+1 -1
@@ -102,7 +102,7 @@ bool entries_parse(const char *entries_str, uint64_t *result, const char *defaul
102 su = su_def;
103 else {
104 if (unit_len >= sizeof(unit)) unit_len = sizeof(unit) - 1;
105 - strncpy(unit, unit_start, unit_len);
105 + memcpy(unit, unit_start, unit_len);
106 unit[unit_len] = '\0';
107 su = entries_find_unit(unit);
108 if (!su) {
src/libnetdata/parsers/size.c
+1 -1
@@ -130,7 +130,7 @@ bool size_parse(const char *size_str, uint64_t *result, const char *default_unit
130 su = su_def;
131 else {
132 if (unit_len >= sizeof(unit)) unit_len = sizeof(unit) - 1;
133 - strncpy(unit, unit_start, unit_len);
133 + memcpy(unit, unit_start, unit_len);
134 unit[unit_len] = '\0';
135 su = size_find_unit(unit);
136 if (!su) {
src/libnetdata/parsers/timeframe.h
-2
@@ -10,8 +10,6 @@ typedef struct {
10 time_t before;
11 } TIMEFRAME;
12
13 -#define API_RELATIVE_TIME_MAX (3 * 365 * 86400)
14 -
13 #define API_RELATIVE_TIME_INVALID (-1000000000)
14
15 #define API_RELATIVE_TIME_THIS_MINUTE (API_RELATIVE_TIME_INVALID - 1) // this minute at 00 seconds
src/libnetdata/query_progress/progress.c
+1 -1
@@ -569,7 +569,7 @@ int progress_function_result(BUFFER *wb, const char *hostname) {
569
570 // duration
571 buffer_rrdf_table_add_field(wb, field_id++, "Duration", "Query Duration",
572 - RRDF_FIELD_TYPE_DURATION, RRDF_FIELD_VISUAL_VALUE, RRDF_FIELD_TRANSFORM_NUMBER,
572 + RRDF_FIELD_TYPE_DURATION, RRDF_FIELD_VISUAL_VALUE, RRDF_FIELD_TRANSFORM_DURATION_S,
573 2, "ms", (double)max_duration_ut / USEC_PER_MS, RRDF_FIELD_SORT_DESCENDING, NULL,
574 RRDF_FIELD_SUMMARY_MAX, RRDF_FIELD_FILTER_RANGE,
575 RRDF_FIELD_OPTS_VISIBLE, NULL);
src/libnetdata/query_progress/progress.h
+1 -1
@@ -13,7 +13,7 @@ void query_progress_functions_update(nd_uuid_t *transaction, size_t done, size_t
13
14 int web_api_v2_report_progress(nd_uuid_t *transaction, BUFFER *wb);
15
16 -#define RRDFUNCTIONS_PROGRESS_HELP "View the progress on the running and latest Netdata API Requests"
16 +#define RRDFUNCTIONS_PROGRESS_HELP "Monitors active and recent Netdata API requests with transaction details, duration, and response sizes."
17 int progress_function_result(BUFFER *wb, const char *hostname);
18
19 #endif // NETDATA_QUERY_PROGRESS_H
src/libnetdata/simple_pattern/simple_pattern.c
+44
@@ -413,3 +413,47 @@ char *simple_pattern_iterate(SIMPLE_PATTERN **p)
413 (*Proot) = (*Proot)->next;
414 return (char *) root->match;
415 }
416 +
417 +bool simple_pattern_contains_wildcards(const char *str, const char *separators) {
418 + if(unlikely(!str || !*str)) return false;
419 +
420 + // Check if it starts with exclamation mark (and not escaped)
421 + if(*str == '!')
422 + return true;
423 +
424 + bool isseparator[256] = {
425 + [' '] = true // space
426 + , ['\t'] = true // tab
427 + , ['\r'] = true // carriage return
428 + , ['\n'] = true // new line
429 + , ['\f'] = true // form feed
430 + , ['\v'] = true // vertical tab
431 + };
432 +
433 + if (unlikely(separators == SIMPLE_PATTERN_NO_SEPARATORS))
434 + memset(isseparator, false, sizeof(isseparator));
435 +
436 + else if (unlikely(separators && *separators)) {
437 + memset(isseparator, false, sizeof(isseparator));
438 + while(*separators) isseparator[(unsigned char)*separators++] = true;
439 + }
440 +
441 + // Check for separators or asterisks in the string, respecting escaping
442 + const char *s = str;
443 + char escape = 0;
444 + while(*s) {
445 + if(*s == '\\' && !escape) {
446 + escape = 1;
447 + s++;
448 + }
449 + else {
450 + if (!escape && (isseparator[(unsigned char)*s] || *s == '*'))
451 + return true;
452 +
453 + s++;
454 + escape = 0;
455 + }
456 + }
457 +
458 + return false;
459 +}
src/libnetdata/simple_pattern/simple_pattern.h
+4
@@ -49,11 +49,15 @@ void simple_pattern_dump(uint64_t debug_type, SIMPLE_PATTERN *p) ;
49 int simple_pattern_is_potential_name(SIMPLE_PATTERN *p) ;
50 char *simple_pattern_iterate(SIMPLE_PATTERN **p);
51
52 +// check if string contains pattern wildcards (*, ! prefix, or separators)
53 +bool simple_pattern_contains_wildcards(const char *str, const char *separators);
54 +
55 #define SIMPLE_PATTERN_DEFAULT_WEB_SEPARATORS ",|\t\r\n\f\v"
56
57 #define is_valid_sp(x) ((x) && *(x) && !((x)[0] == '*' && (x)[1] == '\0'))
58
59 #define string_to_simple_pattern(str) (is_valid_sp(str) ? simple_pattern_create(str, SIMPLE_PATTERN_DEFAULT_WEB_SEPARATORS, SIMPLE_PATTERN_EXACT, true) : NULL)
60 #define string_to_simple_pattern_nocase(str) (is_valid_sp(str) ? simple_pattern_create(str, SIMPLE_PATTERN_DEFAULT_WEB_SEPARATORS, SIMPLE_PATTERN_EXACT, false) : NULL)
61 +#define string_to_simple_pattern_nocase_substring(str) (is_valid_sp(str) ? simple_pattern_create(str, SIMPLE_PATTERN_DEFAULT_WEB_SEPARATORS, SIMPLE_PATTERN_SUBSTRING, false) : NULL)
62
63 #endif //NETDATA_SIMPLE_PATTERN_H
src/libnetdata/string/string.c
+5 -1
@@ -548,6 +548,7 @@ size_t string_destroy(void) {
548 BUFFER *wb = buffer_create(16384, NULL);
549
550 fprintf(stderr, "\n========= STRINGS GROUPED BY CREATION STACKTRACE =========\n");
551 + size_t total_strings = 0;
552 #endif
553
554 // Traverse all partitions
@@ -566,6 +567,9 @@ size_t string_destroy(void) {
567 while (PValue) {
568 STRING *string = (STRING *)string_idx;
569 if(string) {
570 + fprintf(stderr, " > STRING REMAINING No %zu: %d references on: '%s'\n",
571 + ++total_strings, string->refcount, string2str(string));
572 +
573 for (int i = 0; i < string->stacktraces.num_stacktraces; i++) {
574 if (string->stacktraces.stacktraces[i]) {
575 Word_t key = (Word_t)string->stacktraces.stacktraces[i];
@@ -685,7 +689,7 @@ size_t string_destroy(void) {
689 buffer_flush(wb);
690 stacktrace_to_buffer(entries[i].st, wb);
691
688 - fprintf(stderr, "\n > STRINGS REMAINING %zu: %zu strings created from:\n%s\n",
692 + fprintf(stderr, "\n > STRINGS BACKTRACE %zu: %zu strings created from:\n%s\n",
693 i + 1, entries[i].count, buffer_tostring(wb));
694 }
695
src/libnetdata/user-auth/http-access.c renamed
+1 -1
@@ -1,6 +1,6 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#include "../libnetdata.h"
3 +#include "libnetdata/libnetdata.h"
4
5 static struct {
6 HTTP_USER_ROLE role;
src/libnetdata/user-auth/http-access.h renamed
src/libnetdata/user-auth/user-auth.c new
+96
@@ -0,0 +1,96 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "user-auth.h"
4 +
5 +// Define the mapping between enum values and strings
6 +ENUM_STR_MAP_DEFINE(USER_AUTH_METHOD) = {
7 + {USER_AUTH_METHOD_NONE, "none"},
8 + {USER_AUTH_METHOD_CLOUD, "NC"},
9 + {USER_AUTH_METHOD_BEARER, "api-bearer"},
10 + {USER_AUTH_METHOD_GOD, "god"},
11 +
12 + // terminator
13 + {0, NULL},
14 +};
15 +
16 +// Define the functions to convert between enum values and strings
17 +ENUM_STR_DEFINE_FUNCTIONS(USER_AUTH_METHOD, USER_AUTH_METHOD_NONE, "none")
18 +
19 +bool user_auth_source_is_cloud(const char *source) {
20 + return source && *source && strstartswith(source, "method=NC,");
21 +}
22 +
23 +void user_auth_to_source_buffer(USER_AUTH *user_auth, BUFFER *source) {
24 + buffer_reset(source);
25 +
26 + buffer_sprintf(source, "method=%s", USER_AUTH_METHOD_2str(user_auth->method));
27 + buffer_sprintf(source, ",role=%s", user_auth->method == USER_AUTH_METHOD_GOD ? "god" : http_id2user_role(user_auth->user_role));
28 + buffer_sprintf(source, ",permissions="HTTP_ACCESS_FORMAT, (HTTP_ACCESS_FORMAT_CAST)user_auth->access);
29 +
30 + if(user_auth->client_name[0])
31 + buffer_sprintf(source, ",user=%s", user_auth->client_name);
32 +
33 + if(!uuid_is_null(user_auth->cloud_account_id.uuid)) {
34 + char uuid_str[UUID_COMPACT_STR_LEN];
35 + uuid_unparse_lower_compact(user_auth->cloud_account_id.uuid, uuid_str);
36 + buffer_sprintf(source, ",account=%s", uuid_str);
37 + }
38 +
39 + if(user_auth->client_ip[0])
40 + buffer_sprintf(source, ",ip=%s", user_auth->client_ip);
41 +
42 + if(user_auth->forwarded_for[0])
43 + buffer_sprintf(source, ",forwarded_for=%s", user_auth->forwarded_for);
44 +}
45 +
46 +bool user_auth_from_source(const char *src, USER_AUTH *parsed) {
47 + char *buf, *token, *saveptr;
48 +
49 + if (!src || !parsed)
50 + return false;
51 +
52 + // Initialize the structure.
53 + memset(parsed, 0, sizeof(*parsed));
54 +
55 + // Duplicate the source string because strtok_r modifies it.
56 + buf = strdupz(src);
57 + token = strtok_r(buf, ",", &saveptr);
58 + while (token) {
59 + char *equal_sign = strchr(token, '=');
60 + if (equal_sign) {
61 + *equal_sign = '\0';
62 + const char *key = token;
63 + const char *value = equal_sign + 1;
64 +
65 + if (strcmp(key, "method") == 0) {
66 + parsed->method = USER_AUTH_METHOD_2id(value);
67 + }
68 + else if (strcmp(key, "role") == 0) {
69 + if (strcmp(value, "god") == 0)
70 + parsed->method = USER_AUTH_METHOD_GOD;
71 + else
72 + parsed->user_role = http_user_role2id(value);
73 + }
74 + else if (strcmp(key, "permissions") == 0)
75 + parsed->access = http_access_from_hex_str(value);
76 +
77 + else if (strcmp(key, "user") == 0) {
78 + strncpyz(parsed->client_name, value, CLOUD_CLIENT_NAME_LENGTH - 1);
79 + }
80 + else if (strcmp(key, "account") == 0) {
81 + if (uuid_parse(value, parsed->cloud_account_id.uuid) != 0)
82 + parsed->cloud_account_id = UUID_ZERO;
83 + }
84 + else if (strcmp(key, "ip") == 0) {
85 + strncpyz(parsed->client_ip, value, INET6_ADDRSTRLEN - 1);
86 + }
87 + else if (strcmp(key, "forwarded_for") == 0) {
88 + strncpyz(parsed->forwarded_for, value, INET6_ADDRSTRLEN - 1);
89 + }
90 + }
91 + token = strtok_r(NULL, ",", &saveptr);
92 + }
93 +
94 + freez(buf);
95 + return true;
96 +}
src/libnetdata/user-auth/user-auth.h new
+37
@@ -0,0 +1,37 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_USER_AUTH_H
4 +#define NETDATA_USER_AUTH_H
5 +
6 +#include "../libnetdata.h"
7 +#include "../template-enum.h"
8 +
9 +#define CLOUD_CLIENT_NAME_LENGTH 64
10 +
11 +typedef enum auth_type {
12 + USER_AUTH_METHOD_NONE = 0,
13 + USER_AUTH_METHOD_CLOUD,
14 + USER_AUTH_METHOD_BEARER,
15 + USER_AUTH_METHOD_GOD,
16 +} USER_AUTH_METHOD;
17 +
18 +typedef struct user_auth {
19 + char client_ip[INET6_ADDRSTRLEN];
20 + char forwarded_for[INET6_ADDRSTRLEN];
21 + char client_name[CLOUD_CLIENT_NAME_LENGTH];
22 + ND_UUID cloud_account_id;
23 + USER_AUTH_METHOD method;
24 + HTTP_USER_ROLE user_role;
25 + HTTP_ACCESS access;
26 +} USER_AUTH;
27 +
28 +// Declare the enum-to-string conversion functions
29 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(USER_AUTH_METHOD)
30 +
31 +bool user_auth_from_source(const char *src, USER_AUTH *parsed);
32 +
33 +bool user_auth_source_is_cloud(const char *source);
34 +
35 +void user_auth_to_source_buffer(USER_AUTH *user_auth, BUFFER *source);
36 +
37 +#endif //NETDATA_USER_AUTH_H
src/plugins.d/pluginsd_internals.h
+2
@@ -86,6 +86,7 @@ static inline void pluginsd_clear_scope_chart(PARSER *parser, const char *keywor
86
87 parser->user.st = NULL;
88 parser->user.cleanup_slots = false;
89 + parser->user.clabel_count = 0;
90 }
91
92 static ALWAYS_INLINE bool pluginsd_set_scope_chart(PARSER *parser, RRDSET *st, const char *keyword) {
@@ -115,6 +116,7 @@ static ALWAYS_INLINE bool pluginsd_set_scope_chart(PARSER *parser, RRDSET *st, c
116 st->pluginsd.pos = 0;
117 parser->user.st = st;
118 parser->user.cleanup_slots = false;
119 + parser->user.clabel_count = 0;
120
121 return true;
122 }
src/plugins.d/pluginsd_parser.c
+11 -10
@@ -670,13 +670,13 @@ static inline PARSER_RC pluginsd_clabel(char **words, size_t num_words, PARSER *
670 return PLUGINSD_DISABLE_PLUGIN(parser, NULL, NULL);
671 }
672
673 - if(unlikely(!parser->user.chart_rrdlabels_linked_temporarily)) {
674 - RRDSET *st = pluginsd_get_scope_chart(parser);
675 - parser->user.chart_rrdlabels_linked_temporarily = st->rrdlabels;
676 - rrdlabels_unmark_all(parser->user.chart_rrdlabels_linked_temporarily);
677 - }
673 + RRDSET *st = pluginsd_get_scope_chart(parser);
674 + if(!st) return PLUGINSD_DISABLE_PLUGIN(parser, PLUGINSD_KEYWORD_CLABEL, "Got CHART LABEL without a chart");
675
679 - rrdlabels_add(parser->user.chart_rrdlabels_linked_temporarily, name, value, str2l(label_source));
676 + if(unlikely(parser->user.clabel_count++ == 0))
677 + rrdlabels_unmark_all(st->rrdlabels);
678 +
679 + rrdlabels_add(st->rrdlabels, name, value, str2l(label_source));
680
681 return PARSER_RC_OK;
682 }
@@ -690,18 +690,19 @@ static inline PARSER_RC pluginsd_clabel_commit(char **words __maybe_unused, size
690
691 netdata_log_debug(D_PLUGINSD, "requested to commit chart labels");
692
693 - if(!parser->user.chart_rrdlabels_linked_temporarily) {
693 + if(!parser->user.clabel_count) {
694 netdata_log_error("PLUGINSD: 'host:%s' got CLABEL_COMMIT, without a CHART or BEGIN. Ignoring it.", rrdhost_hostname(host));
695 return PLUGINSD_DISABLE_PLUGIN(parser, NULL, NULL);
696 }
697
698 - rrdlabels_remove_all_unmarked(parser->user.chart_rrdlabels_linked_temporarily);
698 + rrdlabels_remove_all_unmarked(st->rrdlabels);
699
700 rrdset_flag_set(st, RRDSET_FLAG_METADATA_UPDATE);
701 rrdhost_flag_set(st->rrdhost, RRDHOST_FLAG_METADATA_UPDATE);
702 rrdset_metadata_updated(st);
703
704 - parser->user.chart_rrdlabels_linked_temporarily = NULL;
704 + parser->user.clabel_count = 0;
705 +
706 return PARSER_RC_OK;
707 }
708
@@ -1124,7 +1125,7 @@ void pluginsd_process_cleanup(PARSER *parser) {
1125 pluginsd_host_define_cleanup(parser);
1126
1127 rrdlabels_destroy(parser->user.new_host_labels);
1127 - rrdlabels_destroy(parser->user.chart_rrdlabels_linked_temporarily);
1128 + parser->user.clabel_count = 0;
1129
1130 parser_destroy(parser);
1131 }
src/plugins.d/pluginsd_parser.h
+1 -1
@@ -57,7 +57,7 @@ typedef struct parser_user_object {
57 struct plugind *cd;
58 int trust_durations;
59 RRDLABELS *new_host_labels;
60 - RRDLABELS *chart_rrdlabels_linked_temporarily;
60 + size_t clabel_count;
61 size_t data_collections_count;
62 int enabled;
63
src/streaming/stream-receiver-connection.c
+3 -3
@@ -342,7 +342,7 @@ int stream_receiver_accept_connection(struct web_client *w, char *decoded_query_
342 rrdhost_system_info_hops_set(rpt->system_info, rpt->hops);
343
344 nd_sock_init(&rpt->sock, netdata_ssl_web_server_ctx, false);
345 - rpt->remote_ip = strdupz(w->client_ip);
345 + rpt->remote_ip = strdupz(w->user_auth.client_ip);
346 rpt->remote_port = strdupz(w->client_port);
347
348 rpt->config.update_every = nd_profile.update_every;
@@ -530,7 +530,7 @@ int stream_receiver_accept_connection(struct web_client *w, char *decoded_query_
530 return stream_receiver_response_permission_denied(w);
531 }
532
533 - if(!stream_conf_api_key_allows_client(rpt->key, w->client_ip)) {
533 + if(!stream_conf_api_key_allows_client(rpt->key, w->user_auth.client_ip)) {
534 stream_receiver_log_status(
535 rpt,
536 "rejecting streaming connection; API key is not allowed from this IP",
@@ -562,7 +562,7 @@ int stream_receiver_accept_connection(struct web_client *w, char *decoded_query_
562 return stream_receiver_response_permission_denied(w);
563 }
564
565 - if(!stream_conf_api_key_allows_client(rpt->machine_guid, w->client_ip)) {
565 + if(!stream_conf_api_key_allows_client(rpt->machine_guid, w->user_auth.client_ip)) {
566 stream_receiver_log_status(
567 rpt,
568 "rejecting streaming connection; machine UUID is not allowed from this IP",
src/web/api/formatters/json/json.c
+29 -8
@@ -1,11 +1,16 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 #include "json.h"
4 +#include "../jsonwrap-internal.h"
5 +#include "web/mcp/mcp.h"
6 +#include "libnetdata/json/json-keys.h"
7
8 #define JSON_DATES_JS 1
9 #define JSON_DATES_TIMESTAMP 2
10
11 void rrdr2json(RRDR *r, BUFFER *wb, RRDR_OPTIONS options, int datatable) {
12 + json_keys_init((options & RRDR_OPTION_LONG_JSON_KEYS) ? JSON_KEYS_OPTION_LONG_KEYS : 0);
13 +
14 //netdata_log_info("RRD2JSON(): %s: BEGIN", r->st->id);
15 int row_annotations = 0, dates, dates_with_new = 0;
16 char kq[2] = "", // key quote
@@ -101,6 +106,9 @@ void rrdr2json(RRDR *r, BUFFER *wb, RRDR_OPTIONS options, int datatable) {
106 size_t overflow_annotation_len = strlen(overflow_annotation);
107 size_t object_rows_time_len = strlen(object_rows_time);
108
109 + // RFC3339 buffer - allocated once and reused
110 + char rfc3339_buf[RFC3339_MAX_LENGTH];
111 +
112 // -------------------------------------------------------------------------
113 // print the JSON header
114
@@ -201,11 +209,21 @@ void rrdr2json(RRDR *r, BUFFER *wb, RRDR_OPTIONS options, int datatable) {
209 if(unlikely( options & RRDR_OPTION_OBJECTSROWS ))
210 buffer_fast_strcat(wb, object_rows_time, object_rows_time_len);
211
204 - buffer_print_netdata_double(wb, (NETDATA_DOUBLE) r->t[i]);
212 + if(unlikely(options & RRDR_OPTION_RFC3339)) {
213 + // Output RFC3339 timestamp
214 + usec_t timestamp_ut = (usec_t)r->t[i] * USEC_PER_SEC;
215 + rfc3339_datetime_ut(rfc3339_buf, sizeof(rfc3339_buf), timestamp_ut, 0, true);
216 + buffer_strcat(wb, "\"");
217 + buffer_strcat(wb, rfc3339_buf);
218 + buffer_strcat(wb, "\"");
219 + }
220 + else {
221 + buffer_print_netdata_double(wb, (NETDATA_DOUBLE) r->t[i]);
222
206 - // in ms
207 - if(unlikely(options & RRDR_OPTION_MILLISECONDS))
208 - buffer_fast_strcat(wb, "000", 3);
223 + // in ms
224 + if(unlikely(options & RRDR_OPTION_MILLISECONDS))
225 + buffer_fast_strcat(wb, "000", 3);
226 + }
227
228 buffer_fast_strcat(wb, post_date, post_date_len);
229 }
@@ -254,6 +272,9 @@ void rrdr2json_v2(RRDR *r, BUFFER *wb) {
272
273 buffer_json_member_add_object(wb, "result");
274
275 + if(options & RRDR_OPTION_MCP_INFO)
276 + buffer_json_member_add_string(wb, "info", MCP_QUERY_INFO_RESULT_SECTION);
277 +
278 buffer_json_member_add_array(wb, "labels");
279 buffer_json_add_array_item_string(wb, "time");
280 long d, i;
@@ -267,12 +288,12 @@ void rrdr2json_v2(RRDR *r, BUFFER *wb) {
288 }
289 buffer_json_array_close(wb); // labels
290
270 - buffer_json_member_add_object(wb, "point");
291 + buffer_json_member_add_object(wb, JSKEY(point_schema));
292 {
293 size_t point_count = 0;
294 buffer_json_member_add_uint64(wb, "value", point_count++);
274 - buffer_json_member_add_uint64(wb, "arp", point_count++);
275 - buffer_json_member_add_uint64(wb, "pa", point_count++);
295 + buffer_json_member_add_uint64(wb, JSKEY(anomaly_rate), point_count++);
296 + buffer_json_member_add_uint64(wb, JSKEY(point_annotations), point_count++);
297 if (send_count)
298 buffer_json_member_add_uint64(wb, "count", point_count++);
299 if (send_hidden)
@@ -303,7 +324,7 @@ void rrdr2json_v2(RRDR *r, BUFFER *wb) {
324 if (options & RRDR_OPTION_MILLISECONDS)
325 buffer_json_add_array_item_time_ms(wb, now); // the time
326 else
306 - buffer_json_add_array_item_time_t(wb, now); // the time
327 + buffer_json_add_array_item_time_t_formatted(wb, now, options & RRDR_OPTION_RFC3339); // the time
328
329 for (d = 0; d < used; d++) {
330 if (!rrdr_dimension_should_be_exposed(r->od[d], options))
src/web/api/formatters/json_wrapper.c deleted
-1577
@@ -1,1577 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -#include "json_wrapper.h"
4 -
5 -static void jsonwrap_query_metric_plan(BUFFER *wb, QUERY_METRIC *qm) {
6 - buffer_json_member_add_array(wb, "plans");
7 - for (size_t p = 0; p < qm->plan.used; p++) {
8 - QUERY_PLAN_ENTRY *qp = &qm->plan.array[p];
9 -
10 - buffer_json_add_array_item_object(wb);
11 - buffer_json_member_add_uint64(wb, "tr", qp->tier);
12 - buffer_json_member_add_time_t(wb, "af", qp->after);
13 - buffer_json_member_add_time_t(wb, "bf", qp->before);
14 - buffer_json_object_close(wb);
15 - }
16 - buffer_json_array_close(wb);
17 -
18 - buffer_json_member_add_array(wb, "tiers");
19 - for (size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
20 - buffer_json_add_array_item_object(wb);
21 - buffer_json_member_add_uint64(wb, "tr", tier);
22 - buffer_json_member_add_time_t(wb, "fe", qm->tiers[tier].db_first_time_s);
23 - buffer_json_member_add_time_t(wb, "le", qm->tiers[tier].db_last_time_s);
24 - buffer_json_member_add_int64(wb, "wg", qm->tiers[tier].weight);
25 - buffer_json_object_close(wb);
26 - }
27 - buffer_json_array_close(wb);
28 -}
29 -
30 -void jsonwrap_query_plan(RRDR *r, BUFFER *wb) {
31 - QUERY_TARGET *qt = r->internal.qt;
32 -
33 - buffer_json_member_add_object(wb, "query_plan");
34 - for(size_t m = 0; m < qt->query.used; m++) {
35 - QUERY_METRIC *qm = query_metric(qt, m);
36 - buffer_json_member_add_object(wb, query_metric_id(qt, qm));
37 - jsonwrap_query_metric_plan(wb, qm);
38 - buffer_json_object_close(wb);
39 - }
40 - buffer_json_object_close(wb);
41 -}
42 -
43 -static inline size_t rrdr_dimension_names(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
44 - const size_t dimensions = r->d;
45 - size_t c, i;
46 -
47 - buffer_json_member_add_array(wb, key);
48 - for(c = 0, i = 0; c < dimensions ; c++) {
49 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
50 - continue;
51 -
52 - buffer_json_add_array_item_string(wb, string2str(r->dn[c]));
53 - i++;
54 - }
55 - buffer_json_array_close(wb);
56 -
57 - return i;
58 -}
59 -
60 -static inline size_t rrdr_dimension_ids(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
61 - const size_t dimensions = r->d;
62 - size_t c, i;
63 -
64 - buffer_json_member_add_array(wb, key);
65 - for(c = 0, i = 0; c < dimensions ; c++) {
66 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
67 - continue;
68 -
69 - buffer_json_add_array_item_string(wb, string2str(r->di[c]));
70 - i++;
71 - }
72 - buffer_json_array_close(wb);
73 -
74 - return i;
75 -}
76 -
77 -static inline long jsonwrap_v1_chart_ids(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
78 - QUERY_TARGET *qt = r->internal.qt;
79 - const long query_used = qt->query.used;
80 - long c, i;
81 -
82 - buffer_json_member_add_array(wb, key);
83 - for (c = 0, i = 0; c < query_used; c++) {
84 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
85 - continue;
86 -
87 - QUERY_METRIC *qm = query_metric(qt, c);
88 - QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
89 - buffer_json_add_array_item_string(wb, rrdinstance_acquired_id(qi->ria));
90 - i++;
91 - }
92 - buffer_json_array_close(wb);
93 -
94 - return i;
95 -}
96 -
97 -struct summary_total_counts {
98 - size_t selected;
99 - size_t excluded;
100 - size_t queried;
101 - size_t failed;
102 -};
103 -
104 -static inline void aggregate_into_summary_totals(struct summary_total_counts *totals, QUERY_METRICS_COUNTS *metrics) {
105 - if(unlikely(!totals || !metrics))
106 - return;
107 -
108 - if(metrics->selected) {
109 - totals->selected++;
110 -
111 - if(metrics->queried)
112 - totals->queried++;
113 -
114 - else if(metrics->failed)
115 - totals->failed++;
116 - }
117 - else
118 - totals->excluded++;
119 -}
120 -
121 -static inline void query_target_total_counts(BUFFER *wb, const char *key, struct summary_total_counts *totals) {
122 - if(!totals->selected && !totals->queried && !totals->failed && !totals->excluded)
123 - return;
124 -
125 - buffer_json_member_add_object(wb, key);
126 -
127 - if(totals->selected)
128 - buffer_json_member_add_uint64(wb, "sl", totals->selected);
129 -
130 - if(totals->excluded)
131 - buffer_json_member_add_uint64(wb, "ex", totals->excluded);
132 -
133 - if(totals->queried)
134 - buffer_json_member_add_uint64(wb, "qr", totals->queried);
135 -
136 - if(totals->failed)
137 - buffer_json_member_add_uint64(wb, "fl", totals->failed);
138 -
139 - buffer_json_object_close(wb);
140 -}
141 -
142 -static inline void query_target_metric_counts(BUFFER *wb, QUERY_METRICS_COUNTS *metrics) {
143 - if(!metrics->selected && !metrics->queried && !metrics->failed && !metrics->excluded)
144 - return;
145 -
146 - buffer_json_member_add_object(wb, "ds");
147 -
148 - if(metrics->selected)
149 - buffer_json_member_add_uint64(wb, "sl", metrics->selected);
150 -
151 - if(metrics->excluded)
152 - buffer_json_member_add_uint64(wb, "ex", metrics->excluded);
153 -
154 - if(metrics->queried)
155 - buffer_json_member_add_uint64(wb, "qr", metrics->queried);
156 -
157 - if(metrics->failed)
158 - buffer_json_member_add_uint64(wb, "fl", metrics->failed);
159 -
160 - buffer_json_object_close(wb);
161 -}
162 -
163 -static inline void query_target_instance_counts(BUFFER *wb, QUERY_INSTANCES_COUNTS *instances) {
164 - if(!instances->selected && !instances->queried && !instances->failed && !instances->excluded)
165 - return;
166 -
167 - buffer_json_member_add_object(wb, "is");
168 -
169 - if(instances->selected)
170 - buffer_json_member_add_uint64(wb, "sl", instances->selected);
171 -
172 - if(instances->excluded)
173 - buffer_json_member_add_uint64(wb, "ex", instances->excluded);
174 -
175 - if(instances->queried)
176 - buffer_json_member_add_uint64(wb, "qr", instances->queried);
177 -
178 - if(instances->failed)
179 - buffer_json_member_add_uint64(wb, "fl", instances->failed);
180 -
181 - buffer_json_object_close(wb);
182 -}
183 -
184 -static inline void query_target_alerts_counts(BUFFER *wb, QUERY_ALERTS_COUNTS *alerts, const char *name, bool array) {
185 - if(!alerts->clear && !alerts->other && !alerts->critical && !alerts->warning)
186 - return;
187 -
188 - if(array)
189 - buffer_json_add_array_item_object(wb);
190 - else
191 - buffer_json_member_add_object(wb, "al");
192 -
193 - if(name)
194 - buffer_json_member_add_string(wb, "nm", name);
195 -
196 - if(alerts->clear)
197 - buffer_json_member_add_uint64(wb, "cl", alerts->clear);
198 -
199 - if(alerts->warning)
200 - buffer_json_member_add_uint64(wb, "wr", alerts->warning);
201 -
202 - if(alerts->critical)
203 - buffer_json_member_add_uint64(wb, "cr", alerts->critical);
204 -
205 - if(alerts->other)
206 - buffer_json_member_add_uint64(wb, "ot", alerts->other);
207 -
208 - buffer_json_object_close(wb);
209 -}
210 -
211 -static inline void query_target_points_statistics(BUFFER *wb, QUERY_TARGET *qt, STORAGE_POINT *sp) {
212 - if(!sp->count)
213 - return;
214 -
215 - buffer_json_member_add_object(wb, "sts");
216 -
217 - buffer_json_member_add_double(wb, "min", sp->min);
218 - buffer_json_member_add_double(wb, "max", sp->max);
219 -
220 - if(query_target_aggregatable(qt)) {
221 - buffer_json_member_add_uint64(wb, "cnt", sp->count);
222 -
223 - if(sp->sum != 0.0) {
224 - buffer_json_member_add_double(wb, "sum", sp->sum);
225 - buffer_json_member_add_double(wb, "vol", sp->sum * (NETDATA_DOUBLE) query_view_update_every(qt));
226 - }
227 -
228 - if(sp->anomaly_count != 0)
229 - buffer_json_member_add_uint64(wb, "arc", sp->anomaly_count);
230 - }
231 - else {
232 - NETDATA_DOUBLE avg = (sp->count) ? sp->sum / (NETDATA_DOUBLE)sp->count : 0.0;
233 - if(avg != 0.0)
234 - buffer_json_member_add_double(wb, "avg", avg);
235 -
236 - NETDATA_DOUBLE arp = storage_point_anomaly_rate(*sp);
237 - if(arp != 0.0)
238 - buffer_json_member_add_double(wb, "arp", arp);
239 -
240 - NETDATA_DOUBLE con = (qt->query_points.sum > 0.0) ? sp->sum * 100.0 / qt->query_points.sum : 0.0;
241 - if(con != 0.0)
242 - buffer_json_member_add_double(wb, "con", con);
243 - }
244 - buffer_json_object_close(wb);
245 -}
246 -
247 -static void query_target_summary_nodes_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals) {
248 - buffer_json_member_add_array(wb, key);
249 - for (size_t c = 0; c < qt->nodes.used; c++) {
250 - QUERY_NODE *qn = query_node(qt, c);
251 - RRDHOST *host = qn->rrdhost;
252 - buffer_json_add_array_item_object(wb);
253 - buffer_json_node_add_v2(wb, host, qn->slot, qn->duration_ut, true);
254 - query_target_instance_counts(wb, &qn->instances);
255 - query_target_metric_counts(wb, &qn->metrics);
256 - query_target_alerts_counts(wb, &qn->alerts, NULL, false);
257 - query_target_points_statistics(wb, qt, &qn->query_points);
258 - buffer_json_object_close(wb);
259 -
260 - aggregate_into_summary_totals(totals, &qn->metrics);
261 - }
262 - buffer_json_array_close(wb);
263 -}
264 -
265 -static size_t query_target_summary_contexts_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals) {
266 - buffer_json_member_add_array(wb, key);
267 - DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
268 -
269 - struct {
270 - STORAGE_POINT query_points;
271 - QUERY_INSTANCES_COUNTS instances;
272 - QUERY_METRICS_COUNTS metrics;
273 - QUERY_ALERTS_COUNTS alerts;
274 - } *z;
275 -
276 - for (long c = 0; c < (long) qt->contexts.used; c++) {
277 - QUERY_CONTEXT *qc = query_context(qt, c);
278 -
279 - z = dictionary_set(dict, rrdcontext_acquired_id(qc->rca), NULL, sizeof(*z));
280 -
281 - z->instances.selected += qc->instances.selected;
282 - z->instances.excluded += qc->instances.selected;
283 - z->instances.queried += qc->instances.queried;
284 - z->instances.failed += qc->instances.failed;
285 -
286 - z->metrics.selected += qc->metrics.selected;
287 - z->metrics.excluded += qc->metrics.excluded;
288 - z->metrics.queried += qc->metrics.queried;
289 - z->metrics.failed += qc->metrics.failed;
290 -
291 - z->alerts.clear += qc->alerts.clear;
292 - z->alerts.warning += qc->alerts.warning;
293 - z->alerts.critical += qc->alerts.critical;
294 -
295 - storage_point_merge_to(z->query_points, qc->query_points);
296 - }
297 -
298 - size_t unique_contexts = dictionary_entries(dict);
299 - dfe_start_read(dict, z) {
300 - buffer_json_add_array_item_object(wb);
301 - buffer_json_member_add_string(wb, "id", z_dfe.name);
302 - query_target_instance_counts(wb, &z->instances);
303 - query_target_metric_counts(wb, &z->metrics);
304 - query_target_alerts_counts(wb, &z->alerts, NULL, false);
305 - query_target_points_statistics(wb, qt, &z->query_points);
306 - buffer_json_object_close(wb);
307 -
308 - aggregate_into_summary_totals(totals, &z->metrics);
309 - }
310 - dfe_done(z);
311 - buffer_json_array_close(wb);
312 - dictionary_destroy(dict);
313 -
314 - return unique_contexts;
315 -}
316 -
317 -static void query_target_summary_instances_v1(BUFFER *wb, QUERY_TARGET *qt, const char *key) {
318 - char name[RRD_ID_LENGTH_MAX * 2 + 2];
319 -
320 - buffer_json_member_add_array(wb, key);
321 - DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
322 - for (long c = 0; c < (long) qt->instances.used; c++) {
323 - QUERY_INSTANCE *qi = query_instance(qt, c);
324 -
325 - snprintfz(name, RRD_ID_LENGTH_MAX * 2 + 1, "%s:%s",
326 - rrdinstance_acquired_id(qi->ria),
327 - rrdinstance_acquired_name(qi->ria));
328 -
329 - bool *set = dictionary_set(dict, name, NULL, sizeof(*set));
330 - if (!*set) {
331 - *set = true;
332 - buffer_json_add_array_item_array(wb);
333 - buffer_json_add_array_item_string(wb, rrdinstance_acquired_id(qi->ria));
334 - buffer_json_add_array_item_string(wb, rrdinstance_acquired_name(qi->ria));
335 - buffer_json_array_close(wb);
336 - }
337 - }
338 - dictionary_destroy(dict);
339 - buffer_json_array_close(wb);
340 -}
341 -
342 -static void query_target_summary_instances_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals) {
343 - buffer_json_member_add_array(wb, key);
344 - for (long c = 0; c < (long) qt->instances.used; c++) {
345 - QUERY_INSTANCE *qi = query_instance(qt, c);
346 -// QUERY_HOST *qh = query_host(qt, qi->query_host_id);
347 -
348 - buffer_json_add_array_item_object(wb);
349 - buffer_json_member_add_string(wb, "id", rrdinstance_acquired_id(qi->ria));
350 -
351 - if(!rrdinstance_acquired_id_and_name_are_same(qi->ria))
352 - buffer_json_member_add_string(wb, "nm", rrdinstance_acquired_name(qi->ria));
353 -
354 - buffer_json_member_add_uint64(wb, "ni", qi->query_host_id);
355 -// buffer_json_member_add_string(wb, "id", string2str(qi->id_fqdn));
356 -// buffer_json_member_add_string(wb, "nm", string2str(qi->name_fqdn));
357 -// buffer_json_member_add_string(wb, "lc", rrdinstance_acquired_name(qi->ria));
358 -// buffer_json_member_add_string(wb, "mg", qh->host->machine_guid);
359 -// if(qh->node_id[0])
360 -// buffer_json_member_add_string(wb, "nd", qh->node_id);
361 - query_target_metric_counts(wb, &qi->metrics);
362 - query_target_alerts_counts(wb, &qi->alerts, NULL, false);
363 - query_target_points_statistics(wb, qt, &qi->query_points);
364 - buffer_json_object_close(wb);
365 -
366 - aggregate_into_summary_totals(totals, &qi->metrics);
367 - }
368 - buffer_json_array_close(wb);
369 -}
370 -
371 -struct dimensions_sorted_walkthrough_data {
372 - BUFFER *wb;
373 - struct summary_total_counts *totals;
374 - QUERY_TARGET *qt;
375 -};
376 -
377 -struct dimensions_sorted_entry {
378 - const char *id;
379 - const char *name;
380 - STORAGE_POINT query_points;
381 - QUERY_METRICS_COUNTS metrics;
382 - uint32_t priority;
383 -};
384 -
385 -static int dimensions_sorted_walktrhough_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
386 - struct dimensions_sorted_walkthrough_data *sdwd = data;
387 - BUFFER *wb = sdwd->wb;
388 - struct summary_total_counts *totals = sdwd->totals;
389 - QUERY_TARGET *qt = sdwd->qt;
390 - struct dimensions_sorted_entry *z = value;
391 -
392 - buffer_json_add_array_item_object(wb);
393 - buffer_json_member_add_string(wb, "id", z->id);
394 - if (z->id != z->name && z->name)
395 - buffer_json_member_add_string(wb, "nm", z->name);
396 -
397 - query_target_metric_counts(wb, &z->metrics);
398 - query_target_points_statistics(wb, qt, &z->query_points);
399 - buffer_json_member_add_uint64(wb, "pri", z->priority);
400 - buffer_json_object_close(wb);
401 -
402 - aggregate_into_summary_totals(totals, &z->metrics);
403 -
404 - return 1;
405 -}
406 -
407 -int dimensions_sorted_compar(const DICTIONARY_ITEM **item1, const DICTIONARY_ITEM **item2) {
408 - struct dimensions_sorted_entry *z1 = dictionary_acquired_item_value(*item1);
409 - struct dimensions_sorted_entry *z2 = dictionary_acquired_item_value(*item2);
410 -
411 - if(z1->priority == z2->priority)
412 - return strcmp(dictionary_acquired_item_name(*item1), dictionary_acquired_item_name(*item2));
413 - else if(z1->priority < z2->priority)
414 - return -1;
415 - else
416 - return 1;
417 -}
418 -
419 -static void query_target_summary_dimensions_v12(BUFFER *wb, QUERY_TARGET *qt, const char *key, bool v2, struct summary_total_counts *totals) {
420 - char buf[RRD_ID_LENGTH_MAX * 2 + 2];
421 -
422 - buffer_json_member_add_array(wb, key);
423 - DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
424 - struct dimensions_sorted_entry *z;
425 - size_t q = 0;
426 - for (long c = 0; c < (long) qt->dimensions.used; c++) {
427 - QUERY_DIMENSION * qd = query_dimension(qt, c);
428 - RRDMETRIC_ACQUIRED *rma = qd->rma;
429 -
430 - QUERY_METRIC *qm = NULL;
431 - for( ; q < qt->query.used ;q++) {
432 - QUERY_METRIC *tqm = query_metric(qt, q);
433 - QUERY_DIMENSION *tqd = query_dimension(qt, tqm->link.query_dimension_id);
434 - if(tqd->rma != rma) break;
435 - qm = tqm;
436 - }
437 -
438 - const char *k, *id, *name;
439 -
440 - if(v2) {
441 - k = rrdmetric_acquired_name(rma);
442 - id = k;
443 - name = k;
444 - }
445 - else {
446 - snprintfz(buf, RRD_ID_LENGTH_MAX * 2 + 1, "%s:%s",
447 - rrdmetric_acquired_id(rma),
448 - rrdmetric_acquired_name(rma));
449 - k = buf;
450 - id = rrdmetric_acquired_id(rma);
451 - name = rrdmetric_acquired_name(rma);
452 - }
453 -
454 - z = dictionary_set(dict, k, NULL, sizeof(*z));
455 - if(!z->id) {
456 - z->id = id;
457 - z->name = name;
458 - z->priority = qd->priority;
459 - }
460 - else {
461 - if(qd->priority < z->priority)
462 - z->priority = qd->priority;
463 - }
464 -
465 - if(qm) {
466 - z->metrics.selected += (qm->status & RRDR_DIMENSION_SELECTED) ? 1 : 0;
467 - z->metrics.failed += (qm->status & RRDR_DIMENSION_FAILED) ? 1 : 0;
468 -
469 - if(qm->status & RRDR_DIMENSION_QUERIED) {
470 - z->metrics.queried++;
471 - storage_point_merge_to(z->query_points, qm->query_points);
472 - }
473 - }
474 - else
475 - z->metrics.excluded++;
476 - }
477 -
478 - if(v2) {
479 - struct dimensions_sorted_walkthrough_data t = {
480 - .wb = wb,
481 - .totals = totals,
482 - .qt = qt,
483 - };
484 - dictionary_sorted_walkthrough_rw(dict, DICTIONARY_LOCK_READ, dimensions_sorted_walktrhough_cb,
485 - &t, dimensions_sorted_compar);
486 - }
487 - else {
488 - // v1
489 - dfe_start_read(dict, z) {
490 - buffer_json_add_array_item_array(wb);
491 - buffer_json_add_array_item_string(wb, z->id);
492 - buffer_json_add_array_item_string(wb, z->name);
493 - buffer_json_array_close(wb);
494 - }
495 - dfe_done(z);
496 - }
497 - dictionary_destroy(dict);
498 - buffer_json_array_close(wb);
499 -}
500 -
501 -struct rrdlabels_formatting_v2 {
502 - DICTIONARY *keys;
503 - QUERY_INSTANCE *qi;
504 - bool v2;
505 -};
506 -
507 -struct rrdlabels_keys_dict_entry {
508 - const char *name;
509 - DICTIONARY *values;
510 - STORAGE_POINT query_points;
511 - QUERY_METRICS_COUNTS metrics;
512 -};
513 -
514 -struct rrdlabels_key_value_dict_entry {
515 - const char *key;
516 - const char *value;
517 - STORAGE_POINT query_points;
518 - QUERY_METRICS_COUNTS metrics;
519 -};
520 -
521 -static int rrdlabels_formatting_v2(const char *name, const char *value, RRDLABEL_SRC ls __maybe_unused, void *data) {
522 - struct rrdlabels_formatting_v2 *t = data;
523 -
524 - struct rrdlabels_keys_dict_entry *d = dictionary_set(t->keys, name, NULL, sizeof(*d));
525 - if(!d->values) {
526 - d->name = name;
527 - d->values = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
528 - }
529 -
530 - char n[RRD_ID_LENGTH_MAX * 2 + 2];
531 - snprintfz(n, RRD_ID_LENGTH_MAX * 2, "%s:%s", name, value);
532 -
533 - struct rrdlabels_key_value_dict_entry *z = dictionary_set(d->values, n, NULL, sizeof(*z));
534 - if(!z->key) {
535 - z->key = name;
536 - z->value = value;
537 - }
538 -
539 - if(t->v2) {
540 - QUERY_INSTANCE *qi = t->qi;
541 -
542 - z->metrics.selected += qi->metrics.selected;
543 - z->metrics.excluded += qi->metrics.excluded;
544 - z->metrics.queried += qi->metrics.queried;
545 - z->metrics.failed += qi->metrics.failed;
546 -
547 - d->metrics.selected += qi->metrics.selected;
548 - d->metrics.excluded += qi->metrics.excluded;
549 - d->metrics.queried += qi->metrics.queried;
550 - d->metrics.failed += qi->metrics.failed;
551 -
552 - storage_point_merge_to(z->query_points, qi->query_points);
553 - storage_point_merge_to(d->query_points, qi->query_points);
554 - }
555 -
556 - return 1;
557 -}
558 -
559 -static void query_target_summary_labels_v12(BUFFER *wb, QUERY_TARGET *qt, const char *key, bool v2, struct summary_total_counts *key_totals, struct summary_total_counts *value_totals) {
560 - buffer_json_member_add_array(wb, key);
561 - struct rrdlabels_formatting_v2 t = {
562 - .keys = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE),
563 - .v2 = v2,
564 - };
565 - for (long c = 0; c < (long) qt->instances.used; c++) {
566 - QUERY_INSTANCE *qi = query_instance(qt, c);
567 - RRDINSTANCE_ACQUIRED *ria = qi->ria;
568 - t.qi = qi;
569 - rrdlabels_walkthrough_read(rrdinstance_acquired_labels(ria), rrdlabels_formatting_v2, &t);
570 - }
571 - struct rrdlabels_keys_dict_entry *d;
572 - dfe_start_read(t.keys, d) {
573 - if(v2) {
574 - buffer_json_add_array_item_object(wb);
575 - buffer_json_member_add_string(wb, "id", d_dfe.name);
576 - query_target_metric_counts(wb, &d->metrics);
577 - query_target_points_statistics(wb, qt, &d->query_points);
578 - aggregate_into_summary_totals(key_totals, &d->metrics);
579 - buffer_json_member_add_array(wb, "vl");
580 - }
581 - struct rrdlabels_key_value_dict_entry *z;
582 - dfe_start_read(d->values, z){
583 - if (v2) {
584 - buffer_json_add_array_item_object(wb);
585 - buffer_json_member_add_string(wb, "id", z->value);
586 - query_target_metric_counts(wb, &z->metrics);
587 - query_target_points_statistics(wb, qt, &z->query_points);
588 - buffer_json_object_close(wb);
589 - aggregate_into_summary_totals(value_totals, &z->metrics);
590 - } else {
591 - buffer_json_add_array_item_array(wb);
592 - buffer_json_add_array_item_string(wb, z->key);
593 - buffer_json_add_array_item_string(wb, z->value);
594 - buffer_json_array_close(wb);
595 - }
596 - }
597 - dfe_done(z);
598 - dictionary_destroy(d->values);
599 - if(v2) {
600 - buffer_json_array_close(wb);
601 - buffer_json_object_close(wb);
602 - }
603 - }
604 - dfe_done(d);
605 - dictionary_destroy(t.keys);
606 - buffer_json_array_close(wb);
607 -}
608 -
609 -static void query_target_summary_alerts_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key) {
610 - buffer_json_member_add_array(wb, key);
611 - QUERY_ALERTS_COUNTS *z;
612 -
613 - DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
614 - for (long c = 0; c < (long) qt->instances.used; c++) {
615 - QUERY_INSTANCE *qi = query_instance(qt, c);
616 - RRDSET *st = rrdinstance_acquired_rrdset(qi->ria);
617 - if (st) {
618 - rw_spinlock_read_lock(&st->alerts.spinlock);
619 - if (st->alerts.base) {
620 - for (RRDCALC *rc = st->alerts.base; rc; rc = rc->next) {
621 - z = dictionary_set(dict, string2str(rc->config.name), NULL, sizeof(*z));
622 -
623 - switch(rc->status) {
624 - case RRDCALC_STATUS_CLEAR:
625 - z->clear++;
626 - break;
627 -
628 - case RRDCALC_STATUS_WARNING:
629 - z->warning++;
630 - break;
631 -
632 - case RRDCALC_STATUS_CRITICAL:
633 - z->critical++;
634 - break;
635 -
636 - default:
637 - case RRDCALC_STATUS_UNINITIALIZED:
638 - case RRDCALC_STATUS_UNDEFINED:
639 - case RRDCALC_STATUS_REMOVED:
640 - z->other++;
641 - break;
642 - }
643 - }
644 - }
645 - rw_spinlock_read_unlock(&st->alerts.spinlock);
646 - }
647 - }
648 - dfe_start_read(dict, z)
649 - query_target_alerts_counts(wb, z, z_dfe.name, true);
650 - dfe_done(z);
651 - dictionary_destroy(dict);
652 - buffer_json_array_close(wb); // alerts
653 -}
654 -
655 -static inline void query_target_functions(BUFFER *wb, const char *key, RRDR *r) {
656 - QUERY_TARGET *qt = r->internal.qt;
657 - const long query_used = qt->query.used;
658 -
659 - DICTIONARY *funcs = dictionary_create(DICT_OPTION_SINGLE_THREADED|DICT_OPTION_DONT_OVERWRITE_VALUE);
660 - RRDINSTANCE_ACQUIRED *ria = NULL;
661 - for (long c = 0; c < query_used ; c++) {
662 - QUERY_METRIC *qm = query_metric(qt, c);
663 - QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
664 - if(qi->ria == ria)
665 - continue;
666 -
667 - ria = qi->ria;
668 - chart_functions_to_dict(rrdinstance_acquired_functions(ria), funcs, NULL, 0);
669 - }
670 -
671 - buffer_json_member_add_array(wb, key);
672 - void *t; (void)t;
673 - dfe_start_read(funcs, t)
674 - buffer_json_add_array_item_string(wb, t_dfe.name);
675 - dfe_done(t);
676 - dictionary_destroy(funcs);
677 - buffer_json_array_close(wb);
678 -}
679 -
680 -static inline long query_target_chart_labels_filter_v1(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
681 - QUERY_TARGET *qt = r->internal.qt;
682 - const long query_used = qt->query.used;
683 - long c, i = 0;
684 -
685 - buffer_json_member_add_object(wb, key);
686 -
687 - SIMPLE_PATTERN *pattern = qt->instances.chart_label_key_pattern;
688 - char *label_key = NULL;
689 - while (pattern && (label_key = simple_pattern_iterate(&pattern))) {
690 - buffer_json_member_add_array(wb, label_key);
691 -
692 - for (c = 0, i = 0; c < query_used; c++) {
693 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
694 - continue;
695 -
696 - QUERY_METRIC *qm = query_metric(qt, c);
697 - QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
698 - rrdlabels_value_to_buffer_array_item_or_null(rrdinstance_acquired_labels(qi->ria), wb, label_key);
699 - i++;
700 - }
701 - buffer_json_array_close(wb);
702 - }
703 -
704 - buffer_json_object_close(wb);
705 -
706 - return i;
707 -}
708 -
709 -static inline long query_target_metrics_latest_values(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
710 - QUERY_TARGET *qt = r->internal.qt;
711 - const long query_used = qt->query.used;
712 - long c, i;
713 -
714 - buffer_json_member_add_array(wb, key);
715 -
716 - for(c = 0, i = 0; c < query_used ;c++) {
717 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
718 - continue;
719 -
720 - QUERY_METRIC *qm = query_metric(qt, c);
721 - QUERY_DIMENSION *qd = query_dimension(qt, qm->link.query_dimension_id);
722 - buffer_json_add_array_item_double(wb, rrdmetric_acquired_last_stored_value(qd->rma));
723 - i++;
724 - }
725 -
726 - buffer_json_array_close(wb);
727 -
728 - return i;
729 -}
730 -
731 -static inline size_t rrdr_dimension_view_latest_values(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
732 - buffer_json_member_add_array(wb, key);
733 -
734 - size_t c, i;
735 - for(c = 0, i = 0; c < r->d ; c++) {
736 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
737 - continue;
738 -
739 - i++;
740 -
741 - NETDATA_DOUBLE *cn = &r->v[ (rrdr_rows(r) - 1) * r->d ];
742 - RRDR_VALUE_FLAGS *co = &r->o[ (rrdr_rows(r) - 1) * r->d ];
743 - NETDATA_DOUBLE n = cn[c];
744 -
745 - if(co[c] & RRDR_VALUE_EMPTY) {
746 - if(options & RRDR_OPTION_NULL2ZERO)
747 - buffer_json_add_array_item_double(wb, 0.0);
748 - else
749 - buffer_json_add_array_item_double(wb, NAN);
750 - }
751 - else
752 - buffer_json_add_array_item_double(wb, n);
753 - }
754 -
755 - buffer_json_array_close(wb);
756 -
757 - return i;
758 -}
759 -
760 -static inline void rrdr_dimension_query_points_statistics(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options, bool dview) {
761 - STORAGE_POINT *sp = (dview) ? r->dview : r->dqp;
762 - NETDATA_DOUBLE anomaly_rate_multiplier = (dview) ? RRDR_DVIEW_ANOMALY_COUNT_MULTIPLIER : 1.0;
763 -
764 - if(unlikely(!sp))
765 - return;
766 -
767 - if(key)
768 - buffer_json_member_add_object(wb, key);
769 -
770 - buffer_json_member_add_array(wb, "min");
771 - for(size_t c = 0; c < r->d ; c++) {
772 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
773 - continue;
774 -
775 - buffer_json_add_array_item_double(wb, sp[c].min);
776 - }
777 - buffer_json_array_close(wb);
778 -
779 - buffer_json_member_add_array(wb, "max");
780 - for(size_t c = 0; c < r->d ; c++) {
781 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
782 - continue;
783 -
784 - buffer_json_add_array_item_double(wb, sp[c].max);
785 - }
786 - buffer_json_array_close(wb);
787 -
788 - if(options & RRDR_OPTION_RETURN_RAW) {
789 - buffer_json_member_add_array(wb, "sum");
790 - for(size_t c = 0; c < r->d ; c++) {
791 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
792 - continue;
793 -
794 - buffer_json_add_array_item_double(wb, sp[c].sum);
795 - }
796 - buffer_json_array_close(wb);
797 -
798 - buffer_json_member_add_array(wb, "cnt");
799 - for(size_t c = 0; c < r->d ; c++) {
800 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
801 - continue;
802 -
803 - buffer_json_add_array_item_uint64(wb, sp[c].count);
804 - }
805 - buffer_json_array_close(wb);
806 -
807 - buffer_json_member_add_array(wb, "arc");
808 - for(size_t c = 0; c < r->d ; c++) {
809 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
810 - continue;
811 -
812 - buffer_json_add_array_item_uint64(wb, storage_point_anomaly_rate(sp[c]) / anomaly_rate_multiplier / 100.0 * sp[c].count);
813 - }
814 - buffer_json_array_close(wb);
815 - }
816 - else {
817 - NETDATA_DOUBLE sum = 0.0;
818 - for(size_t c = 0; c < r->d ; c++) {
819 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
820 - continue;
821 -
822 - sum += ABS(sp[c].sum);
823 - }
824 -
825 - buffer_json_member_add_array(wb, "avg");
826 - for(size_t c = 0; c < r->d ; c++) {
827 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
828 - continue;
829 -
830 - buffer_json_add_array_item_double(wb, storage_point_average_value(sp[c]));
831 - }
832 - buffer_json_array_close(wb);
833 -
834 - buffer_json_member_add_array(wb, "arp");
835 - for(size_t c = 0; c < r->d ; c++) {
836 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
837 - continue;
838 -
839 - buffer_json_add_array_item_double(wb, storage_point_anomaly_rate(sp[c]) / anomaly_rate_multiplier);
840 - }
841 - buffer_json_array_close(wb);
842 -
843 - buffer_json_member_add_array(wb, "con");
844 - for(size_t c = 0; c < r->d ; c++) {
845 - if (!rrdr_dimension_should_be_exposed(r->od[c], options))
846 - continue;
847 -
848 - NETDATA_DOUBLE con = (sum > 0.0) ? ABS(sp[c].sum) * 100.0 / sum : 0.0;
849 - buffer_json_add_array_item_double(wb, con);
850 - }
851 - buffer_json_array_close(wb);
852 - }
853 -
854 - if(key)
855 - buffer_json_object_close(wb);
856 -}
857 -
858 -void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb) {
859 - QUERY_TARGET *qt = r->internal.qt;
860 - DATASOURCE_FORMAT format = qt->request.format;
861 - RRDR_OPTIONS options = qt->window.options;
862 -
863 - long rows = rrdr_rows(r);
864 -
865 - char kq[2] = "", // key quote
866 - sq[2] = ""; // string quote
867 -
868 - if( options & RRDR_OPTION_GOOGLE_JSON ) {
869 - kq[0] = '\0';
870 - sq[0] = '\'';
871 - }
872 - else {
873 - kq[0] = '"';
874 - sq[0] = '"';
875 - }
876 -
877 - buffer_json_initialize(
878 - wb, kq, sq, 0, true, (options & RRDR_OPTION_MINIFY) ? BUFFER_JSON_OPTIONS_MINIFY : BUFFER_JSON_OPTIONS_DEFAULT);
879 -
880 - buffer_json_member_add_uint64(wb, "api", 1);
881 - buffer_json_member_add_string(wb, "id", qt->id);
882 - buffer_json_member_add_string(wb, "name", qt->id);
883 - buffer_json_member_add_time_t(wb, "view_update_every", r->view.update_every);
884 - buffer_json_member_add_time_t(wb, "update_every", qt->db.minimum_latest_update_every_s);
885 - buffer_json_member_add_time_t(wb, "first_entry", qt->db.first_time_s);
886 - buffer_json_member_add_time_t(wb, "last_entry", qt->db.last_time_s);
887 - buffer_json_member_add_time_t(wb, "after", r->view.after);
888 - buffer_json_member_add_time_t(wb, "before", r->view.before);
889 - buffer_json_member_add_string(wb, "group", time_grouping_tostring(qt->request.time_group_method));
890 - rrdr_options_to_buffer_json_array(wb, "options", options);
891 -
892 - if(!rrdr_dimension_names(wb, "dimension_names", r, options))
893 - rows = 0;
894 -
895 - if(!rrdr_dimension_ids(wb, "dimension_ids", r, options))
896 - rows = 0;
897 -
898 - if (options & RRDR_OPTION_ALL_DIMENSIONS) {
899 - query_target_summary_instances_v1(wb, qt, "full_chart_list");
900 - query_target_summary_dimensions_v12(wb, qt, "full_dimension_list", false, NULL);
901 - query_target_summary_labels_v12(wb, qt, "full_chart_labels", false, NULL, NULL);
902 - }
903 -
904 - query_target_functions(wb, "functions", r);
905 -
906 - if (!qt->request.st && !jsonwrap_v1_chart_ids(wb, "chart_ids", r, options))
907 - rows = 0;
908 -
909 - if (qt->instances.chart_label_key_pattern && !query_target_chart_labels_filter_v1(wb, "chart_labels", r, options))
910 - rows = 0;
911 -
912 - if(!query_target_metrics_latest_values(wb, "latest_values", r, options))
913 - rows = 0;
914 -
915 - size_t dimensions = rrdr_dimension_view_latest_values(wb, "view_latest_values", r, options);
916 - if(!dimensions)
917 - rows = 0;
918 -
919 - buffer_json_member_add_uint64(wb, "dimensions", dimensions);
920 - buffer_json_member_add_uint64(wb, "points", rows);
921 - buffer_json_member_add_string(wb, "format", rrdr_format_to_string(format));
922 -
923 - buffer_json_member_add_array(wb, "db_points_per_tier");
924 - for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++)
925 - buffer_json_add_array_item_uint64(wb, qt->db.tiers[tier].points);
926 - buffer_json_array_close(wb);
927 -
928 - if(options & RRDR_OPTION_DEBUG)
929 - jsonwrap_query_plan(r, wb);
930 -}
931 -
932 -static void rrdset_rrdcalc_entries_v2(BUFFER *wb, RRDINSTANCE_ACQUIRED *ria) {
933 - RRDSET *st = rrdinstance_acquired_rrdset(ria);
934 - if(st) {
935 - rw_spinlock_read_lock(&st->alerts.spinlock);
936 - if(st->alerts.base) {
937 - buffer_json_member_add_object(wb, "alerts");
938 - for(RRDCALC *rc = st->alerts.base; rc ;rc = rc->next) {
939 - if(rc->status < RRDCALC_STATUS_CLEAR)
940 - continue;
941 -
942 - buffer_json_member_add_object(wb, string2str(rc->config.name));
943 - buffer_json_member_add_string(wb, "st", rrdcalc_status2string(rc->status));
944 - buffer_json_member_add_double(wb, "vl", rc->value);
945 - buffer_json_member_add_string(wb, "un", string2str(rc->config.units));
946 - buffer_json_object_close(wb);
947 - }
948 - buffer_json_object_close(wb);
949 - }
950 - rw_spinlock_read_unlock(&st->alerts.spinlock);
951 - }
952 -}
953 -
954 -static void query_target_combined_units_v2(BUFFER *wb, QUERY_TARGET *qt, size_t contexts, bool ignore_percentage) {
955 - if(!ignore_percentage && query_target_has_percentage_units(qt)) {
956 - buffer_json_member_add_string(wb, "units", "%");
957 - }
958 - else if(contexts == 1) {
959 - buffer_json_member_add_string(wb, "units", rrdcontext_acquired_units(qt->contexts.array[0].rca));
960 - }
961 - else if(contexts > 1) {
962 - DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
963 - for(size_t c = 0; c < qt->contexts.used ;c++)
964 - dictionary_set(dict, rrdcontext_acquired_units(qt->contexts.array[c].rca), NULL, 0);
965 -
966 - if(dictionary_entries(dict) == 1)
967 - buffer_json_member_add_string(wb, "units", rrdcontext_acquired_units(qt->contexts.array[0].rca));
968 - else {
969 - buffer_json_member_add_array(wb, "units");
970 - const char *s;
971 - dfe_start_read(dict, s)
972 - buffer_json_add_array_item_string(wb, s_dfe.name);
973 - dfe_done(s);
974 - buffer_json_array_close(wb);
975 - }
976 - dictionary_destroy(dict);
977 - }
978 -}
979 -
980 -static void query_target_combined_chart_type(BUFFER *wb, QUERY_TARGET *qt, size_t contexts) {
981 - if(contexts >= 1)
982 - buffer_json_member_add_string(wb, "chart_type", rrdset_type_name(rrdcontext_acquired_chart_type(qt->contexts.array[0].rca)));
983 -}
984 -
985 -static void rrdr_grouped_by_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options __maybe_unused) {
986 - QUERY_TARGET *qt = r->internal.qt;
987 -
988 - buffer_json_member_add_array(wb, key);
989 -
990 - // find the deeper group-by
991 - ssize_t g = 0;
992 - for(g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
993 - if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_NONE)
994 - break;
995 - }
996 -
997 - if(g > 0)
998 - g--;
999 -
1000 - RRDR_GROUP_BY group_by = qt->request.group_by[g].group_by;
1001 -
1002 - if(group_by & RRDR_GROUP_BY_SELECTED)
1003 - buffer_json_add_array_item_string(wb, "selected");
1004 -
1005 - else if(group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
1006 - buffer_json_add_array_item_string(wb, "percentage-of-instance");
1007 -
1008 - else {
1009 -
1010 - if(group_by & RRDR_GROUP_BY_DIMENSION)
1011 - buffer_json_add_array_item_string(wb, "dimension");
1012 -
1013 - if(group_by & RRDR_GROUP_BY_INSTANCE)
1014 - buffer_json_add_array_item_string(wb, "instance");
1015 -
1016 - if(group_by & RRDR_GROUP_BY_LABEL) {
1017 - BUFFER *b = buffer_create(0, NULL);
1018 - for (size_t l = 0; l < qt->group_by[g].used; l++) {
1019 - buffer_flush(b);
1020 - buffer_fast_strcat(b, "label:", 6);
1021 - buffer_strcat(b, qt->group_by[g].label_keys[l]);
1022 - buffer_json_add_array_item_string(wb, buffer_tostring(b));
1023 - }
1024 - buffer_free(b);
1025 - }
1026 -
1027 - if(group_by & RRDR_GROUP_BY_NODE)
1028 - buffer_json_add_array_item_string(wb, "node");
1029 -
1030 - if(group_by & RRDR_GROUP_BY_CONTEXT)
1031 - buffer_json_add_array_item_string(wb, "context");
1032 -
1033 - if(group_by & RRDR_GROUP_BY_UNITS)
1034 - buffer_json_add_array_item_string(wb, "units");
1035 - }
1036 -
1037 - buffer_json_array_close(wb); // group_by_order
1038 -}
1039 -
1040 -static void rrdr_dimension_units_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options, bool ignore_percentage) {
1041 - if(!r->du)
1042 - return;
1043 -
1044 - bool percentage = !ignore_percentage && query_target_has_percentage_units(r->internal.qt);
1045 -
1046 - buffer_json_member_add_array(wb, key);
1047 - for(size_t c = 0; c < r->d ; c++) {
1048 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
1049 - continue;
1050 -
1051 - if(percentage)
1052 - buffer_json_add_array_item_string(wb, "%");
1053 - else
1054 - buffer_json_add_array_item_string(wb, string2str(r->du[c]));
1055 - }
1056 - buffer_json_array_close(wb);
1057 -}
1058 -
1059 -static void rrdr_dimension_priority_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
1060 - if(!r->dp)
1061 - return;
1062 -
1063 - buffer_json_member_add_array(wb, key);
1064 - for(size_t c = 0; c < r->d ; c++) {
1065 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
1066 - continue;
1067 -
1068 - buffer_json_add_array_item_uint64(wb, r->dp[c]);
1069 - }
1070 - buffer_json_array_close(wb);
1071 -}
1072 -
1073 -static void rrdr_dimension_aggregated_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
1074 - if(!r->dgbc)
1075 - return;
1076 -
1077 - buffer_json_member_add_array(wb, key);
1078 - for(size_t c = 0; c < r->d ;c++) {
1079 - if(!rrdr_dimension_should_be_exposed(r->od[c], options))
1080 - continue;
1081 -
1082 - buffer_json_add_array_item_uint64(wb, r->dgbc[c]);
1083 - }
1084 - buffer_json_array_close(wb);
1085 -}
1086 -
1087 -static void query_target_title(BUFFER *wb, QUERY_TARGET *qt, size_t contexts) {
1088 - if(contexts == 1) {
1089 - buffer_json_member_add_string(wb, "title", rrdcontext_acquired_title(qt->contexts.array[0].rca));
1090 - }
1091 - else if(contexts > 1) {
1092 - BUFFER *t = buffer_create(0, NULL);
1093 - DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
1094 -
1095 - buffer_strcat(t, "Chart for contexts: ");
1096 -
1097 - size_t added = 0;
1098 - for(size_t c = 0; c < qt->contexts.used ;c++) {
1099 - bool *set = dictionary_set(dict, rrdcontext_acquired_id(qt->contexts.array[c].rca), NULL, sizeof(*set));
1100 - if(!*set) {
1101 - *set = true;
1102 - if(added)
1103 - buffer_fast_strcat(t, ", ", 2);
1104 -
1105 - buffer_strcat(t, rrdcontext_acquired_id(qt->contexts.array[c].rca));
1106 - added++;
1107 - }
1108 - }
1109 - buffer_json_member_add_string(wb, "title", buffer_tostring(t));
1110 - dictionary_destroy(dict);
1111 - buffer_free(t);
1112 - }
1113 -}
1114 -
1115 -static void query_target_detailed_objects_tree(BUFFER *wb, RRDR *r, RRDR_OPTIONS options) {
1116 - QUERY_TARGET *qt = r->internal.qt;
1117 - buffer_json_member_add_object(wb, "nodes");
1118 -
1119 - time_t now_s = now_realtime_sec();
1120 - RRDHOST *last_host = NULL;
1121 - RRDCONTEXT_ACQUIRED *last_rca = NULL;
1122 - RRDINSTANCE_ACQUIRED *last_ria = NULL;
1123 -
1124 - size_t h = 0, c = 0, i = 0, m = 0, q = 0;
1125 - for(; h < qt->nodes.used ; h++) {
1126 - QUERY_NODE *qn = query_node(qt, h);
1127 - RRDHOST *host = qn->rrdhost;
1128 -
1129 - for( ;c < qt->contexts.used ;c++) {
1130 - QUERY_CONTEXT *qc = query_context(qt, c);
1131 - RRDCONTEXT_ACQUIRED *rca = qc->rca;
1132 - if(!rrdcontext_acquired_belongs_to_host(rca, host)) break;
1133 -
1134 - for( ;i < qt->instances.used ;i++) {
1135 - QUERY_INSTANCE *qi = query_instance(qt, i);
1136 - RRDINSTANCE_ACQUIRED *ria = qi->ria;
1137 - if(!rrdinstance_acquired_belongs_to_context(ria, rca)) break;
1138 -
1139 - for( ; m < qt->dimensions.used ; m++) {
1140 - QUERY_DIMENSION *qd = query_dimension(qt, m);
1141 - RRDMETRIC_ACQUIRED *rma = qd->rma;
1142 - if(!rrdmetric_acquired_belongs_to_instance(rma, ria)) break;
1143 -
1144 - QUERY_METRIC *qm = NULL;
1145 - bool queried = false;
1146 - for( ; q < qt->query.used ;q++) {
1147 - QUERY_METRIC *tqm = query_metric(qt, q);
1148 - QUERY_DIMENSION *tqd = query_dimension(qt, tqm->link.query_dimension_id);
1149 - if(tqd->rma != rma) break;
1150 -
1151 - queried = tqm->status & RRDR_DIMENSION_QUERIED;
1152 - qm = tqm;
1153 - }
1154 -
1155 - if(!queried & !(options & RRDR_OPTION_ALL_DIMENSIONS))
1156 - continue;
1157 -
1158 - if(host != last_host) {
1159 - if(last_host) {
1160 - if(last_rca) {
1161 - if(last_ria) {
1162 - buffer_json_object_close(wb); // dimensions
1163 - buffer_json_object_close(wb); // instance
1164 - last_ria = NULL;
1165 - }
1166 - buffer_json_object_close(wb); // instances
1167 - buffer_json_object_close(wb); // context
1168 - last_rca = NULL;
1169 - }
1170 - buffer_json_object_close(wb); // contexts
1171 - buffer_json_object_close(wb); // host
1172 - }
1173 -
1174 - buffer_json_member_add_object(wb, host->machine_guid);
1175 - if(qn->node_id[0])
1176 - buffer_json_member_add_string(wb, "nd", qn->node_id);
1177 - buffer_json_member_add_uint64(wb, "ni", qn->slot);
1178 - buffer_json_member_add_string(wb, "nm", rrdhost_hostname(host));
1179 - buffer_json_member_add_object(wb, "contexts");
1180 -
1181 - last_host = host;
1182 - }
1183 -
1184 - if(rca != last_rca) {
1185 - if(last_rca) {
1186 - if(last_ria) {
1187 - buffer_json_object_close(wb); // dimensions
1188 - buffer_json_object_close(wb); // instance
1189 - last_ria = NULL;
1190 - }
1191 - buffer_json_object_close(wb); // instances
1192 - buffer_json_object_close(wb); // context
1193 - last_rca = NULL;
1194 - }
1195 -
1196 - buffer_json_member_add_object(wb, rrdcontext_acquired_id(rca));
1197 - buffer_json_member_add_object(wb, "instances");
1198 -
1199 - last_rca = rca;
1200 - }
1201 -
1202 - if(ria != last_ria) {
1203 - if(last_ria) {
1204 - buffer_json_object_close(wb); // dimensions
1205 - buffer_json_object_close(wb); // instance
1206 - last_ria = NULL;
1207 - }
1208 -
1209 - buffer_json_member_add_object(wb, rrdinstance_acquired_id(ria));
1210 - buffer_json_member_add_string(wb, "nm", rrdinstance_acquired_name(ria));
1211 - buffer_json_member_add_time_t(wb, "ue", rrdinstance_acquired_update_every(ria));
1212 - RRDLABELS *labels = rrdinstance_acquired_labels(ria);
1213 - if(labels) {
1214 - buffer_json_member_add_object(wb, "labels");
1215 - rrdlabels_to_buffer_json_members(labels, wb);
1216 - buffer_json_object_close(wb);
1217 - }
1218 - rrdset_rrdcalc_entries_v2(wb, ria);
1219 - buffer_json_member_add_object(wb, "dimensions");
1220 -
1221 - last_ria = ria;
1222 - }
1223 -
1224 - buffer_json_member_add_object(wb, rrdmetric_acquired_id(rma));
1225 - {
1226 - buffer_json_member_add_string(wb, "nm", rrdmetric_acquired_name(rma));
1227 - buffer_json_member_add_uint64(wb, "qr", queried ? 1 : 0);
1228 - time_t first_entry_s = rrdmetric_acquired_first_entry(rma);
1229 - time_t last_entry_s = rrdmetric_acquired_last_entry(rma);
1230 - buffer_json_member_add_time_t(wb, "fe", first_entry_s);
1231 - buffer_json_member_add_time_t(wb, "le", last_entry_s ? last_entry_s : now_s);
1232 -
1233 - if(qm) {
1234 - if(qm->status & RRDR_DIMENSION_GROUPED) {
1235 - // buffer_json_member_add_string(wb, "grouped_as_id", string2str(qm->grouped_as.id));
1236 - buffer_json_member_add_string(wb, "as", string2str(qm->grouped_as.name));
1237 - }
1238 -
1239 - query_target_points_statistics(wb, qt, &qm->query_points);
1240 -
1241 - if(options & RRDR_OPTION_DEBUG)
1242 - jsonwrap_query_metric_plan(wb, qm);
1243 - }
1244 - }
1245 - buffer_json_object_close(wb); // metric
1246 - }
1247 - }
1248 - }
1249 - }
1250 -
1251 - if(last_host) {
1252 - if(last_rca) {
1253 - if(last_ria) {
1254 - buffer_json_object_close(wb); // dimensions
1255 - buffer_json_object_close(wb); // instance
1256 - last_ria = NULL;
1257 - }
1258 - buffer_json_object_close(wb); // instances
1259 - buffer_json_object_close(wb); // context
1260 - last_rca = NULL;
1261 - }
1262 - buffer_json_object_close(wb); // contexts
1263 - buffer_json_object_close(wb); // host
1264 - last_host = NULL;
1265 - }
1266 - buffer_json_object_close(wb); // hosts
1267 -}
1268 -
1269 -void version_hashes_api_v2(BUFFER *wb, struct query_versions *versions) {
1270 - buffer_json_member_add_object(wb, "versions");
1271 - buffer_json_member_add_uint64(wb, "routing_hard_hash", 1);
1272 - buffer_json_member_add_uint64(wb, "nodes_hard_hash", dictionary_version(rrdhost_root_index));
1273 - buffer_json_member_add_uint64(wb, "contexts_hard_hash", versions->contexts_hard_hash);
1274 - buffer_json_member_add_uint64(wb, "contexts_soft_hash", versions->contexts_soft_hash);
1275 - buffer_json_member_add_uint64(wb, "alerts_hard_hash", versions->alerts_hard_hash);
1276 - buffer_json_member_add_uint64(wb, "alerts_soft_hash", versions->alerts_soft_hash);
1277 - buffer_json_object_close(wb);
1278 -}
1279 -
1280 -void rrdr_json_wrapper_begin2(RRDR *r, BUFFER *wb) {
1281 - QUERY_TARGET *qt = r->internal.qt;
1282 - RRDR_OPTIONS options = qt->window.options;
1283 -
1284 - char kq[2] = "\"", // key quote
1285 - sq[2] = "\""; // string quote
1286 -
1287 - if(unlikely(options & RRDR_OPTION_GOOGLE_JSON)) {
1288 - kq[0] = '\0';
1289 - sq[0] = '\'';
1290 - }
1291 -
1292 - buffer_json_initialize(
1293 - wb, kq, sq, 0, true, (options & RRDR_OPTION_MINIFY) ? BUFFER_JSON_OPTIONS_MINIFY : BUFFER_JSON_OPTIONS_DEFAULT);
1294 - buffer_json_member_add_uint64(wb, "api", 2);
1295 -
1296 - if(options & RRDR_OPTION_DEBUG) {
1297 - buffer_json_member_add_string(wb, "id", qt->id);
1298 - buffer_json_member_add_object(wb, "request");
1299 - {
1300 - buffer_json_member_add_string(wb, "format", rrdr_format_to_string(qt->request.format));
1301 - rrdr_options_to_buffer_json_array(wb, "options", qt->request.options);
1302 -
1303 - buffer_json_member_add_object(wb, "scope");
1304 - buffer_json_member_add_string(wb, "scope_nodes", qt->request.scope_nodes);
1305 - buffer_json_member_add_string(wb, "scope_contexts", qt->request.scope_contexts);
1306 - buffer_json_object_close(wb); // scope
1307 -
1308 - buffer_json_member_add_object(wb, "selectors");
1309 - if (qt->request.host)
1310 - buffer_json_member_add_string(wb, "nodes", rrdhost_hostname(qt->request.host));
1311 - else
1312 - buffer_json_member_add_string(wb, "nodes", qt->request.nodes);
1313 - buffer_json_member_add_string(wb, "contexts", qt->request.contexts);
1314 - buffer_json_member_add_string(wb, "instances", qt->request.instances);
1315 - buffer_json_member_add_string(wb, "dimensions", qt->request.dimensions);
1316 - buffer_json_member_add_string(wb, "labels", qt->request.labels);
1317 - buffer_json_member_add_string(wb, "alerts", qt->request.alerts);
1318 - buffer_json_object_close(wb); // selectors
1319 -
1320 - buffer_json_member_add_object(wb, "window");
1321 - buffer_json_member_add_time_t(wb, "after", qt->request.after);
1322 - buffer_json_member_add_time_t(wb, "before", qt->request.before);
1323 - buffer_json_member_add_uint64(wb, "points", qt->request.points);
1324 - if (qt->request.options & RRDR_OPTION_SELECTED_TIER)
1325 - buffer_json_member_add_uint64(wb, "tier", qt->request.tier);
1326 - else
1327 - buffer_json_member_add_string(wb, "tier", NULL);
1328 - buffer_json_object_close(wb); // window
1329 -
1330 - buffer_json_member_add_object(wb, "aggregations");
1331 - {
1332 - buffer_json_member_add_object(wb, "time");
1333 - buffer_json_member_add_string(wb, "time_group", time_grouping_tostring(qt->request.time_group_method));
1334 - buffer_json_member_add_string(wb, "time_group_options", qt->request.time_group_options);
1335 - if (qt->request.resampling_time > 0)
1336 - buffer_json_member_add_time_t(wb, "time_resampling", qt->request.resampling_time);
1337 - else
1338 - buffer_json_member_add_string(wb, "time_resampling", NULL);
1339 - buffer_json_object_close(wb); // time
1340 -
1341 - buffer_json_member_add_array(wb, "metrics");
1342 - for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
1343 - if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_NONE)
1344 - break;
1345 -
1346 - buffer_json_add_array_item_object(wb);
1347 - {
1348 - buffer_json_member_add_array(wb, "group_by");
1349 - buffer_json_group_by_to_array(wb, qt->request.group_by[g].group_by);
1350 - buffer_json_array_close(wb);
1351 -
1352 - buffer_json_member_add_array(wb, "group_by_label");
1353 - for (size_t l = 0; l < qt->group_by[g].used; l++)
1354 - buffer_json_add_array_item_string(wb, qt->group_by[g].label_keys[l]);
1355 - buffer_json_array_close(wb);
1356 -
1357 - buffer_json_member_add_string(
1358 - wb, "aggregation",group_by_aggregate_function_to_string(qt->request.group_by[g].aggregation));
1359 - }
1360 - buffer_json_object_close(wb);
1361 - }
1362 - buffer_json_array_close(wb); // group_by
1363 - }
1364 - buffer_json_object_close(wb); // aggregations
1365 -
1366 - buffer_json_member_add_uint64(wb, "timeout", qt->request.timeout_ms);
1367 - }
1368 - buffer_json_object_close(wb); // request
1369 - }
1370 -
1371 - version_hashes_api_v2(wb, &qt->versions);
1372 -
1373 - buffer_json_member_add_object(wb, "summary");
1374 - struct summary_total_counts
1375 - nodes_totals = { 0 },
1376 - contexts_totals = { 0 },
1377 - instances_totals = { 0 },
1378 - metrics_totals = { 0 },
1379 - label_key_totals = { 0 },
1380 - label_key_value_totals = { 0 };
1381 - {
1382 - query_target_summary_nodes_v2(wb, qt, "nodes", &nodes_totals);
1383 - r->internal.contexts = query_target_summary_contexts_v2(wb, qt, "contexts", &contexts_totals);
1384 - query_target_summary_instances_v2(wb, qt, "instances", &instances_totals);
1385 - query_target_summary_dimensions_v12(wb, qt, "dimensions", true, &metrics_totals);
1386 - query_target_summary_labels_v12(wb, qt, "labels", true, &label_key_totals, &label_key_value_totals);
1387 - query_target_summary_alerts_v2(wb, qt, "alerts");
1388 - }
1389 - if(query_target_aggregatable(qt)) {
1390 - buffer_json_member_add_object(wb, "globals");
1391 - query_target_points_statistics(wb, qt, &qt->query_points);
1392 - buffer_json_object_close(wb); // globals
1393 - }
1394 - buffer_json_object_close(wb); // summary
1395 -
1396 - buffer_json_member_add_object(wb, "totals");
1397 - query_target_total_counts(wb, "nodes", &nodes_totals);
1398 - query_target_total_counts(wb, "contexts", &contexts_totals);
1399 - query_target_total_counts(wb, "instances", &instances_totals);
1400 - query_target_total_counts(wb, "dimensions", &metrics_totals);
1401 - query_target_total_counts(wb, "label_keys", &label_key_totals);
1402 - query_target_total_counts(wb, "label_key_values", &label_key_value_totals);
1403 - buffer_json_object_close(wb); // totals
1404 -
1405 - if(options & RRDR_OPTION_SHOW_DETAILS) {
1406 - buffer_json_member_add_object(wb, "detailed");
1407 - query_target_detailed_objects_tree(wb, r, options);
1408 - buffer_json_object_close(wb); // detailed
1409 - }
1410 -
1411 - query_target_functions(wb, "functions", r);
1412 -}
1413 -
1414 -//static void annotations_range_for_value_flags(RRDR *r, BUFFER *wb, DATASOURCE_FORMAT format __maybe_unused, RRDR_OPTIONS options, RRDR_VALUE_FLAGS flags, const char *type) {
1415 -// const size_t dims = r->d, rows = r->rows;
1416 -// size_t next_d_idx = 0;
1417 -// for(size_t d = 0; d < dims ; d++) {
1418 -// if(!rrdr_dimension_should_be_exposed(r->od[d], options))
1419 -// continue;
1420 -//
1421 -// size_t d_idx = next_d_idx++;
1422 -//
1423 -// size_t t = 0;
1424 -// while(t < rows) {
1425 -//
1426 -// // find the beginning
1427 -// time_t started = 0;
1428 -// for(; t < rows ;t++) {
1429 -// RRDR_VALUE_FLAGS o = r->o[t * r->d + d];
1430 -// if(o & flags) {
1431 -// started = r->t[t];
1432 -// break;
1433 -// }
1434 -// }
1435 -//
1436 -// if(started) {
1437 -// time_t ended = 0;
1438 -// for(; t < rows ;t++) {
1439 -// RRDR_VALUE_FLAGS o = r->o[t * r->d + d];
1440 -// if(!(o & flags)) {
1441 -// ended = r->t[t];
1442 -// break;
1443 -// }
1444 -// }
1445 -//
1446 -// if(!ended)
1447 -// ended = r->t[rows - 1];
1448 -//
1449 -// buffer_json_add_array_item_object(wb);
1450 -// buffer_json_member_add_string(wb, "t", type);
1451 -// // buffer_json_member_add_string(wb, "d", string2str(r->dn[d]));
1452 -// buffer_json_member_add_uint64(wb, "d", d_idx);
1453 -// if(started == ended) {
1454 -// if(options & RRDR_OPTION_MILLISECONDS)
1455 -// buffer_json_member_add_time_t2ms(wb, "x", started);
1456 -// else
1457 -// buffer_json_member_add_time_t(wb, "x", started);
1458 -// }
1459 -// else {
1460 -// buffer_json_member_add_array(wb, "x");
1461 -// if(options & RRDR_OPTION_MILLISECONDS) {
1462 -// buffer_json_add_array_item_time_t2ms(wb, started);
1463 -// buffer_json_add_array_item_time_t2ms(wb, ended);
1464 -// }
1465 -// else {
1466 -// buffer_json_add_array_item_time_t(wb, started);
1467 -// buffer_json_add_array_item_time_t(wb, ended);
1468 -// }
1469 -// buffer_json_array_close(wb);
1470 -// }
1471 -// buffer_json_object_close(wb);
1472 -// }
1473 -// }
1474 -// }
1475 -//}
1476 -//
1477 -//void rrdr_json_wrapper_annotations(RRDR *r, BUFFER *wb, DATASOURCE_FORMAT format __maybe_unused, RRDR_OPTIONS options) {
1478 -// buffer_json_member_add_array(wb, "annotations");
1479 -//
1480 -// annotations_range_for_value_flags(r, wb, format, options, RRDR_VALUE_EMPTY, "G"); // Gap
1481 -// annotations_range_for_value_flags(r, wb, format, options, RRDR_VALUE_RESET, "O"); // Overflow
1482 -// annotations_range_for_value_flags(r, wb, format, options, RRDR_VALUE_PARTIAL, "P"); // Partial
1483 -//
1484 -// buffer_json_array_close(wb); // annotations
1485 -//}
1486 -
1487 -void rrdr_json_wrapper_end(RRDR *r, BUFFER *wb) {
1488 - buffer_json_member_add_double(wb, "min", r->view.min);
1489 - buffer_json_member_add_double(wb, "max", r->view.max);
1490 -
1491 - buffer_json_query_timings(wb, "timings", &r->internal.qt->timings);
1492 - buffer_json_finalize(wb);
1493 -}
1494 -
1495 -void rrdr_json_wrapper_end2(RRDR *r, BUFFER *wb) {
1496 - QUERY_TARGET *qt = r->internal.qt;
1497 - DATASOURCE_FORMAT format = qt->request.format;
1498 - RRDR_OPTIONS options = qt->window.options;
1499 -
1500 - buffer_json_member_add_object(wb, "db");
1501 - {
1502 - buffer_json_member_add_uint64(wb, "tiers", nd_profile.storage_tiers);
1503 - buffer_json_member_add_time_t(wb, "update_every", qt->db.minimum_latest_update_every_s);
1504 - buffer_json_member_add_time_t(wb, "first_entry", qt->db.first_time_s);
1505 - buffer_json_member_add_time_t(wb, "last_entry", qt->db.last_time_s);
1506 -
1507 - query_target_combined_units_v2(wb, qt, r->internal.contexts, true);
1508 - buffer_json_member_add_object(wb, "dimensions");
1509 - {
1510 - rrdr_dimension_ids(wb, "ids", r, options);
1511 - rrdr_dimension_units_array_v2(wb, "units", r, options, true);
1512 - rrdr_dimension_query_points_statistics(wb, "sts", r, options, false);
1513 - }
1514 - buffer_json_object_close(wb); // dimensions
1515 -
1516 - buffer_json_member_add_array(wb, "per_tier");
1517 - for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
1518 - buffer_json_add_array_item_object(wb);
1519 - buffer_json_member_add_uint64(wb, "tier", tier);
1520 - buffer_json_member_add_uint64(wb, "queries", qt->db.tiers[tier].queries);
1521 - buffer_json_member_add_uint64(wb, "points", qt->db.tiers[tier].points);
1522 - buffer_json_member_add_time_t(wb, "update_every", qt->db.tiers[tier].update_every);
1523 - buffer_json_member_add_time_t(wb, "first_entry", qt->db.tiers[tier].retention.first_time_s);
1524 - buffer_json_member_add_time_t(wb, "last_entry", qt->db.tiers[tier].retention.last_time_s);
1525 - buffer_json_object_close(wb);
1526 - }
1527 - buffer_json_array_close(wb);
1528 - }
1529 - buffer_json_object_close(wb);
1530 -
1531 - buffer_json_member_add_object(wb, "view");
1532 - {
1533 - query_target_title(wb, qt, r->internal.contexts);
1534 - buffer_json_member_add_time_t(wb, "update_every", r->view.update_every);
1535 - buffer_json_member_add_time_t(wb, "after", r->view.after);
1536 - buffer_json_member_add_time_t(wb, "before", r->view.before);
1537 -
1538 - if(options & RRDR_OPTION_DEBUG) {
1539 - buffer_json_member_add_string(wb, "format", rrdr_format_to_string(format));
1540 - rrdr_options_to_buffer_json_array(wb, "options", options);
1541 - buffer_json_member_add_string(wb, "time_group", time_grouping_tostring(qt->request.time_group_method));
1542 - }
1543 -
1544 - if(options & RRDR_OPTION_DEBUG) {
1545 - buffer_json_member_add_object(wb, "partial_data_trimming");
1546 - buffer_json_member_add_time_t(wb, "max_update_every", r->partial_data_trimming.max_update_every);
1547 - buffer_json_member_add_time_t(wb, "expected_after", r->partial_data_trimming.expected_after);
1548 - buffer_json_member_add_time_t(wb, "trimmed_after", r->partial_data_trimming.trimmed_after);
1549 - buffer_json_object_close(wb);
1550 - }
1551 -
1552 - if(options & RRDR_OPTION_RETURN_RAW)
1553 - buffer_json_member_add_uint64(wb, "points", rrdr_rows(r));
1554 -
1555 - query_target_combined_units_v2(wb, qt, r->internal.contexts, false);
1556 - query_target_combined_chart_type(wb, qt, r->internal.contexts);
1557 - buffer_json_member_add_object(wb, "dimensions");
1558 - {
1559 - rrdr_grouped_by_array_v2(wb, "grouped_by", r, options);
1560 - rrdr_dimension_ids(wb, "ids", r, options);
1561 - rrdr_dimension_names(wb, "names", r, options);
1562 - rrdr_dimension_units_array_v2(wb, "units", r, options, false);
1563 - rrdr_dimension_priority_array_v2(wb, "priorities", r, options);
1564 - rrdr_dimension_aggregated_array_v2(wb, "aggregated", r, options);
1565 - rrdr_dimension_query_points_statistics(wb, "sts", r, options, true);
1566 - rrdr_json_group_by_labels(wb, "labels", r, options);
1567 - }
1568 - buffer_json_object_close(wb); // dimensions
1569 - buffer_json_member_add_double(wb, "min", r->view.min);
1570 - buffer_json_member_add_double(wb, "max", r->view.max);
1571 - }
1572 - buffer_json_object_close(wb); // view
1573 -
1574 - buffer_json_agents_v2(wb, &r->internal.qt->timings, 0, false, true);
1575 - buffer_json_cloud_timings(wb, "timings", &r->internal.qt->timings);
1576 - buffer_json_finalize(wb);
1577 -}
src/web/api/formatters/jsonwrap-internal.h new
+52
@@ -0,0 +1,52 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_JSONWRAP_INTERNAL_H
4 +#define NETDATA_JSONWRAP_INTERNAL_H
5 +
6 +#include "jsonwrap.h"
7 +#include "libnetdata/json/json-keys.h"
8 +
9 +struct summary_total_counts {
10 + size_t selected;
11 + size_t excluded;
12 + size_t queried;
13 + size_t failed;
14 +};
15 +
16 +
17 +// visualizers
18 +void query_target_total_counts(BUFFER *wb, const char *key, struct summary_total_counts *totals);
19 +void query_target_metric_counts(BUFFER *wb, QUERY_METRICS_COUNTS *metrics);
20 +void query_target_instance_counts(BUFFER *wb, QUERY_INSTANCES_COUNTS *instances);
21 +void query_target_alerts_counts(BUFFER *wb, QUERY_ALERTS_COUNTS *alerts, const char *name, bool array);
22 +void query_target_points_statistics(BUFFER *wb, QUERY_TARGET *qt, STORAGE_POINT *sp);
23 +
24 +// aggregators
25 +void aggregate_metrics_counts(QUERY_METRICS_COUNTS *dst, const QUERY_METRICS_COUNTS *src);
26 +void aggregate_instances_counts(QUERY_INSTANCES_COUNTS *dst, const QUERY_INSTANCES_COUNTS *src);
27 +void aggregate_alerts_counts(QUERY_ALERTS_COUNTS *dst, const QUERY_ALERTS_COUNTS *src);
28 +void aggregate_into_summary_totals(struct summary_total_counts *totals, QUERY_METRICS_COUNTS *metrics);
29 +
30 +
31 +// common library calls
32 +size_t rrdr_dimension_names(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options);
33 +size_t rrdr_dimension_ids(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options);
34 +void query_target_functions(BUFFER *wb, const char *key, RRDR *r);
35 +void query_target_info_nodes(BUFFER *wb, const char *key, RRDR_OPTIONS options);
36 +
37 +// summary formatters
38 +void query_target_summary_labels_v12(BUFFER *wb, QUERY_TARGET *qt, const char *key, bool v2, struct summary_total_counts *key_totals, struct summary_total_counts *value_totals);
39 +void query_target_summary_nodes_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals);
40 +void query_target_summary_instances_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals);
41 +void query_target_summary_dimensions_v12(BUFFER *wb, QUERY_TARGET *qt, const char *key, bool v2, struct summary_total_counts *totals);
42 +size_t query_target_summary_contexts_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals);
43 +void query_target_summary_alerts_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key);
44 +
45 +void query_target_summary_instances_v1(BUFFER *wb, QUERY_TARGET *qt, const char *key);
46 +
47 +void jsonwrap_query_plan(RRDR *r, BUFFER *wb, RRDR_OPTIONS options);
48 +void jsonwrap_query_metric_plan(BUFFER *wb, QUERY_METRIC *qm, RRDR_OPTIONS options);
49 +
50 +void query_target_detailed_objects_tree(BUFFER *wb, RRDR *r, RRDR_OPTIONS options);
51 +
52 +#endif //NETDATA_JSONWRAP_INTERNAL_H
src/web/api/formatters/jsonwrap-objects-tree.c new
+180
@@ -0,0 +1,180 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +static void rrdset_rrdcalc_entries_v2(BUFFER *wb, RRDINSTANCE_ACQUIRED *ria) {
7 + RRDSET *st = rrdinstance_acquired_rrdset(ria);
8 + if(st) {
9 + rw_spinlock_read_lock(&st->alerts.spinlock);
10 + if(st->alerts.base) {
11 + buffer_json_member_add_object(wb, "alerts");
12 + for(RRDCALC *rc = st->alerts.base; rc ;rc = rc->next) {
13 + if(rc->status < RRDCALC_STATUS_CLEAR)
14 + continue;
15 +
16 + buffer_json_member_add_object(wb, string2str(rc->config.name));
17 + buffer_json_member_add_string(wb, JSKEY(status), rrdcalc_status2string(rc->status));
18 + buffer_json_member_add_double(wb, JSKEY(value), rc->value);
19 + buffer_json_member_add_string(wb, JSKEY(units), string2str(rc->config.units));
20 + buffer_json_object_close(wb);
21 + }
22 + buffer_json_object_close(wb);
23 + }
24 + rw_spinlock_read_unlock(&st->alerts.spinlock);
25 + }
26 +}
27 +
28 +void query_target_detailed_objects_tree(BUFFER *wb, RRDR *r, RRDR_OPTIONS options) {
29 + QUERY_TARGET *qt = r->internal.qt;
30 + buffer_json_member_add_object(wb, "nodes");
31 +
32 + time_t now_s = now_realtime_sec();
33 + RRDHOST *last_host = NULL;
34 + RRDCONTEXT_ACQUIRED *last_rca = NULL;
35 + RRDINSTANCE_ACQUIRED *last_ria = NULL;
36 +
37 + size_t h = 0, c = 0, i = 0, m = 0, q = 0;
38 + for(; h < qt->nodes.used ; h++) {
39 + QUERY_NODE *qn = query_node(qt, h);
40 + RRDHOST *host = qn->rrdhost;
41 +
42 + for( ;c < qt->contexts.used ;c++) {
43 + QUERY_CONTEXT *qc = query_context(qt, c);
44 + RRDCONTEXT_ACQUIRED *rca = qc->rca;
45 + if(!rrdcontext_acquired_belongs_to_host(rca, host)) break;
46 +
47 + for( ;i < qt->instances.used ;i++) {
48 + QUERY_INSTANCE *qi = query_instance(qt, i);
49 + RRDINSTANCE_ACQUIRED *ria = qi->ria;
50 + if(!rrdinstance_acquired_belongs_to_context(ria, rca)) break;
51 +
52 + for( ; m < qt->dimensions.used ; m++) {
53 + QUERY_DIMENSION *qd = query_dimension(qt, m);
54 + RRDMETRIC_ACQUIRED *rma = qd->rma;
55 + if(!rrdmetric_acquired_belongs_to_instance(rma, ria)) break;
56 +
57 + QUERY_METRIC *qm = NULL;
58 + bool queried = false;
59 + for( ; q < qt->query.used ;q++) {
60 + QUERY_METRIC *tqm = query_metric(qt, q);
61 + QUERY_DIMENSION *tqd = query_dimension(qt, tqm->link.query_dimension_id);
62 + if(tqd->rma != rma) break;
63 +
64 + queried = tqm->status & RRDR_DIMENSION_QUERIED;
65 + qm = tqm;
66 + }
67 +
68 + if(!queried & !(options & RRDR_OPTION_ALL_DIMENSIONS))
69 + continue;
70 +
71 + if(host != last_host) {
72 + if(last_host) {
73 + if(last_rca) {
74 + if(last_ria) {
75 + buffer_json_object_close(wb); // dimensions
76 + buffer_json_object_close(wb); // instance
77 + last_ria = NULL;
78 + }
79 + buffer_json_object_close(wb); // instances
80 + buffer_json_object_close(wb); // context
81 + last_rca = NULL;
82 + }
83 + buffer_json_object_close(wb); // contexts
84 + buffer_json_object_close(wb); // host
85 + }
86 +
87 + buffer_json_member_add_object(wb, host->machine_guid);
88 + if(qn->node_id[0])
89 + buffer_json_member_add_string(wb, JSKEY(node_id), qn->node_id);
90 + buffer_json_member_add_uint64(wb, JSKEY(node_index), qn->slot);
91 + buffer_json_member_add_string(wb, JSKEY(hostname), rrdhost_hostname(host));
92 + buffer_json_member_add_object(wb, "contexts");
93 +
94 + last_host = host;
95 + }
96 +
97 + if(rca != last_rca) {
98 + if(last_rca) {
99 + if(last_ria) {
100 + buffer_json_object_close(wb); // dimensions
101 + buffer_json_object_close(wb); // instance
102 + last_ria = NULL;
103 + }
104 + buffer_json_object_close(wb); // instances
105 + buffer_json_object_close(wb); // context
106 + last_rca = NULL;
107 + }
108 +
109 + buffer_json_member_add_object(wb, rrdcontext_acquired_id(rca));
110 + buffer_json_member_add_object(wb, "instances");
111 +
112 + last_rca = rca;
113 + }
114 +
115 + if(ria != last_ria) {
116 + if(last_ria) {
117 + buffer_json_object_close(wb); // dimensions
118 + buffer_json_object_close(wb); // instance
119 + last_ria = NULL;
120 + }
121 +
122 + buffer_json_member_add_object(wb, rrdinstance_acquired_id(ria));
123 + buffer_json_member_add_string(wb, JSKEY(name), rrdinstance_acquired_name(ria));
124 + buffer_json_member_add_time_t(wb, JSKEY(update_every), rrdinstance_acquired_update_every(ria));
125 + RRDLABELS *labels = rrdinstance_acquired_labels(ria);
126 + if(labels) {
127 + buffer_json_member_add_object(wb, "labels");
128 + rrdlabels_to_buffer_json_members(labels, wb);
129 + buffer_json_object_close(wb);
130 + }
131 + rrdset_rrdcalc_entries_v2(wb, ria);
132 + buffer_json_member_add_object(wb, "dimensions");
133 +
134 + last_ria = ria;
135 + }
136 +
137 + buffer_json_member_add_object(wb, rrdmetric_acquired_id(rma));
138 + {
139 + buffer_json_member_add_string(wb, JSKEY(name), rrdmetric_acquired_name(rma));
140 + buffer_json_member_add_uint64(wb, JSKEY(queried), queried ? 1 : 0);
141 + time_t first_entry_s = rrdmetric_acquired_first_entry(rma);
142 + time_t last_entry_s = rrdmetric_acquired_last_entry(rma);
143 + buffer_json_member_add_time_t_formatted(wb, JSKEY(first_entry), first_entry_s, options & RRDR_OPTION_RFC3339);
144 + buffer_json_member_add_time_t_formatted(wb, JSKEY(last_entry), last_entry_s ? last_entry_s : now_s, options & RRDR_OPTION_RFC3339);
145 +
146 + if(qm) {
147 + if(qm->status & RRDR_DIMENSION_GROUPED) {
148 + // buffer_json_member_add_string(wb, "grouped_as_id", string2str(qm->grouped_as.id));
149 + buffer_json_member_add_string(wb, "as", string2str(qm->grouped_as.name));
150 + }
151 +
152 + query_target_points_statistics(wb, qt, &qm->query_points);
153 +
154 + if(options & RRDR_OPTION_DEBUG)
155 + jsonwrap_query_metric_plan(wb, qm, options);
156 + }
157 + }
158 + buffer_json_object_close(wb); // metric
159 + }
160 + }
161 + }
162 + }
163 +
164 + if(last_host) {
165 + if(last_rca) {
166 + if(last_ria) {
167 + buffer_json_object_close(wb); // dimensions
168 + buffer_json_object_close(wb); // instance
169 + last_ria = NULL;
170 + }
171 + buffer_json_object_close(wb); // instances
172 + buffer_json_object_close(wb); // context
173 + last_rca = NULL;
174 + }
175 + buffer_json_object_close(wb); // contexts
176 + buffer_json_object_close(wb); // host
177 + last_host = NULL;
178 + }
179 + buffer_json_object_close(wb); // hosts
180 +}
src/web/api/formatters/jsonwrap-query-plan.c new
+43
@@ -0,0 +1,43 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +void jsonwrap_query_metric_plan(BUFFER *wb, QUERY_METRIC *qm, RRDR_OPTIONS options) {
7 + buffer_json_member_add_array(wb, "plans");
8 + for (size_t p = 0; p < qm->plan.used; p++) {
9 + QUERY_PLAN_ENTRY *qp = &qm->plan.array[p];
10 +
11 + buffer_json_add_array_item_object(wb);
12 + buffer_json_member_add_uint64(wb, JSKEY(tier), qp->tier);
13 + buffer_json_member_add_time_t_formatted(wb, JSKEY(after), qp->after, options & RRDR_OPTION_RFC3339);
14 + buffer_json_member_add_time_t_formatted(wb, JSKEY(before), qp->before, options & RRDR_OPTION_RFC3339);
15 + buffer_json_object_close(wb);
16 + }
17 + buffer_json_array_close(wb);
18 +
19 + buffer_json_member_add_array(wb, "tiers");
20 + for (size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
21 + buffer_json_add_array_item_object(wb);
22 + buffer_json_member_add_uint64(wb, JSKEY(tier), tier);
23 + buffer_json_member_add_time_t_formatted(wb, JSKEY(first_entry), qm->tiers[tier].db_first_time_s, options & RRDR_OPTION_RFC3339);
24 + buffer_json_member_add_time_t_formatted(wb, JSKEY(last_entry), qm->tiers[tier].db_last_time_s, options & RRDR_OPTION_RFC3339);
25 + buffer_json_member_add_int64(wb, JSKEY(weight), qm->tiers[tier].weight);
26 + buffer_json_object_close(wb);
27 + }
28 + buffer_json_array_close(wb);
29 +}
30 +
31 +void jsonwrap_query_plan(RRDR *r, BUFFER *wb, RRDR_OPTIONS options) {
32 + QUERY_TARGET *qt = r->internal.qt;
33 +
34 + buffer_json_member_add_object(wb, "query_plan");
35 + for(size_t m = 0; m < qt->query.used; m++) {
36 + QUERY_METRIC *qm = query_metric(qt, m);
37 + buffer_json_member_add_object(wb, query_metric_id(qt, qm));
38 + jsonwrap_query_metric_plan(wb, qm, options);
39 + buffer_json_object_close(wb);
40 + }
41 + buffer_json_object_close(wb);
42 +}
43 +
src/web/api/formatters/jsonwrap-summary-alerts.c new
+50
@@ -0,0 +1,50 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +void query_target_summary_alerts_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key) {
7 + buffer_json_member_add_array(wb, key);
8 + QUERY_ALERTS_COUNTS *z;
9 +
10 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
11 + for (long c = 0; c < (long) qt->instances.used; c++) {
12 + QUERY_INSTANCE *qi = query_instance(qt, c);
13 + RRDSET *st = rrdinstance_acquired_rrdset(qi->ria);
14 + if (st) {
15 + rw_spinlock_read_lock(&st->alerts.spinlock);
16 + if (st->alerts.base) {
17 + for (RRDCALC *rc = st->alerts.base; rc; rc = rc->next) {
18 + z = dictionary_set(dict, string2str(rc->config.name), NULL, sizeof(*z));
19 +
20 + switch(rc->status) {
21 + case RRDCALC_STATUS_CLEAR:
22 + z->clear++;
23 + break;
24 +
25 + case RRDCALC_STATUS_WARNING:
26 + z->warning++;
27 + break;
28 +
29 + case RRDCALC_STATUS_CRITICAL:
30 + z->critical++;
31 + break;
32 +
33 + default:
34 + case RRDCALC_STATUS_UNINITIALIZED:
35 + case RRDCALC_STATUS_UNDEFINED:
36 + case RRDCALC_STATUS_REMOVED:
37 + z->other++;
38 + break;
39 + }
40 + }
41 + }
42 + rw_spinlock_read_unlock(&st->alerts.spinlock);
43 + }
44 + }
45 + dfe_start_read(dict, z)
46 + query_target_alerts_counts(wb, z, z_dfe.name, true);
47 + dfe_done(z);
48 + dictionary_destroy(dict);
49 + buffer_json_array_close(wb); // alerts
50 +}
src/web/api/formatters/jsonwrap-summary-contexts.c new
+61
@@ -0,0 +1,61 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +size_t query_target_summary_contexts_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals) {
7 + buffer_json_member_add_array(wb, key);
8 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
9 +
10 + struct {
11 + STORAGE_POINT query_points;
12 + QUERY_INSTANCES_COUNTS instances;
13 + QUERY_METRICS_COUNTS metrics;
14 + QUERY_ALERTS_COUNTS alerts;
15 + } *z;
16 +
17 + for (long c = 0; c < (long) qt->contexts.used; c++) {
18 + QUERY_CONTEXT *qc = query_context(qt, c);
19 +
20 + z = dictionary_set(dict, rrdcontext_acquired_id(qc->rca), NULL, sizeof(*z));
21 +
22 + z->instances.selected += qc->instances.selected;
23 + z->instances.excluded += qc->instances.selected;
24 + z->instances.queried += qc->instances.queried;
25 + z->instances.failed += qc->instances.failed;
26 +
27 + z->metrics.selected += qc->metrics.selected;
28 + z->metrics.excluded += qc->metrics.excluded;
29 + z->metrics.queried += qc->metrics.queried;
30 + z->metrics.failed += qc->metrics.failed;
31 +
32 + z->alerts.clear += qc->alerts.clear;
33 + z->alerts.warning += qc->alerts.warning;
34 + z->alerts.critical += qc->alerts.critical;
35 +
36 + storage_point_merge_to(z->query_points, qc->query_points);
37 + }
38 +
39 + size_t unique_contexts = dictionary_entries(dict);
40 + dfe_start_read(dict, z) {
41 + buffer_json_add_array_item_object(wb);
42 + buffer_json_member_add_string(wb, "id", z_dfe.name);
43 +
44 + // Only include detailed statistics if MINIMAL_STATS option is not set
45 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
46 + query_target_instance_counts(wb, &z->instances);
47 + query_target_metric_counts(wb, &z->metrics);
48 + query_target_alerts_counts(wb, &z->alerts, NULL, false);
49 + }
50 +
51 + query_target_points_statistics(wb, qt, &z->query_points);
52 + buffer_json_object_close(wb);
53 +
54 + aggregate_into_summary_totals(totals, &z->metrics);
55 + }
56 + dfe_done(z);
57 + buffer_json_array_close(wb);
58 + dictionary_destroy(dict);
59 +
60 + return unique_contexts;
61 +}
src/web/api/formatters/jsonwrap-summary-dimensions.c new
+236
@@ -0,0 +1,236 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +struct dimensions_sorted_walkthrough_data {
7 + BUFFER *wb;
8 + struct summary_total_counts *totals;
9 + QUERY_TARGET *qt;
10 + size_t cardinality_limit;
11 + size_t count;
12 + QUERY_METRICS_COUNTS aggregated_metrics;
13 + STORAGE_POINT aggregated_points;
14 + NETDATA_DOUBLE remaining_contribution;
15 + size_t remaining_count;
16 +};
17 +
18 +struct dimensions_sorted_entry {
19 + const char *id;
20 + const char *name;
21 + STORAGE_POINT query_points;
22 + QUERY_METRICS_COUNTS metrics;
23 + uint32_t priority;
24 +};
25 +
26 +static int dimensions_sorted_walktrhough_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
27 + struct dimensions_sorted_walkthrough_data *sdwd = data;
28 + BUFFER *wb = sdwd->wb;
29 + struct summary_total_counts *totals = sdwd->totals;
30 + QUERY_TARGET *qt = sdwd->qt;
31 + struct dimensions_sorted_entry *z = value;
32 +
33 + sdwd->count++;
34 +
35 + // Check if we need to apply cardinality limiting
36 + if (sdwd->cardinality_limit > 0 && sdwd->count > sdwd->cardinality_limit - 1) {
37 + // This dimension exceeds our limit - aggregate it
38 +
39 + // Increment our remaining count
40 + sdwd->remaining_count++;
41 +
42 + // Calculate contribution for this dimension
43 + if (qt->query_points.sum > 0) {
44 + NETDATA_DOUBLE contribution = z->query_points.sum * 100.0 / qt->query_points.sum;
45 + sdwd->remaining_contribution += contribution;
46 + }
47 +
48 + // Aggregate metrics counts
49 + aggregate_metrics_counts(&sdwd->aggregated_metrics, &z->metrics);
50 +
51 + // Aggregate points
52 + storage_point_merge_to(sdwd->aggregated_points, z->query_points);
53 +
54 + // Still add this to summary totals
55 + aggregate_into_summary_totals(totals, &z->metrics);
56 +
57 + return 1; // Continue processing next dimension
58 + }
59 +
60 + // Output this dimension normally
61 + buffer_json_add_array_item_object(wb);
62 + buffer_json_member_add_string(wb, "id", z->id);
63 + if (z->id != z->name && z->name)
64 + buffer_json_member_add_string(wb, JSKEY(name), z->name);
65 +
66 + // Only include detailed statistics if MINIMAL_STATS option is not set
67 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
68 + query_target_metric_counts(wb, &z->metrics);
69 + }
70 +
71 + query_target_points_statistics(wb, qt, &z->query_points);
72 + buffer_json_member_add_uint64(wb, JSKEY(priority), z->priority);
73 + buffer_json_object_close(wb);
74 +
75 + aggregate_into_summary_totals(totals, &z->metrics);
76 +
77 + return 1;
78 +}
79 +
80 +// Standard sort by priority
81 +static int dimensions_sorted_priority_compar(const DICTIONARY_ITEM **item1, const DICTIONARY_ITEM **item2) {
82 + struct dimensions_sorted_entry *z1 = dictionary_acquired_item_value(*item1);
83 + struct dimensions_sorted_entry *z2 = dictionary_acquired_item_value(*item2);
84 +
85 + if(z1->priority == z2->priority)
86 + return strcmp(dictionary_acquired_item_name(*item1), dictionary_acquired_item_name(*item2));
87 + else if(z1->priority < z2->priority)
88 + return -1;
89 + else
90 + return 1;
91 +}
92 +
93 +// Sort by sum value (highest first), then priority
94 +static int dimensions_sorted_sum_compar(const DICTIONARY_ITEM **item1, const DICTIONARY_ITEM **item2) {
95 + struct dimensions_sorted_entry *z1 = dictionary_acquired_item_value(*item1);
96 + struct dimensions_sorted_entry *z2 = dictionary_acquired_item_value(*item2);
97 +
98 + // Sort by sum (highest first)
99 + if (z1->query_points.sum > z2->query_points.sum) return -1;
100 + if (z1->query_points.sum < z2->query_points.sum) return 1;
101 +
102 + // If equal sum, use priority as a secondary sort
103 + if (z1->priority != z2->priority) {
104 + if (z1->priority < z2->priority) return -1;
105 + else return 1;
106 + }
107 +
108 + // If still equal, sort alphabetically
109 + return strcmp(dictionary_acquired_item_name(*item1), dictionary_acquired_item_name(*item2));
110 +}
111 +
112 +void query_target_summary_dimensions_v12(BUFFER *wb, QUERY_TARGET *qt, const char *key, bool v2, struct summary_total_counts *totals) {
113 + char buf[RRD_ID_LENGTH_MAX * 2 + 2];
114 +
115 + buffer_json_member_add_array(wb, key);
116 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
117 + struct dimensions_sorted_entry *z;
118 + size_t q = 0;
119 + for (long c = 0; c < (long) qt->dimensions.used; c++) {
120 + QUERY_DIMENSION * qd = query_dimension(qt, c);
121 + RRDMETRIC_ACQUIRED *rma = qd->rma;
122 +
123 + QUERY_METRIC *qm = NULL;
124 + for( ; q < qt->query.used ;q++) {
125 + QUERY_METRIC *tqm = query_metric(qt, q);
126 + QUERY_DIMENSION *tqd = query_dimension(qt, tqm->link.query_dimension_id);
127 + if(tqd->rma != rma) break;
128 + qm = tqm;
129 + }
130 +
131 + const char *k, *id, *name;
132 +
133 + if(v2) {
134 + k = rrdmetric_acquired_name(rma);
135 + id = k;
136 + name = k;
137 + }
138 + else {
139 + snprintfz(buf, RRD_ID_LENGTH_MAX * 2 + 1, "%s:%s",
140 + rrdmetric_acquired_id(rma),
141 + rrdmetric_acquired_name(rma));
142 + k = buf;
143 + id = rrdmetric_acquired_id(rma);
144 + name = rrdmetric_acquired_name(rma);
145 + }
146 +
147 + z = dictionary_set(dict, k, NULL, sizeof(*z));
148 + if(!z->id) {
149 + z->id = id;
150 + z->name = name;
151 + z->priority = qd->priority;
152 + }
153 + else {
154 + if(qd->priority < z->priority)
155 + z->priority = qd->priority;
156 + }
157 +
158 + if(qm) {
159 + z->metrics.selected += (qm->status & RRDR_DIMENSION_SELECTED) ? 1 : 0;
160 + z->metrics.failed += (qm->status & RRDR_DIMENSION_FAILED) ? 1 : 0;
161 +
162 + if(qm->status & RRDR_DIMENSION_QUERIED) {
163 + z->metrics.queried++;
164 + storage_point_merge_to(z->query_points, qm->query_points);
165 + }
166 + }
167 + else
168 + z->metrics.excluded++;
169 + }
170 +
171 + if(v2) {
172 + size_t cardinality_limit = qt->request.cardinality_limit;
173 + size_t dict_entries = dictionary_entries(dict);
174 +
175 + // Use the enhanced walkthrough with cardinality limiting
176 + struct dimensions_sorted_walkthrough_data t = {
177 + .wb = wb,
178 + .totals = totals,
179 + .qt = qt,
180 + .cardinality_limit = cardinality_limit,
181 + .count = 0,
182 + .remaining_count = 0,
183 + .remaining_contribution = 0.0,
184 + .aggregated_metrics = {0},
185 + .aggregated_points = STORAGE_POINT_UNSET
186 + };
187 +
188 + // Execute the sorted walkthrough which will either output dimensions directly
189 + // or aggregate them if they exceed the cardinality limit
190 + // Choose the appropriate comparison function based on cardinality limiting
191 + dict_item_comparator_t comparator = (cardinality_limit > 0 && dict_entries > cardinality_limit) ?
192 + dimensions_sorted_sum_compar :
193 + dimensions_sorted_priority_compar;
194 +
195 + if (comparator)
196 + dictionary_sorted_walkthrough_rw(dict, DICTIONARY_LOCK_READ,
197 + dimensions_sorted_walktrhough_cb, &t, comparator);
198 + else
199 + dictionary_walkthrough_rw(dict, DICTIONARY_LOCK_READ,
200 + dimensions_sorted_walktrhough_cb, &t);
201 +
202 +
203 + // Add the aggregated "remaining" dimension if there are any
204 + if (t.remaining_count > 0) {
205 + buffer_json_add_array_item_object(wb);
206 +
207 + // Add basic info for the aggregated dimension
208 + char remaining_label[50];
209 + snprintfz(remaining_label, sizeof(remaining_label), "remaining %zu dimensions", t.remaining_count);
210 +
211 + buffer_json_member_add_string(wb, "id", "__remaining_dimensions__");
212 + buffer_json_member_add_string(wb, JSKEY(name), remaining_label);
213 + buffer_json_member_add_double(wb, JSKEY(contribution), t.remaining_contribution);
214 +
215 + // Only include detailed statistics if MINIMAL_STATS option is not set
216 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
217 + query_target_metric_counts(wb, &t.aggregated_metrics);
218 + }
219 +
220 + query_target_points_statistics(wb, qt, &t.aggregated_points);
221 + buffer_json_object_close(wb);
222 + }
223 + }
224 + else {
225 + // v1
226 + dfe_start_read(dict, z) {
227 + buffer_json_add_array_item_array(wb);
228 + buffer_json_add_array_item_string(wb, z->id);
229 + buffer_json_add_array_item_string(wb, z->name);
230 + buffer_json_array_close(wb);
231 + }
232 + dfe_done(z);
233 + }
234 + dictionary_destroy(dict);
235 + buffer_json_array_close(wb);
236 +}
src/web/api/formatters/jsonwrap-summary-instances.c new
+191
@@ -0,0 +1,191 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +void query_target_summary_instances_v1(BUFFER *wb, QUERY_TARGET *qt, const char *key) {
7 + char name[RRD_ID_LENGTH_MAX * 2 + 2];
8 +
9 + buffer_json_member_add_array(wb, key);
10 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
11 + for (long c = 0; c < (long) qt->instances.used; c++) {
12 + QUERY_INSTANCE *qi = query_instance(qt, c);
13 +
14 + snprintfz(name, RRD_ID_LENGTH_MAX * 2 + 1, "%s:%s",
15 + rrdinstance_acquired_id(qi->ria),
16 + rrdinstance_acquired_name(qi->ria));
17 +
18 + bool *set = dictionary_set(dict, name, NULL, sizeof(*set));
19 + if (!*set) {
20 + *set = true;
21 + buffer_json_add_array_item_array(wb);
22 + buffer_json_add_array_item_string(wb, rrdinstance_acquired_id(qi->ria));
23 + buffer_json_add_array_item_string(wb, rrdinstance_acquired_name(qi->ria));
24 + buffer_json_array_close(wb);
25 + }
26 + }
27 + dictionary_destroy(dict);
28 + buffer_json_array_close(wb);
29 +}
30 +
31 +// Helper structure for sorting and limiting instances based on their contribution
32 +typedef struct {
33 + size_t index; // Original index in the array
34 + NETDATA_DOUBLE contribution; // Sorting metric (usually volume contribution)
35 + const char *id; // Instance ID
36 + const char *name; // Instance name
37 +} INSTANCE_CARDINALITY_ITEM;
38 +
39 +// Comparison function for sorting instances by contribution
40 +static int instance_cardinality_item_compare(const void *a, const void *b) {
41 + const INSTANCE_CARDINALITY_ITEM *item_a = (const INSTANCE_CARDINALITY_ITEM *)a;
42 + const INSTANCE_CARDINALITY_ITEM *item_b = (const INSTANCE_CARDINALITY_ITEM *)b;
43 +
44 + // Sort by contribution (highest first)
45 + if (item_a->contribution > item_b->contribution) return -1;
46 + if (item_a->contribution < item_b->contribution) return 1;
47 +
48 + // If equal contribution, sort alphabetically by id
49 + if (item_a->id && item_b->id)
50 + return strcmp(item_a->id, item_b->id);
51 +
52 + return 0;
53 +}
54 +
55 +void query_target_summary_instances_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals) {
56 + buffer_json_member_add_array(wb, key);
57 + long count = (long) qt->instances.used;
58 + size_t cardinality_limit = qt->request.cardinality_limit;
59 +
60 + // Check if we need to apply cardinality limiting
61 + if (cardinality_limit > 0 && count > (long)cardinality_limit) {
62 + // We'll need to sort and limit the instances
63 + INSTANCE_CARDINALITY_ITEM *items = mallocz(sizeof(INSTANCE_CARDINALITY_ITEM) * count);
64 +
65 + // Collect contribution data for each instance
66 + for (long c = 0; c < count; c++) {
67 + QUERY_INSTANCE *qi = query_instance(qt, c);
68 + items[c].index = c;
69 + items[c].id = rrdinstance_acquired_id(qi->ria);
70 + items[c].name = rrdinstance_acquired_name(qi->ria);
71 +
72 + // Use query points as the metric for contribution
73 + if (qt->query_points.sum > 0)
74 + items[c].contribution = qi->query_points.sum * 100.0 / qt->query_points.sum;
75 + else
76 + items[c].contribution = 0.0;
77 + }
78 +
79 + // Sort by contribution
80 + qsort(items, count, sizeof(INSTANCE_CARDINALITY_ITEM), instance_cardinality_item_compare);
81 +
82 + // First add the top (limit-1) instances
83 + size_t instances_to_show = cardinality_limit - 1;
84 + QUERY_METRICS_COUNTS aggregated_metrics = {0};
85 + QUERY_ALERTS_COUNTS aggregated_alerts = {0};
86 + STORAGE_POINT aggregated_points = STORAGE_POINT_UNSET;
87 + NETDATA_DOUBLE remaining_contribution = 0.0;
88 + size_t remaining_count = 0;
89 +
90 + // Output the top instances
91 + for (size_t i = 0; (long)i < count; i++) {
92 + if (i < instances_to_show) {
93 + // Output this instance normally
94 + QUERY_INSTANCE *qi = query_instance(qt, items[i].index);
95 +
96 + buffer_json_add_array_item_object(wb);
97 + buffer_json_member_add_string(wb, "id", items[i].id);
98 +
99 + if(!rrdinstance_acquired_id_and_name_are_same(qi->ria))
100 + buffer_json_member_add_string(wb, JSKEY(name), items[i].name);
101 +
102 + buffer_json_member_add_uint64(wb, JSKEY(node_index), qi->query_host_id);
103 +
104 + if (items[i].contribution > 0.0)
105 + buffer_json_member_add_double(wb, JSKEY(contribution), items[i].contribution);
106 +
107 + // Only include detailed statistics if MINIMAL_STATS option is not set
108 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
109 + query_target_metric_counts(wb, &qi->metrics);
110 + query_target_alerts_counts(wb, &qi->alerts, NULL, false);
111 + }
112 +
113 + query_target_points_statistics(wb, qt, &qi->query_points);
114 + buffer_json_object_close(wb);
115 +
116 + aggregate_into_summary_totals(totals, &qi->metrics);
117 + } else {
118 + // Aggregate the remaining instances
119 + QUERY_INSTANCE *qi = query_instance(qt, items[i].index);
120 + remaining_contribution += items[i].contribution;
121 + remaining_count++;
122 +
123 + // Aggregate metrics and alerts counts
124 + aggregate_metrics_counts(&aggregated_metrics, &qi->metrics);
125 + aggregate_alerts_counts(&aggregated_alerts, &qi->alerts);
126 +
127 + // Aggregate points
128 + storage_point_merge_to(aggregated_points, qi->query_points);
129 +
130 + // Still add to summary totals
131 + aggregate_into_summary_totals(totals, &qi->metrics);
132 + }
133 + }
134 +
135 + // Add the aggregated "remaining" entry if there are any
136 + if (remaining_count > 0) {
137 + char remaining_instance[50];
138 + snprintfz(remaining_instance, sizeof(remaining_instance), "remaining %zu instances", remaining_count);
139 +
140 + buffer_json_add_array_item_object(wb);
141 + buffer_json_member_add_string(wb, "id", "__remaining_instances__");
142 + buffer_json_member_add_string(wb, JSKEY(name), remaining_instance);
143 +
144 + if (remaining_contribution > 0.0)
145 + buffer_json_member_add_double(wb, JSKEY(contribution), remaining_contribution);
146 +
147 + // Only include detailed statistics if MINIMAL_STATS option is not set
148 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
149 + query_target_metric_counts(wb, &aggregated_metrics);
150 + query_target_alerts_counts(wb, &aggregated_alerts, NULL, false);
151 + }
152 +
153 + query_target_points_statistics(wb, qt, &aggregated_points);
154 + buffer_json_object_close(wb);
155 + }
156 +
157 + freez(items);
158 + } else {
159 + // No limiting needed, output all instances
160 + for (long c = 0; c < count; c++) {
161 + QUERY_INSTANCE *qi = query_instance(qt, c);
162 +
163 + buffer_json_add_array_item_object(wb);
164 + buffer_json_member_add_string(wb, "id", rrdinstance_acquired_id(qi->ria));
165 +
166 + if(!rrdinstance_acquired_id_and_name_are_same(qi->ria))
167 + buffer_json_member_add_string(wb, JSKEY(name), rrdinstance_acquired_name(qi->ria));
168 +
169 + buffer_json_member_add_uint64(wb, JSKEY(node_index), qi->query_host_id);
170 +
171 + // Calculate contribution for this instance
172 + if (qt->query_points.sum > 0) {
173 + NETDATA_DOUBLE contribution = qi->query_points.sum * 100.0 / qt->query_points.sum;
174 + if (contribution > 0.0)
175 + buffer_json_member_add_double(wb, JSKEY(contribution), contribution);
176 + }
177 +
178 + // Only include detailed statistics if MINIMAL_STATS option is not set
179 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
180 + query_target_metric_counts(wb, &qi->metrics);
181 + query_target_alerts_counts(wb, &qi->alerts, NULL, false);
182 + }
183 +
184 + query_target_points_statistics(wb, qt, &qi->query_points);
185 + buffer_json_object_close(wb);
186 +
187 + aggregate_into_summary_totals(totals, &qi->metrics);
188 + }
189 + }
190 + buffer_json_array_close(wb);
191 +}
src/web/api/formatters/jsonwrap-summary-labels.c new
+238
@@ -0,0 +1,238 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +struct rrdlabels_formatting_v2 {
7 + DICTIONARY *keys;
8 + QUERY_INSTANCE *qi;
9 + bool v2;
10 +};
11 +
12 +struct rrdlabels_keys_dict_entry {
13 + const char *name;
14 + DICTIONARY *values;
15 + STORAGE_POINT query_points;
16 + QUERY_METRICS_COUNTS metrics;
17 +};
18 +
19 +struct rrdlabels_key_value_dict_entry {
20 + const char *key;
21 + const char *value;
22 + STORAGE_POINT query_points;
23 + QUERY_METRICS_COUNTS metrics;
24 +};
25 +
26 +static int rrdlabels_formatting_v2(const char *name, const char *value, RRDLABEL_SRC ls __maybe_unused, void *data) {
27 + struct rrdlabels_formatting_v2 *t = data;
28 +
29 + struct rrdlabels_keys_dict_entry *d = dictionary_set(t->keys, name, NULL, sizeof(*d));
30 + if(!d->values) {
31 + d->name = name;
32 + d->values = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
33 + }
34 +
35 + char n[RRD_ID_LENGTH_MAX * 2 + 2];
36 + snprintfz(n, RRD_ID_LENGTH_MAX * 2, "%s:%s", name, value);
37 +
38 + struct rrdlabels_key_value_dict_entry *z = dictionary_set(d->values, n, NULL, sizeof(*z));
39 + if(!z->key) {
40 + z->key = name;
41 + z->value = value;
42 + }
43 +
44 + if(t->v2) {
45 + QUERY_INSTANCE *qi = t->qi;
46 +
47 + z->metrics.selected += qi->metrics.selected;
48 + z->metrics.excluded += qi->metrics.excluded;
49 + z->metrics.queried += qi->metrics.queried;
50 + z->metrics.failed += qi->metrics.failed;
51 +
52 + d->metrics.selected += qi->metrics.selected;
53 + d->metrics.excluded += qi->metrics.excluded;
54 + d->metrics.queried += qi->metrics.queried;
55 + d->metrics.failed += qi->metrics.failed;
56 +
57 + storage_point_merge_to(z->query_points, qi->query_points);
58 + storage_point_merge_to(d->query_points, qi->query_points);
59 + }
60 +
61 + return 1;
62 +}
63 +
64 +// Sort label values by sum value (highest first)
65 +int label_values_sorted_sum_compar(const DICTIONARY_ITEM **item1, const DICTIONARY_ITEM **item2) {
66 + struct rrdlabels_key_value_dict_entry *z1 = dictionary_acquired_item_value(*item1);
67 + struct rrdlabels_key_value_dict_entry *z2 = dictionary_acquired_item_value(*item2);
68 +
69 + // Sort by sum (highest first)
70 + if (z1->query_points.sum > z2->query_points.sum) return -1;
71 + if (z1->query_points.sum < z2->query_points.sum) return 1;
72 +
73 + // If equal sum, sort alphabetically
74 + return strcmp(dictionary_acquired_item_name(*item1), dictionary_acquired_item_name(*item2));
75 +}
76 +
77 +/**
78 + * Output a label value to the JSON buffer
79 + */
80 +static inline void output_label_value(BUFFER *wb, QUERY_TARGET *qt, const char *id, const char *name,
81 + QUERY_METRICS_COUNTS *metrics, STORAGE_POINT *points) {
82 + buffer_json_add_array_item_object(wb);
83 + buffer_json_member_add_string(wb, "id", id);
84 +
85 + if (name)
86 + buffer_json_member_add_string(wb, JSKEY(name), name);
87 +
88 + // Only include detailed statistics if MINIMAL_STATS option is not set
89 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
90 + query_target_metric_counts(wb, metrics);
91 + }
92 +
93 + query_target_points_statistics(wb, qt, points);
94 + buffer_json_object_close(wb);
95 +}
96 +
97 +// Callback for label values walkthrough
98 +struct label_values_walkthrough_data {
99 + BUFFER *wb;
100 + struct summary_total_counts *totals;
101 + QUERY_TARGET *qt;
102 + size_t cardinality_limit;
103 + size_t count;
104 +
105 + // Aggregated "remaining" value data
106 + struct {
107 + QUERY_METRICS_COUNTS metrics;
108 + STORAGE_POINT points;
109 + size_t count;
110 + } remaining;
111 +};
112 +
113 +static int label_values_walkthrough_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
114 + struct label_values_walkthrough_data *lwt = data;
115 + struct summary_total_counts *totals = lwt->totals;
116 + QUERY_TARGET *qt = lwt->qt;
117 + struct rrdlabels_key_value_dict_entry *z = value;
118 +
119 + lwt->count++;
120 +
121 + // Check if we need to apply cardinality limiting
122 + if (lwt->cardinality_limit > 0 && lwt->count > lwt->cardinality_limit - 1) {
123 + // This value exceeds our limit - aggregate it
124 +
125 + // Increment our remaining count
126 + lwt->remaining.count++;
127 +
128 + // Aggregate metrics counts
129 + aggregate_metrics_counts(&lwt->remaining.metrics, &z->metrics);
130 +
131 + // Aggregate points
132 + storage_point_merge_to(lwt->remaining.points, z->query_points);
133 +
134 + // Still add this to summary totals
135 + aggregate_into_summary_totals(totals, &z->metrics);
136 +
137 + return 1; // Continue processing next value
138 + }
139 +
140 + // Output this value normally using our helper function
141 + output_label_value(lwt->wb, qt, z->value, NULL, &z->metrics, &z->query_points);
142 +
143 + // Add to summary totals
144 + aggregate_into_summary_totals(totals, &z->metrics);
145 +
146 + return 1;
147 +}
148 +
149 +void query_target_summary_labels_v12(BUFFER *wb, QUERY_TARGET *qt, const char *key, bool v2, struct summary_total_counts *key_totals, struct summary_total_counts *value_totals) {
150 + buffer_json_member_add_array(wb, key);
151 + struct rrdlabels_formatting_v2 t = {
152 + .keys = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE),
153 + .v2 = v2,
154 + };
155 + for (long c = 0; c < (long) qt->instances.used; c++) {
156 + QUERY_INSTANCE *qi = query_instance(qt, c);
157 + RRDINSTANCE_ACQUIRED *ria = qi->ria;
158 + t.qi = qi;
159 + rrdlabels_walkthrough_read(rrdinstance_acquired_labels(ria), rrdlabels_formatting_v2, &t);
160 + }
161 +
162 + size_t cardinality_limit = qt->request.cardinality_limit;
163 +
164 + struct rrdlabels_keys_dict_entry *d;
165 + dfe_start_read(t.keys, d) {
166 + if(v2) {
167 + buffer_json_add_array_item_object(wb);
168 + buffer_json_member_add_string(wb, "id", d_dfe.name);
169 +
170 + // Only include detailed statistics if MINIMAL_STATS option is not set
171 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
172 + query_target_metric_counts(wb, &d->metrics);
173 + }
174 +
175 + query_target_points_statistics(wb, qt, &d->query_points);
176 + aggregate_into_summary_totals(key_totals, &d->metrics);
177 + buffer_json_member_add_array(wb, JSKEY(label_values));
178 +
179 + // Apply cardinality limiting to label values
180 + size_t values_count = dictionary_entries(d->values);
181 +
182 + // Setup walkthrough data regardless of whether we'll apply cardinality limiting
183 + struct label_values_walkthrough_data vt = {
184 + .wb = wb,
185 + .totals = value_totals,
186 + .qt = qt,
187 + .cardinality_limit = (cardinality_limit > 0 && values_count > cardinality_limit) ? cardinality_limit : 0,
188 + .count = 0,
189 + .remaining = {
190 + .metrics = {0},
191 + .points = STORAGE_POINT_UNSET,
192 + .count = 0
193 + }
194 + };
195 +
196 + // Choose appropriate comparison function based on whether we need cardinality limiting
197 + dict_item_comparator_t comparator = (vt.cardinality_limit > 0) ?
198 + label_values_sorted_sum_compar :
199 + NULL; // No specific ordering for normal case
200 +
201 + if (comparator)
202 + dictionary_sorted_walkthrough_rw(d->values, DICTIONARY_LOCK_READ,
203 + label_values_walkthrough_cb, &vt, comparator);
204 + else
205 + dictionary_walkthrough_rw(d->values, DICTIONARY_LOCK_READ,
206 + label_values_walkthrough_cb, &vt);
207 +
208 + // Add the aggregated "remaining" value if there are any
209 + if (vt.remaining.count > 0) {
210 + char remaining_label[50];
211 + snprintfz(remaining_label, sizeof(remaining_label), "remaining %zu values", vt.remaining.count);
212 +
213 + // Use our helper function for consistency
214 + output_label_value(wb, qt, "__remaining_values__", remaining_label,
215 + &vt.remaining.metrics, &vt.remaining.points);
216 + }
217 +
218 + buffer_json_array_close(wb); // vl
219 + buffer_json_object_close(wb); // label key
220 +
221 + } else {
222 + // v1 format - no cardinality limiting here
223 + struct rrdlabels_key_value_dict_entry *z;
224 + dfe_start_read(d->values, z){
225 + buffer_json_add_array_item_array(wb);
226 + buffer_json_add_array_item_string(wb, z->key);
227 + buffer_json_add_array_item_string(wb, z->value);
228 + buffer_json_array_close(wb);
229 + }
230 + dfe_done(z);
231 + }
232 +
233 + dictionary_destroy(d->values);
234 + }
235 + dfe_done(d);
236 + dictionary_destroy(t.keys);
237 + buffer_json_array_close(wb);
238 +}
src/web/api/formatters/jsonwrap-summary-nodes.c new
+150
@@ -0,0 +1,150 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +// Helper structure for sorting and limiting items based on their contribution
7 +typedef struct {
8 + size_t index; // Original index in the array
9 + NETDATA_DOUBLE contribution; // Sorting metric (usually volume contribution)
10 + const char *name; // Name for display
11 +} CARDINALITY_ITEM;
12 +
13 +// Comparison function for sorting items by contribution
14 +static int cardinality_item_compare(const void *a, const void *b) {
15 + const CARDINALITY_ITEM *item_a = (const CARDINALITY_ITEM *)a;
16 + const CARDINALITY_ITEM *item_b = (const CARDINALITY_ITEM *)b;
17 +
18 + // Sort by contribution (highest first)
19 + if (item_a->contribution > item_b->contribution) return -1;
20 + if (item_a->contribution < item_b->contribution) return 1;
21 +
22 + // If equal contribution, sort alphabetically by name
23 + if (item_a->name && item_b->name)
24 + return strcmp(item_a->name, item_b->name);
25 +
26 + return 0;
27 +}
28 +
29 +void query_target_summary_nodes_v2(BUFFER *wb, QUERY_TARGET *qt, const char *key, struct summary_total_counts *totals) {
30 + buffer_json_member_add_array(wb, key);
31 + size_t count = qt->nodes.used;
32 + size_t cardinality_limit = qt->request.cardinality_limit;
33 +
34 + bool show_node_status = !(qt->request.options & RRDR_OPTION_MINIMAL_STATS);
35 +
36 + // Check if we need to apply cardinality limiting
37 + if (cardinality_limit > 0 && count > cardinality_limit) {
38 + // We'll need to sort and limit the nodes
39 + CARDINALITY_ITEM *items = mallocz(sizeof(CARDINALITY_ITEM) * count);
40 +
41 + // Collect contribution data for each node
42 + for (size_t c = 0; c < count; c++) {
43 + QUERY_NODE *qn = query_node(qt, c);
44 + items[c].index = c;
45 + items[c].name = rrdhost_hostname(qn->rrdhost);
46 +
47 + // Use query points as the metric for contribution
48 + if (qt->query_points.sum > 0)
49 + items[c].contribution = qn->query_points.sum * 100.0 / qt->query_points.sum;
50 + else
51 + items[c].contribution = 0.0;
52 + }
53 +
54 + // Sort by contribution
55 + qsort(items, count, sizeof(CARDINALITY_ITEM), cardinality_item_compare);
56 +
57 + // First add the top (limit-1) nodes
58 + size_t nodes_to_show = cardinality_limit - 1;
59 + NETDATA_DOUBLE remaining_contribution = 0.0;
60 + size_t remaining_count = 0;
61 + QUERY_METRICS_COUNTS aggregated_metrics = {0};
62 + QUERY_INSTANCES_COUNTS aggregated_instances = {0};
63 + QUERY_ALERTS_COUNTS aggregated_alerts = {0};
64 + STORAGE_POINT aggregated_points = STORAGE_POINT_UNSET;
65 +
66 + for (size_t i = 0; i < count; i++) {
67 + if (i < nodes_to_show) {
68 + // Output this node normally
69 + QUERY_NODE *qn = query_node(qt, items[i].index);
70 + RRDHOST *host = qn->rrdhost;
71 + buffer_json_add_array_item_object(wb);
72 + buffer_json_node_add_v2(wb, host, qn->slot, qn->duration_ut, show_node_status);
73 +
74 + // Only include detailed statistics if MINIMAL_STATS option is not set
75 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
76 + query_target_instance_counts(wb, &qn->instances);
77 + query_target_metric_counts(wb, &qn->metrics);
78 + query_target_alerts_counts(wb, &qn->alerts, NULL, false);
79 + }
80 +
81 + query_target_points_statistics(wb, qt, &qn->query_points);
82 + buffer_json_object_close(wb);
83 +
84 + aggregate_into_summary_totals(totals, &qn->metrics);
85 + } else {
86 + // Aggregate the remaining nodes
87 + QUERY_NODE *qn = query_node(qt, items[i].index);
88 + remaining_contribution += items[i].contribution;
89 + remaining_count++;
90 +
91 + // Aggregate metrics, instances, and alerts counts
92 + aggregate_metrics_counts(&aggregated_metrics, &qn->metrics);
93 + aggregate_instances_counts(&aggregated_instances, &qn->instances);
94 + aggregate_alerts_counts(&aggregated_alerts, &qn->alerts);
95 +
96 + // Aggregate points
97 + storage_point_merge_to(aggregated_points, qn->query_points);
98 +
99 + aggregate_into_summary_totals(totals, &qn->metrics);
100 + }
101 + }
102 +
103 + // Add the aggregated "remaining" node if there are any
104 + if (remaining_count > 0) {
105 + buffer_json_add_array_item_object(wb);
106 +
107 + // Add basic info for the aggregated node
108 + char remaining_label[50];
109 + snprintfz(remaining_label, sizeof(remaining_label), "remaining %zu nodes", remaining_count);
110 +
111 + buffer_json_member_add_string(wb, "id", "__remaining_nodes__");
112 + buffer_json_member_add_string(wb, JSKEY(hostname), remaining_label);
113 + buffer_json_member_add_double(wb, JSKEY(contribution), remaining_contribution);
114 +
115 + // Only include detailed statistics if MINIMAL_STATS option is not set
116 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
117 + query_target_instance_counts(wb, &aggregated_instances);
118 + query_target_metric_counts(wb, &aggregated_metrics);
119 + query_target_alerts_counts(wb, &aggregated_alerts, NULL, false);
120 + }
121 +
122 + query_target_points_statistics(wb, qt, &aggregated_points);
123 + buffer_json_object_close(wb);
124 + }
125 +
126 + freez(items);
127 + } else {
128 + // No limiting needed, output all nodes
129 + for (size_t c = 0; c < count; c++) {
130 + QUERY_NODE *qn = query_node(qt, c);
131 + RRDHOST *host = qn->rrdhost;
132 + buffer_json_add_array_item_object(wb);
133 + buffer_json_node_add_v2(wb, host, qn->slot, qn->duration_ut, show_node_status);
134 +
135 + // Only include detailed statistics if MINIMAL_STATS option is not set
136 + if (!(qt->window.options & RRDR_OPTION_MINIMAL_STATS)) {
137 + query_target_instance_counts(wb, &qn->instances);
138 + query_target_metric_counts(wb, &qn->metrics);
139 + query_target_alerts_counts(wb, &qn->alerts, NULL, false);
140 + }
141 +
142 + query_target_points_statistics(wb, qt, &qn->query_points);
143 + buffer_json_object_close(wb);
144 +
145 + aggregate_into_summary_totals(totals, &qn->metrics);
146 + }
147 + }
148 +
149 + buffer_json_array_close(wb);
150 +}
src/web/api/formatters/jsonwrap-v1.c new
+189
@@ -0,0 +1,189 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +#include "libnetdata/json/json-keys.h"
6 +
7 +static inline long jsonwrap_v1_chart_ids(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
8 + QUERY_TARGET *qt = r->internal.qt;
9 + const long query_used = qt->query.used;
10 + long c, i;
11 +
12 + buffer_json_member_add_array(wb, key);
13 + for (c = 0, i = 0; c < query_used; c++) {
14 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
15 + continue;
16 +
17 + QUERY_METRIC *qm = query_metric(qt, c);
18 + QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
19 + buffer_json_add_array_item_string(wb, rrdinstance_acquired_id(qi->ria));
20 + i++;
21 + }
22 + buffer_json_array_close(wb);
23 +
24 + return i;
25 +}
26 +
27 +static inline long query_target_chart_labels_filter_v1(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
28 + QUERY_TARGET *qt = r->internal.qt;
29 + const long query_used = qt->query.used;
30 + long c, i = 0;
31 +
32 + buffer_json_member_add_object(wb, key);
33 +
34 + SIMPLE_PATTERN *pattern = qt->instances.chart_label_key_pattern;
35 + char *label_key = NULL;
36 + while (pattern && (label_key = simple_pattern_iterate(&pattern))) {
37 + buffer_json_member_add_array(wb, label_key);
38 +
39 + for (c = 0, i = 0; c < query_used; c++) {
40 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
41 + continue;
42 +
43 + QUERY_METRIC *qm = query_metric(qt, c);
44 + QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
45 + rrdlabels_value_to_buffer_array_item_or_null(rrdinstance_acquired_labels(qi->ria), wb, label_key);
46 + i++;
47 + }
48 + buffer_json_array_close(wb);
49 + }
50 +
51 + buffer_json_object_close(wb);
52 +
53 + return i;
54 +}
55 +
56 +static inline long query_target_metrics_latest_values(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
57 + QUERY_TARGET *qt = r->internal.qt;
58 + const long query_used = qt->query.used;
59 + long c, i;
60 +
61 + buffer_json_member_add_array(wb, key);
62 +
63 + for(c = 0, i = 0; c < query_used ;c++) {
64 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
65 + continue;
66 +
67 + QUERY_METRIC *qm = query_metric(qt, c);
68 + QUERY_DIMENSION *qd = query_dimension(qt, qm->link.query_dimension_id);
69 + buffer_json_add_array_item_double(wb, rrdmetric_acquired_last_stored_value(qd->rma));
70 + i++;
71 + }
72 +
73 + buffer_json_array_close(wb);
74 +
75 + return i;
76 +}
77 +
78 +static inline size_t rrdr_dimension_view_latest_values(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
79 + buffer_json_member_add_array(wb, key);
80 +
81 + size_t c, i;
82 + for(c = 0, i = 0; c < r->d ; c++) {
83 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
84 + continue;
85 +
86 + i++;
87 +
88 + NETDATA_DOUBLE *cn = &r->v[ (rrdr_rows(r) - 1) * r->d ];
89 + RRDR_VALUE_FLAGS *co = &r->o[ (rrdr_rows(r) - 1) * r->d ];
90 + NETDATA_DOUBLE n = cn[c];
91 +
92 + if(co[c] & RRDR_VALUE_EMPTY) {
93 + if(options & RRDR_OPTION_NULL2ZERO)
94 + buffer_json_add_array_item_double(wb, 0.0);
95 + else
96 + buffer_json_add_array_item_double(wb, NAN);
97 + }
98 + else
99 + buffer_json_add_array_item_double(wb, n);
100 + }
101 +
102 + buffer_json_array_close(wb);
103 +
104 + return i;
105 +}
106 +
107 +void rrdr_json_wrapper_begin(RRDR *r, BUFFER *wb) {
108 + QUERY_TARGET *qt = r->internal.qt;
109 + DATASOURCE_FORMAT format = qt->request.format;
110 + RRDR_OPTIONS options = qt->window.options;
111 +
112 + long rows = rrdr_rows(r);
113 +
114 + char kq[2] = "", // key quote
115 + sq[2] = ""; // string quote
116 +
117 + if( options & RRDR_OPTION_GOOGLE_JSON ) {
118 + kq[0] = '\0';
119 + sq[0] = '\'';
120 + }
121 + else {
122 + kq[0] = '"';
123 + sq[0] = '"';
124 + }
125 +
126 + buffer_json_initialize(
127 + wb, kq, sq, 0, true, (options & RRDR_OPTION_MINIFY) ? BUFFER_JSON_OPTIONS_MINIFY : BUFFER_JSON_OPTIONS_DEFAULT);
128 +
129 + json_keys_init((options & RRDR_OPTION_LONG_JSON_KEYS) ? JSON_KEYS_OPTION_LONG_KEYS : 0);
130 + buffer_json_member_add_uint64(wb, "api", 1);
131 + buffer_json_member_add_string(wb, "id", qt->id);
132 + buffer_json_member_add_string(wb, "name", qt->id);
133 + buffer_json_member_add_time_t(wb, "view_update_every", r->view.update_every);
134 + buffer_json_member_add_time_t(wb, "update_every", qt->db.minimum_latest_update_every_s);
135 + buffer_json_member_add_time_t_formatted(wb, "first_entry", qt->db.first_time_s, options & RRDR_OPTION_RFC3339);
136 + buffer_json_member_add_time_t_formatted(wb, "last_entry", qt->db.last_time_s, options & RRDR_OPTION_RFC3339);
137 + buffer_json_member_add_time_t_formatted(wb, "after", r->view.after, options & RRDR_OPTION_RFC3339);
138 + buffer_json_member_add_time_t_formatted(wb, "before", r->view.before, options & RRDR_OPTION_RFC3339);
139 + buffer_json_member_add_string(wb, "group", time_grouping_tostring(qt->request.time_group_method));
140 + rrdr_options_to_buffer_json_array(wb, "options", options);
141 +
142 + if(!rrdr_dimension_names(wb, "dimension_names", r, options))
143 + rows = 0;
144 +
145 + if(!rrdr_dimension_ids(wb, "dimension_ids", r, options))
146 + rows = 0;
147 +
148 + if (options & RRDR_OPTION_ALL_DIMENSIONS) {
149 + query_target_summary_instances_v1(wb, qt, "full_chart_list");
150 + query_target_summary_dimensions_v12(wb, qt, "full_dimension_list", false, NULL);
151 + query_target_summary_labels_v12(wb, qt, "full_chart_labels", false, NULL, NULL);
152 + }
153 +
154 + query_target_functions(wb, "functions", r);
155 +
156 + if (!qt->request.st && !jsonwrap_v1_chart_ids(wb, "chart_ids", r, options))
157 + rows = 0;
158 +
159 + if (qt->instances.chart_label_key_pattern && !query_target_chart_labels_filter_v1(wb, "chart_labels", r, options))
160 + rows = 0;
161 +
162 + if(!query_target_metrics_latest_values(wb, "latest_values", r, options))
163 + rows = 0;
164 +
165 + size_t dimensions = rrdr_dimension_view_latest_values(wb, "view_latest_values", r, options);
166 + if(!dimensions)
167 + rows = 0;
168 +
169 + buffer_json_member_add_uint64(wb, "dimensions", dimensions);
170 + buffer_json_member_add_uint64(wb, "points", rows);
171 + buffer_json_member_add_string(wb, "format", rrdr_format_to_string(format));
172 +
173 + buffer_json_member_add_array(wb, "db_points_per_tier");
174 + for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++)
175 + buffer_json_add_array_item_uint64(wb, qt->db.tiers[tier].points);
176 + buffer_json_array_close(wb);
177 +
178 + if(options & RRDR_OPTION_DEBUG)
179 + jsonwrap_query_plan(r, wb, options);
180 +}
181 +
182 +void rrdr_json_wrapper_end(RRDR *r, BUFFER *wb) {
183 + buffer_json_member_add_double(wb, "min", r->view.min);
184 + buffer_json_member_add_double(wb, "max", r->view.max);
185 +
186 + buffer_json_query_timings(wb, "timings", &r->internal.qt->timings);
187 + buffer_json_finalize(wb);
188 + json_keys_reset();
189 +}
src/web/api/formatters/jsonwrap-v2.c new
+539
@@ -0,0 +1,539 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +#include "web/mcp/mcp.h"
6 +#include "libnetdata/json/json-keys.h"
7 +
8 +void buffer_json_agent_status_id(BUFFER *wb, size_t ai, usec_t duration_ut) {
9 + buffer_json_member_add_object(wb, JSKEY(status));
10 + {
11 + buffer_json_member_add_uint64(wb, JSKEY(agent_index), ai);
12 + buffer_json_member_add_uint64(wb, "code", 200);
13 + buffer_json_member_add_string(wb, "msg", "");
14 + if (duration_ut)
15 + buffer_json_member_add_double(wb, "ms", (NETDATA_DOUBLE) duration_ut / 1000.0);
16 + }
17 + buffer_json_object_close(wb);
18 +}
19 +
20 +void buffer_json_node_add_v2(BUFFER *wb, RRDHOST *host, size_t ni, usec_t duration_ut, bool status) {
21 + buffer_json_member_add_string(wb, JSKEY(machine_guid), host->machine_guid);
22 +
23 + if(!UUIDiszero(host->node_id))
24 + buffer_json_member_add_uuid(wb, JSKEY(node_id), host->node_id.uuid);
25 + buffer_json_member_add_string(wb, JSKEY(hostname), rrdhost_hostname(host));
26 + buffer_json_member_add_uint64(wb, JSKEY(node_index), ni);
27 +
28 + if(status)
29 + buffer_json_agent_status_id(wb, 0, duration_ut);
30 +}
31 +
32 +static void query_target_combined_chart_type(BUFFER *wb, QUERY_TARGET *qt, size_t contexts) {
33 + if(contexts >= 1)
34 + buffer_json_member_add_string(wb, "chart_type", rrdset_type_name(rrdcontext_acquired_chart_type(qt->contexts.array[0].rca)));
35 +}
36 +
37 +static void query_target_title(BUFFER *wb, QUERY_TARGET *qt, size_t contexts) {
38 + if(contexts == 1) {
39 + buffer_json_member_add_string(wb, "title", rrdcontext_acquired_title(qt->contexts.array[0].rca));
40 + }
41 + else if(contexts > 1) {
42 + BUFFER *t = buffer_create(0, NULL);
43 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
44 +
45 + buffer_strcat(t, "Chart for contexts: ");
46 +
47 + size_t added = 0;
48 + for(size_t c = 0; c < qt->contexts.used ;c++) {
49 + bool *set = dictionary_set(dict, rrdcontext_acquired_id(qt->contexts.array[c].rca), NULL, sizeof(*set));
50 + if(!*set) {
51 + *set = true;
52 + if(added)
53 + buffer_fast_strcat(t, ", ", 2);
54 +
55 + buffer_strcat(t, rrdcontext_acquired_id(qt->contexts.array[c].rca));
56 + added++;
57 + }
58 + }
59 + buffer_json_member_add_string(wb, "title", buffer_tostring(t));
60 + dictionary_destroy(dict);
61 + buffer_free(t);
62 + }
63 +}
64 +
65 +void version_hashes_api_v2(BUFFER *wb, struct query_versions *versions) {
66 + buffer_json_member_add_object(wb, "versions");
67 + buffer_json_member_add_uint64(wb, "routing_hard_hash", 1);
68 + buffer_json_member_add_uint64(wb, "nodes_hard_hash", dictionary_version(rrdhost_root_index));
69 + buffer_json_member_add_uint64(wb, "contexts_hard_hash", versions->contexts_hard_hash);
70 + buffer_json_member_add_uint64(wb, "contexts_soft_hash", versions->contexts_soft_hash);
71 + buffer_json_member_add_uint64(wb, "alerts_hard_hash", versions->alerts_hard_hash);
72 + buffer_json_member_add_uint64(wb, "alerts_soft_hash", versions->alerts_soft_hash);
73 + buffer_json_object_close(wb);
74 +}
75 +
76 +static void query_target_combined_units_v2(BUFFER *wb, QUERY_TARGET *qt, size_t contexts, bool ignore_percentage) {
77 + if(!ignore_percentage && query_target_has_percentage_units(qt)) {
78 + buffer_json_member_add_string(wb, "units", "%");
79 + }
80 + else if(contexts == 1) {
81 + buffer_json_member_add_string(wb, "units", rrdcontext_acquired_units(qt->contexts.array[0].rca));
82 + }
83 + else if(contexts > 1) {
84 + DICTIONARY *dict = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
85 + for(size_t c = 0; c < qt->contexts.used ;c++)
86 + dictionary_set(dict, rrdcontext_acquired_units(qt->contexts.array[c].rca), NULL, 0);
87 +
88 + if(dictionary_entries(dict) == 1)
89 + buffer_json_member_add_string(wb, "units", rrdcontext_acquired_units(qt->contexts.array[0].rca));
90 + else {
91 + buffer_json_member_add_array(wb, "units");
92 + const char *s;
93 + dfe_start_read(dict, s)
94 + buffer_json_add_array_item_string(wb, s_dfe.name);
95 + dfe_done(s);
96 + buffer_json_array_close(wb);
97 + }
98 + dictionary_destroy(dict);
99 + }
100 +}
101 +
102 +static inline void rrdr_dimension_query_points_statistics(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options, bool dview) {
103 + STORAGE_POINT *sp = (dview) ? r->dview : r->dqp;
104 + NETDATA_DOUBLE anomaly_rate_multiplier = (dview) ? RRDR_DVIEW_ANOMALY_COUNT_MULTIPLIER : 1.0;
105 +
106 + if(unlikely(!sp))
107 + return;
108 +
109 + if(key)
110 + buffer_json_member_add_object(wb, key);
111 +
112 + buffer_json_member_add_array(wb, "min");
113 + for(size_t c = 0; c < r->d ; c++) {
114 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
115 + continue;
116 +
117 + buffer_json_add_array_item_double(wb, sp[c].min);
118 + }
119 + buffer_json_array_close(wb);
120 +
121 + buffer_json_member_add_array(wb, "max");
122 + for(size_t c = 0; c < r->d ; c++) {
123 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
124 + continue;
125 +
126 + buffer_json_add_array_item_double(wb, sp[c].max);
127 + }
128 + buffer_json_array_close(wb);
129 +
130 + if(options & RRDR_OPTION_RETURN_RAW) {
131 + buffer_json_member_add_array(wb, "sum");
132 + for(size_t c = 0; c < r->d ; c++) {
133 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
134 + continue;
135 +
136 + buffer_json_add_array_item_double(wb, sp[c].sum);
137 + }
138 + buffer_json_array_close(wb);
139 +
140 + buffer_json_member_add_array(wb, JSKEY(count));
141 + for(size_t c = 0; c < r->d ; c++) {
142 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
143 + continue;
144 +
145 + buffer_json_add_array_item_uint64(wb, sp[c].count);
146 + }
147 + buffer_json_array_close(wb);
148 +
149 + buffer_json_member_add_array(wb, JSKEY(anomaly_count));
150 + for(size_t c = 0; c < r->d ; c++) {
151 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
152 + continue;
153 +
154 + buffer_json_add_array_item_uint64(wb, storage_point_anomaly_rate(sp[c]) / anomaly_rate_multiplier / 100.0 * sp[c].count);
155 + }
156 + buffer_json_array_close(wb);
157 + }
158 + else {
159 + NETDATA_DOUBLE sum = 0.0;
160 + for(size_t c = 0; c < r->d ; c++) {
161 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
162 + continue;
163 +
164 + sum += ABS(sp[c].sum);
165 + }
166 +
167 + buffer_json_member_add_array(wb, "avg");
168 + for(size_t c = 0; c < r->d ; c++) {
169 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
170 + continue;
171 +
172 + buffer_json_add_array_item_double(wb, storage_point_average_value(sp[c]));
173 + }
174 + buffer_json_array_close(wb);
175 +
176 + buffer_json_member_add_array(wb, JSKEY(anomaly_rate));
177 + for(size_t c = 0; c < r->d ; c++) {
178 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
179 + continue;
180 +
181 + buffer_json_add_array_item_double(wb, storage_point_anomaly_rate(sp[c]) / anomaly_rate_multiplier);
182 + }
183 + buffer_json_array_close(wb);
184 +
185 + buffer_json_member_add_array(wb, JSKEY(contribution));
186 + for(size_t c = 0; c < r->d ; c++) {
187 + if (!rrdr_dimension_should_be_exposed(r->od[c], options))
188 + continue;
189 +
190 + NETDATA_DOUBLE con = (sum > 0.0) ? ABS(sp[c].sum) * 100.0 / sum : 0.0;
191 + buffer_json_add_array_item_double(wb, con);
192 + }
193 + buffer_json_array_close(wb);
194 + }
195 +
196 + if(key)
197 + buffer_json_object_close(wb);
198 +}
199 +
200 +static void rrdr_grouped_by_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options __maybe_unused) {
201 + QUERY_TARGET *qt = r->internal.qt;
202 +
203 + buffer_json_member_add_array(wb, key);
204 +
205 + // find the deeper group-by
206 + ssize_t g = 0;
207 + for(g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
208 + if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_NONE)
209 + break;
210 + }
211 +
212 + if(g > 0)
213 + g--;
214 +
215 + RRDR_GROUP_BY group_by = qt->request.group_by[g].group_by;
216 +
217 + if(group_by & RRDR_GROUP_BY_SELECTED)
218 + buffer_json_add_array_item_string(wb, "selected");
219 +
220 + else if(group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
221 + buffer_json_add_array_item_string(wb, "percentage-of-instance");
222 +
223 + else {
224 +
225 + if(group_by & RRDR_GROUP_BY_DIMENSION)
226 + buffer_json_add_array_item_string(wb, "dimension");
227 +
228 + if(group_by & RRDR_GROUP_BY_INSTANCE)
229 + buffer_json_add_array_item_string(wb, "instance");
230 +
231 + if(group_by & RRDR_GROUP_BY_LABEL) {
232 + BUFFER *b = buffer_create(0, NULL);
233 + for (size_t l = 0; l < qt->group_by[g].used; l++) {
234 + buffer_flush(b);
235 + buffer_fast_strcat(b, "label:", 6);
236 + buffer_strcat(b, qt->group_by[g].label_keys[l]);
237 + buffer_json_add_array_item_string(wb, buffer_tostring(b));
238 + }
239 + buffer_free(b);
240 + }
241 +
242 + if(group_by & RRDR_GROUP_BY_NODE)
243 + buffer_json_add_array_item_string(wb, "node");
244 +
245 + if(group_by & RRDR_GROUP_BY_CONTEXT)
246 + buffer_json_add_array_item_string(wb, "context");
247 +
248 + if(group_by & RRDR_GROUP_BY_UNITS)
249 + buffer_json_add_array_item_string(wb, "units");
250 + }
251 +
252 + buffer_json_array_close(wb); // group_by_order
253 +}
254 +
255 +static void rrdr_dimension_units_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options, bool ignore_percentage) {
256 + if(!r->du)
257 + return;
258 +
259 + bool percentage = !ignore_percentage && query_target_has_percentage_units(r->internal.qt);
260 +
261 + buffer_json_member_add_array(wb, key);
262 + for(size_t c = 0; c < r->d ; c++) {
263 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
264 + continue;
265 +
266 + if(percentage)
267 + buffer_json_add_array_item_string(wb, "%");
268 + else
269 + buffer_json_add_array_item_string(wb, string2str(r->du[c]));
270 + }
271 + buffer_json_array_close(wb);
272 +}
273 +
274 +static void rrdr_dimension_priority_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
275 + if(!r->dp)
276 + return;
277 +
278 + buffer_json_member_add_array(wb, key);
279 + for(size_t c = 0; c < r->d ; c++) {
280 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
281 + continue;
282 +
283 + buffer_json_add_array_item_uint64(wb, r->dp[c]);
284 + }
285 + buffer_json_array_close(wb);
286 +}
287 +
288 +static void rrdr_dimension_aggregated_array_v2(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
289 + if(!r->dgbc)
290 + return;
291 +
292 + buffer_json_member_add_array(wb, key);
293 + for(size_t c = 0; c < r->d ;c++) {
294 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
295 + continue;
296 +
297 + buffer_json_add_array_item_uint64(wb, r->dgbc[c]);
298 + }
299 + buffer_json_array_close(wb);
300 +}
301 +
302 +void rrdr_json_wrapper_begin2(RRDR *r, BUFFER *wb) {
303 + QUERY_TARGET *qt = r->internal.qt;
304 + RRDR_OPTIONS options = qt->window.options;
305 +
306 + char kq[2] = "\"", // key quote
307 + sq[2] = "\""; // string quote
308 +
309 + if(unlikely(options & RRDR_OPTION_GOOGLE_JSON)) {
310 + kq[0] = '\0';
311 + sq[0] = '\'';
312 + }
313 +
314 + buffer_json_initialize(
315 + wb, kq, sq, 0, true, (options & RRDR_OPTION_MINIFY) ? BUFFER_JSON_OPTIONS_MINIFY : BUFFER_JSON_OPTIONS_DEFAULT);
316 +
317 + json_keys_init((options & RRDR_OPTION_LONG_JSON_KEYS) ? JSON_KEYS_OPTION_LONG_KEYS : 0);
318 + buffer_json_member_add_uint64(wb, "api", qt->request.version);
319 +
320 + if(options & RRDR_OPTION_DEBUG) {
321 + buffer_json_member_add_string(wb, "id", qt->id);
322 + buffer_json_member_add_object(wb, "request");
323 + {
324 + buffer_json_member_add_string(wb, "format", rrdr_format_to_string(qt->request.format));
325 + rrdr_options_to_buffer_json_array(wb, "options", qt->request.options);
326 +
327 + buffer_json_member_add_object(wb, "scope");
328 + buffer_json_member_add_string(wb, "scope_nodes", qt->request.scope_nodes);
329 + buffer_json_member_add_string(wb, "scope_contexts", qt->request.scope_contexts);
330 + buffer_json_object_close(wb); // scope
331 +
332 + buffer_json_member_add_object(wb, "selectors");
333 + if (qt->request.host)
334 + buffer_json_member_add_string(wb, "nodes", rrdhost_hostname(qt->request.host));
335 + else
336 + buffer_json_member_add_string(wb, "nodes", qt->request.nodes);
337 + buffer_json_member_add_string(wb, "contexts", qt->request.contexts);
338 + buffer_json_member_add_string(wb, "instances", qt->request.instances);
339 + buffer_json_member_add_string(wb, "dimensions", qt->request.dimensions);
340 + buffer_json_member_add_string(wb, "labels", qt->request.labels);
341 + buffer_json_member_add_string(wb, "alerts", qt->request.alerts);
342 + buffer_json_object_close(wb); // selectors
343 +
344 + buffer_json_member_add_object(wb, "window");
345 + buffer_json_member_add_time_t_formatted(wb, "after", qt->request.after, qt->request.options & RRDR_OPTION_RFC3339);
346 + buffer_json_member_add_time_t_formatted(wb, "before", qt->request.before, qt->request.options & RRDR_OPTION_RFC3339);
347 + buffer_json_member_add_uint64(wb, "points", qt->request.points);
348 + if (qt->request.options & RRDR_OPTION_SELECTED_TIER)
349 + buffer_json_member_add_uint64(wb, "tier", qt->request.tier);
350 + else
351 + buffer_json_member_add_string(wb, "tier", NULL);
352 + buffer_json_object_close(wb); // window
353 +
354 + buffer_json_member_add_object(wb, "aggregations");
355 + {
356 + buffer_json_member_add_object(wb, "time");
357 + buffer_json_member_add_string(wb, "time_group", time_grouping_tostring(qt->request.time_group_method));
358 + buffer_json_member_add_string(wb, "time_group_options", qt->request.time_group_options);
359 + if (qt->request.resampling_time > 0)
360 + buffer_json_member_add_time_t(wb, "time_resampling", qt->request.resampling_time);
361 + else
362 + buffer_json_member_add_string(wb, "time_resampling", NULL);
363 + buffer_json_object_close(wb); // time
364 +
365 + buffer_json_member_add_array(wb, "metrics");
366 + for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
367 + if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_NONE)
368 + break;
369 +
370 + buffer_json_add_array_item_object(wb);
371 + {
372 + buffer_json_member_add_array(wb, "group_by");
373 + buffer_json_group_by_to_array(wb, qt->request.group_by[g].group_by);
374 + buffer_json_array_close(wb);
375 +
376 + buffer_json_member_add_array(wb, "group_by_label");
377 + for (size_t l = 0; l < qt->group_by[g].used; l++)
378 + buffer_json_add_array_item_string(wb, qt->group_by[g].label_keys[l]);
379 + buffer_json_array_close(wb);
380 +
381 + buffer_json_member_add_string(
382 + wb, "aggregation",group_by_aggregate_function_to_string(qt->request.group_by[g].aggregation));
383 + }
384 + buffer_json_object_close(wb);
385 + }
386 + buffer_json_array_close(wb); // group_by
387 + }
388 + buffer_json_object_close(wb); // aggregations
389 +
390 + buffer_json_member_add_uint64(wb, "timeout", qt->request.timeout_ms);
391 + }
392 + buffer_json_object_close(wb); // request
393 + }
394 +
395 + if(!(options & RRDR_OPTION_MINIMAL_STATS))
396 + version_hashes_api_v2(wb, &qt->versions);
397 +
398 + struct summary_total_counts
399 + nodes_totals = { 0 },
400 + contexts_totals = { 0 },
401 + instances_totals = { 0 },
402 + metrics_totals = { 0 },
403 + label_key_totals = { 0 },
404 + label_key_value_totals = { 0 };
405 +
406 + buffer_json_member_add_object(wb, "summary");
407 + {
408 + if(options & RRDR_OPTION_MCP_INFO)
409 + buffer_json_member_add_string(wb, "info", MCP_QUERY_INFO_SUMMARY_SECTION);
410 +
411 + query_target_summary_nodes_v2(wb, qt, "nodes", &nodes_totals);
412 + r->internal.contexts = query_target_summary_contexts_v2(wb, qt, "contexts", &contexts_totals);
413 + query_target_summary_instances_v2(wb, qt, "instances", &instances_totals);
414 + query_target_summary_dimensions_v12(wb, qt, "dimensions", true, &metrics_totals);
415 + query_target_summary_labels_v12(wb, qt, "labels", true, &label_key_totals, &label_key_value_totals);
416 + if(!(options & RRDR_OPTION_MINIMAL_STATS))
417 + query_target_summary_alerts_v2(wb, qt, "alerts");
418 + }
419 + if(query_target_aggregatable(qt)) {
420 + buffer_json_member_add_object(wb, "globals");
421 + query_target_points_statistics(wb, qt, &qt->query_points);
422 + buffer_json_object_close(wb); // globals
423 + }
424 + buffer_json_object_close(wb); // summary
425 +
426 + // Only include the totals section if MINIMAL_STATS option is not set
427 + if (!(options & RRDR_OPTION_MINIMAL_STATS)) {
428 + buffer_json_member_add_object(wb, "totals");
429 + query_target_total_counts(wb, "nodes", &nodes_totals);
430 + query_target_total_counts(wb, "contexts", &contexts_totals);
431 + query_target_total_counts(wb, "instances", &instances_totals);
432 + query_target_total_counts(wb, "dimensions", &metrics_totals);
433 + query_target_total_counts(wb, "label_keys", &label_key_totals);
434 + query_target_total_counts(wb, "label_key_values", &label_key_value_totals);
435 + buffer_json_object_close(wb); // totals
436 + }
437 +
438 + if(options & RRDR_OPTION_SHOW_DETAILS) {
439 + buffer_json_member_add_object(wb, "detailed");
440 + query_target_detailed_objects_tree(wb, r, options);
441 + buffer_json_object_close(wb); // detailed
442 + }
443 +
444 + if(!(options & RRDR_OPTION_MINIMAL_STATS))
445 + query_target_functions(wb, "functions", r);
446 +}
447 +
448 +void rrdr_json_wrapper_end2(RRDR *r, BUFFER *wb) {
449 + QUERY_TARGET *qt = r->internal.qt;
450 + DATASOURCE_FORMAT format = qt->request.format;
451 + RRDR_OPTIONS options = qt->window.options;
452 +
453 + buffer_json_member_add_object(wb, "db");
454 + {
455 + if(options & RRDR_OPTION_MCP_INFO)
456 + buffer_json_member_add_string(wb, "info", MCP_QUERY_INFO_DATABASE_SECTION);
457 +
458 + buffer_json_member_add_uint64(wb, "tiers", nd_profile.storage_tiers);
459 + buffer_json_member_add_time_t(wb, "update_every", qt->db.minimum_latest_update_every_s);
460 + buffer_json_member_add_time_t_formatted(wb, "first_entry", qt->db.first_time_s, options & RRDR_OPTION_RFC3339);
461 + buffer_json_member_add_time_t_formatted(wb, "last_entry", qt->db.last_time_s, options & RRDR_OPTION_RFC3339);
462 +
463 + query_target_combined_units_v2(wb, qt, r->internal.contexts, true);
464 + buffer_json_member_add_object(wb, "dimensions");
465 + {
466 + rrdr_dimension_ids(wb, "ids", r, options);
467 + rrdr_dimension_units_array_v2(wb, "units", r, options, true);
468 + rrdr_dimension_query_points_statistics(wb, JSKEY(statistics), r, options, false);
469 + }
470 + buffer_json_object_close(wb); // dimensions
471 +
472 + buffer_json_member_add_array(wb, "per_tier");
473 + for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
474 + buffer_json_add_array_item_object(wb);
475 + buffer_json_member_add_uint64(wb, "tier", tier);
476 + buffer_json_member_add_uint64(wb, "queries", qt->db.tiers[tier].queries);
477 + buffer_json_member_add_uint64(wb, "points", qt->db.tiers[tier].points);
478 + buffer_json_member_add_time_t(wb, "update_every", qt->db.tiers[tier].update_every);
479 + buffer_json_member_add_time_t_formatted(wb, "first_entry", qt->db.tiers[tier].retention.first_time_s, options & RRDR_OPTION_RFC3339);
480 + buffer_json_member_add_time_t_formatted(wb, "last_entry", qt->db.tiers[tier].retention.last_time_s, options & RRDR_OPTION_RFC3339);
481 + buffer_json_object_close(wb);
482 + }
483 + buffer_json_array_close(wb);
484 + }
485 + buffer_json_object_close(wb);
486 +
487 + buffer_json_member_add_object(wb, "view");
488 + {
489 + if(options & RRDR_OPTION_MCP_INFO)
490 + buffer_json_member_add_string(wb, "info", MCP_QUERY_INFO_VIEW_SECTION);
491 +
492 + query_target_title(wb, qt, r->internal.contexts);
493 + buffer_json_member_add_time_t(wb, "update_every", r->view.update_every);
494 + buffer_json_member_add_time_t_formatted(wb, "after", r->view.after, options & RRDR_OPTION_RFC3339);
495 + buffer_json_member_add_time_t_formatted(wb, "before", r->view.before, options & RRDR_OPTION_RFC3339);
496 +
497 + if(options & RRDR_OPTION_DEBUG) {
498 + buffer_json_member_add_string(wb, "format", rrdr_format_to_string(format));
499 + rrdr_options_to_buffer_json_array(wb, "options", options);
500 + buffer_json_member_add_string(wb, "time_group", time_grouping_tostring(qt->request.time_group_method));
501 + }
502 +
503 + if(options & RRDR_OPTION_DEBUG) {
504 + buffer_json_member_add_object(wb, "partial_data_trimming");
505 + buffer_json_member_add_time_t(wb, "max_update_every", r->partial_data_trimming.max_update_every);
506 + buffer_json_member_add_time_t_formatted(wb, "expected_after", r->partial_data_trimming.expected_after, options & RRDR_OPTION_RFC3339);
507 + buffer_json_member_add_time_t_formatted(wb, "trimmed_after", r->partial_data_trimming.trimmed_after, options & RRDR_OPTION_RFC3339);
508 + buffer_json_object_close(wb);
509 + }
510 +
511 + if(options & RRDR_OPTION_RETURN_RAW)
512 + buffer_json_member_add_uint64(wb, "points", rrdr_rows(r));
513 +
514 + query_target_combined_units_v2(wb, qt, r->internal.contexts, false);
515 + query_target_combined_chart_type(wb, qt, r->internal.contexts);
516 + buffer_json_member_add_object(wb, "dimensions");
517 + {
518 + rrdr_grouped_by_array_v2(wb, "grouped_by", r, options);
519 + rrdr_dimension_ids(wb, "ids", r, options);
520 + rrdr_dimension_names(wb, "names", r, options);
521 + rrdr_dimension_units_array_v2(wb, "units", r, options, false);
522 + rrdr_dimension_priority_array_v2(wb, "priorities", r, options);
523 + rrdr_dimension_aggregated_array_v2(wb, "aggregated", r, options);
524 + rrdr_dimension_query_points_statistics(wb, JSKEY(statistics), r, options, true);
525 + rrdr_json_group_by_labels(wb, "labels", r, options);
526 + }
527 + buffer_json_object_close(wb); // dimensions
528 + buffer_json_member_add_double(wb, "min", r->view.min);
529 + buffer_json_member_add_double(wb, "max", r->view.max);
530 + }
531 + buffer_json_object_close(wb); // view
532 +
533 + if(!(options & RRDR_OPTION_MINIMAL_STATS)) {
534 + buffer_json_agents_v2(wb, &r->internal.qt->timings, 0, false, true, rrdr_options_to_contexts_options(options));
535 + buffer_json_cloud_timings(wb, "timings", &r->internal.qt->timings);
536 + }
537 + buffer_json_finalize(wb);
538 + json_keys_reset();
539 +}
src/web/api/formatters/jsonwrap.c new
+234
@@ -0,0 +1,234 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "jsonwrap.h"
4 +#include "jsonwrap-internal.h"
5 +
6 +
7 +ALWAYS_INLINE
8 +size_t rrdr_dimension_names(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
9 + const size_t dimensions = r->d;
10 + size_t c, i;
11 +
12 + buffer_json_member_add_array(wb, key);
13 + for(c = 0, i = 0; c < dimensions ; c++) {
14 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
15 + continue;
16 +
17 + buffer_json_add_array_item_string(wb, string2str(r->dn[c]));
18 + i++;
19 + }
20 + buffer_json_array_close(wb);
21 +
22 + return i;
23 +}
24 +
25 +ALWAYS_INLINE
26 +size_t rrdr_dimension_ids(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
27 + const size_t dimensions = r->d;
28 + size_t c, i;
29 +
30 + buffer_json_member_add_array(wb, key);
31 + for(c = 0, i = 0; c < dimensions ; c++) {
32 + if(!rrdr_dimension_should_be_exposed(r->od[c], options))
33 + continue;
34 +
35 + buffer_json_add_array_item_string(wb, string2str(r->di[c]));
36 + i++;
37 + }
38 + buffer_json_array_close(wb);
39 +
40 + return i;
41 +}
42 +
43 +ALWAYS_INLINE
44 +void query_target_functions(BUFFER *wb, const char *key, RRDR *r) {
45 + QUERY_TARGET *qt = r->internal.qt;
46 + const long query_used = qt->query.used;
47 +
48 + DICTIONARY *funcs = dictionary_create(DICT_OPTION_SINGLE_THREADED|DICT_OPTION_DONT_OVERWRITE_VALUE);
49 + RRDINSTANCE_ACQUIRED *ria = NULL;
50 + for (long c = 0; c < query_used ; c++) {
51 + QUERY_METRIC *qm = query_metric(qt, c);
52 + QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
53 + if(qi->ria == ria)
54 + continue;
55 +
56 + ria = qi->ria;
57 + chart_functions_to_dict(rrdinstance_acquired_functions(ria), funcs, NULL, 0);
58 + }
59 +
60 + buffer_json_member_add_array(wb, key);
61 + void *t; (void)t;
62 + dfe_start_read(funcs, t)
63 + buffer_json_add_array_item_string(wb, t_dfe.name);
64 + dfe_done(t);
65 + dictionary_destroy(funcs);
66 + buffer_json_array_close(wb);
67 +}
68 +
69 +ALWAYS_INLINE
70 +void query_target_total_counts(BUFFER *wb, const char *key, struct summary_total_counts *totals) {
71 + if(!totals->selected && !totals->queried && !totals->failed && !totals->excluded)
72 + return;
73 +
74 + buffer_json_member_add_object(wb, key);
75 +
76 + if(totals->selected)
77 + buffer_json_member_add_uint64(wb, JSKEY(selected), totals->selected);
78 +
79 + if(totals->excluded)
80 + buffer_json_member_add_uint64(wb, JSKEY(excluded), totals->excluded);
81 +
82 + if(totals->queried)
83 + buffer_json_member_add_uint64(wb, JSKEY(queried), totals->queried);
84 +
85 + if(totals->failed)
86 + buffer_json_member_add_uint64(wb, JSKEY(failed), totals->failed);
87 +
88 + buffer_json_object_close(wb);
89 +}
90 +
91 +ALWAYS_INLINE
92 +void query_target_metric_counts(BUFFER *wb, QUERY_METRICS_COUNTS *metrics) {
93 + if(!metrics->selected && !metrics->queried && !metrics->failed && !metrics->excluded)
94 + return;
95 +
96 + buffer_json_member_add_object(wb, JSKEY(dimensions));
97 +
98 + if(metrics->selected)
99 + buffer_json_member_add_uint64(wb, JSKEY(selected), metrics->selected);
100 +
101 + if(metrics->excluded)
102 + buffer_json_member_add_uint64(wb, JSKEY(excluded), metrics->excluded);
103 +
104 + if(metrics->queried)
105 + buffer_json_member_add_uint64(wb, JSKEY(queried), metrics->queried);
106 +
107 + if(metrics->failed)
108 + buffer_json_member_add_uint64(wb, JSKEY(failed), metrics->failed);
109 +
110 + buffer_json_object_close(wb);
111 +}
112 +
113 +ALWAYS_INLINE
114 +void query_target_instance_counts(BUFFER *wb, QUERY_INSTANCES_COUNTS *instances) {
115 + if(!instances->selected && !instances->queried && !instances->failed && !instances->excluded)
116 + return;
117 +
118 + buffer_json_member_add_object(wb, JSKEY(instances));
119 +
120 + if(instances->selected)
121 + buffer_json_member_add_uint64(wb, JSKEY(selected), instances->selected);
122 +
123 + if(instances->excluded)
124 + buffer_json_member_add_uint64(wb, JSKEY(excluded), instances->excluded);
125 +
126 + if(instances->queried)
127 + buffer_json_member_add_uint64(wb, JSKEY(queried), instances->queried);
128 +
129 + if(instances->failed)
130 + buffer_json_member_add_uint64(wb, JSKEY(failed), instances->failed);
131 +
132 + buffer_json_object_close(wb);
133 +}
134 +
135 +ALWAYS_INLINE
136 +void query_target_alerts_counts(BUFFER *wb, QUERY_ALERTS_COUNTS *alerts, const char *name, bool array) {
137 + if(!alerts->clear && !alerts->other && !alerts->critical && !alerts->warning)
138 + return;
139 +
140 + if(array)
141 + buffer_json_add_array_item_object(wb);
142 + else
143 + buffer_json_member_add_object(wb, JSKEY(alerts));
144 +
145 + if(name)
146 + buffer_json_member_add_string(wb, JSKEY(name), name);
147 +
148 + if(alerts->clear)
149 + buffer_json_member_add_uint64(wb, JSKEY(clear), alerts->clear);
150 +
151 + if(alerts->warning)
152 + buffer_json_member_add_uint64(wb, JSKEY(warning), alerts->warning);
153 +
154 + if(alerts->critical)
155 + buffer_json_member_add_uint64(wb, JSKEY(critical), alerts->critical);
156 +
157 + if(alerts->other)
158 + buffer_json_member_add_uint64(wb, JSKEY(other), alerts->other);
159 +
160 + buffer_json_object_close(wb);
161 +}
162 +
163 +ALWAYS_INLINE
164 +void query_target_points_statistics(BUFFER *wb, QUERY_TARGET *qt, STORAGE_POINT *sp) {
165 + if(!sp->count)
166 + return;
167 +
168 + buffer_json_member_add_object(wb, JSKEY(statistics));
169 +
170 + buffer_json_member_add_double(wb, "min", sp->min);
171 + buffer_json_member_add_double(wb, "max", sp->max);
172 +
173 + if(query_target_aggregatable(qt)) {
174 + buffer_json_member_add_uint64(wb, JSKEY(count), sp->count);
175 +
176 + buffer_json_member_add_double(wb, "sum", sp->sum);
177 + buffer_json_member_add_double(wb, JSKEY(volume), sp->sum * (NETDATA_DOUBLE) query_view_update_every(qt));
178 +
179 + buffer_json_member_add_uint64(wb, JSKEY(anomaly_count), sp->anomaly_count);
180 + }
181 + else {
182 + NETDATA_DOUBLE avg = (sp->count) ? sp->sum / (NETDATA_DOUBLE)sp->count : 0.0;
183 + buffer_json_member_add_double(wb, "avg", avg);
184 +
185 + NETDATA_DOUBLE arp = storage_point_anomaly_rate(*sp);
186 + buffer_json_member_add_double(wb, JSKEY(anomaly_rate), arp);
187 +
188 + NETDATA_DOUBLE con = (qt->query_points.sum > 0.0) ? sp->sum * 100.0 / qt->query_points.sum : 0.0;
189 + buffer_json_member_add_double(wb, JSKEY(contribution), con);
190 + }
191 + buffer_json_object_close(wb);
192 +}
193 +
194 +ALWAYS_INLINE
195 +void aggregate_metrics_counts(QUERY_METRICS_COUNTS *dst, const QUERY_METRICS_COUNTS *src) {
196 + dst->selected += src->selected;
197 + dst->excluded += src->excluded;
198 + dst->queried += src->queried;
199 + dst->failed += src->failed;
200 +}
201 +
202 +ALWAYS_INLINE
203 +void aggregate_instances_counts(QUERY_INSTANCES_COUNTS *dst, const QUERY_INSTANCES_COUNTS *src) {
204 + dst->selected += src->selected;
205 + dst->excluded += src->excluded;
206 + dst->queried += src->queried;
207 + dst->failed += src->failed;
208 +}
209 +
210 +ALWAYS_INLINE
211 +void aggregate_alerts_counts(QUERY_ALERTS_COUNTS *dst, const QUERY_ALERTS_COUNTS *src) {
212 + dst->clear += src->clear;
213 + dst->warning += src->warning;
214 + dst->critical += src->critical;
215 + dst->other += src->other;
216 +}
217 +
218 +ALWAYS_INLINE
219 +void aggregate_into_summary_totals(struct summary_total_counts *totals, QUERY_METRICS_COUNTS *metrics) {
220 + if(unlikely(!totals || !metrics))
221 + return;
222 +
223 + if(metrics->selected) {
224 + totals->selected++;
225 +
226 + if(metrics->queried)
227 + totals->queried++;
228 +
229 + else if(metrics->failed)
230 + totals->failed++;
231 + }
232 + else
233 + totals->excluded++;
234 +}
src/web/api/formatters/jsonwrap.h renamed
src/web/api/formatters/rrd2json.c
+1 -1
@@ -119,7 +119,7 @@ int data_query_execute(ONEWAYALLOC *owa, BUFFER *wb, QUERY_TARGET *qt, time_t *l
119 wrapper_begin_t wrapper_begin = rrdr_json_wrapper_begin;
120 wrapper_end_t wrapper_end = rrdr_json_wrapper_end;
121
122 - if(qt->request.version == 2) {
122 + if(qt->request.version >= 2) {
123 wrapper_begin = rrdr_json_wrapper_begin2;
124 wrapper_end = rrdr_json_wrapper_end2;
125 }
src/web/api/formatters/rrd2json.h
+1 -1
@@ -14,7 +14,7 @@
14
15 #include "web/api/formatters/rrdset2json.h"
16 #include "web/api/formatters/charts2json.h"
17 -#include "web/api/formatters/json_wrapper.h"
17 +#include "web/api/formatters/jsonwrap.h"
18
19 #include "web/server/web_client.h"
20
src/web/api/functions/function-bearer_get_token.c
+7 -7
@@ -27,7 +27,7 @@ static bool bearer_parse_json_payload(json_object *jobj, void *data, BUFFER *err
27 }
28
29 int function_bearer_get_token(BUFFER *wb, const char *function __maybe_unused, BUFFER *payload, const char *source) {
30 - if(!request_source_is_cloud(source))
30 + if(!user_auth_source_is_cloud(source))
31 return rrd_call_function_error(
32 wb, "Bearer tokens can only be provided via NC.", HTTP_RESP_BAD_REQUEST);
33
@@ -62,19 +62,19 @@ int call_function_bearer_get_token(RRDHOST *host, struct web_client *w, const ch
62 buffer_json_member_add_string(payload, "claim_id", claim_id);
63 buffer_json_member_add_string(payload, "machine_guid", machine_guid);
64 buffer_json_member_add_string(payload, "node_id", node_id);
65 - buffer_json_member_add_string(payload, "user_role", http_id2user_role(w->user_role));
66 - http_access2buffer_json_array(payload, "access", w->access);
67 - buffer_json_member_add_uuid(payload, "cloud_account_id", w->auth.cloud_account_id);
68 - buffer_json_member_add_string(payload, "client_name", w->auth.client_name);
65 + buffer_json_member_add_string(payload, "user_role", http_id2user_role(w->user_auth.user_role));
66 + http_access2buffer_json_array(payload, "access", w->user_auth.access);
67 + buffer_json_member_add_uuid(payload, "cloud_account_id", w->user_auth.cloud_account_id.uuid);
68 + buffer_json_member_add_string(payload, "client_name", w->user_auth.client_name);
69 buffer_json_finalize(payload);
70
71 CLEAN_BUFFER *source = buffer_create(0, NULL);
72 - web_client_api_request_vX_source_to_buffer(w, source);
72 + user_auth_to_source_buffer(&w->user_auth, source);
73
74 char transaction_str[UUID_COMPACT_STR_LEN];
75 uuid_unparse_lower_compact(w->transaction, transaction_str);
76 return rrd_function_run(host, w->response.data, 10,
77 - w->access, RRDFUNCTIONS_BEARER_GET_TOKEN, true,
77 + w->user_auth.access, RRDFUNCTIONS_BEARER_GET_TOKEN, true,
78 transaction_str, NULL, NULL,
79 NULL, NULL,
80 NULL, NULL,
src/web/api/functions/function-metrics-cardinality.h
+1 -1
@@ -5,7 +5,7 @@
5
6 #include "libnetdata/libnetdata.h"
7
8 -#define RRDFUNCTIONS_METRICS_CARDINALITY_HELP "Shows metrics cardinality per context across all nodes."
8 +#define RRDFUNCTIONS_METRICS_CARDINALITY_HELP "Displays metrics cardinality statistics showing distribution of instances and time-series across contexts and nodes. To change grouping, append parameter to function name: 'netdata-metrics-cardinality' (default, group by context) or 'netdata-metrics-cardinality group:by-node' (group by node)."
9
10 int function_metrics_cardinality(BUFFER *wb, const char *function, BUFFER *payload, const char *source);
11
src/web/api/functions/function-streaming.h
+1 -1
@@ -5,7 +5,7 @@
5
6 #include "database/rrd.h"
7
8 -#define RRDFUNCTIONS_STREAMING_HELP "Streaming status for parents and children."
8 +#define RRDFUNCTIONS_STREAMING_HELP "Shows real-time streaming connections and replication status between parent and child nodes, including connection health, data flow metrics, and ML status."
9
10 int function_streaming(BUFFER *wb, const char *function, BUFFER *payload, const char *source);
11
src/web/api/http_auth.c
+4 -4
@@ -317,9 +317,9 @@ bool web_client_bearer_token_auth(struct web_client *w, const char *v) {
317 if(item) {
318 struct bearer_token *bt = dictionary_acquired_item_value(item);
319 if (bt->expires_s > now_realtime_sec()) {
320 - strncpyz(w->auth.client_name, bt->client_name, sizeof(w->auth.client_name) - 1);
321 - uuid_copy(w->auth.cloud_account_id, bt->cloud_account_id);
322 - web_client_set_permissions(w, bt->access, bt->user_role, WEB_CLIENT_FLAG_AUTH_BEARER);
320 + strncpyz(w->user_auth.client_name, bt->client_name, sizeof(w->user_auth.client_name) - 1);
321 + uuid_copy(w->user_auth.cloud_account_id.uuid, bt->cloud_account_id);
322 + web_client_set_permissions(w, bt->access, bt->user_role, USER_AUTH_METHOD_BEARER);
323 rc = true;
324 }
325
@@ -349,7 +349,7 @@ void bearer_tokens_destroy(void) {
349 }
350
351 bool extract_bearer_token_from_request(struct web_client *w, char *dst, size_t dst_len) {
352 - if(!web_client_flag_check(w, WEB_CLIENT_FLAG_AUTH_BEARER) || dst_len != UUID_STR_LEN)
352 + if(w->user_auth.method != USER_AUTH_METHOD_BEARER || dst_len != UUID_STR_LEN)
353 return false;
354
355 uuid_unparse_lower(w->auth.bearer_token, dst);
src/web/api/http_header.c
+93 -103
@@ -113,10 +113,9 @@ static void http_header_x_forwarded_host(struct web_client *w, const char *v, si
113 }
114
115 static void http_header_x_forwarded_for(struct web_client *w, const char *v, size_t len) {
116 - char buffer[NI_MAXHOST];
117 - strncpyz(buffer, v, (len < sizeof(buffer) - 1 ? len : sizeof(buffer) - 1));
118 - freez(w->forwarded_for);
119 - w->forwarded_for = strdupz(buffer);
116 + if(len)
117 + strncpyz(w->user_auth.forwarded_for, v,
118 + (len < sizeof(w->user_auth.forwarded_for) - 1 ? len : sizeof(w->user_auth.forwarded_for) - 1));
119 }
120
121 static void http_header_x_transaction_id(struct web_client *w, const char *v, size_t len) {
@@ -129,7 +128,7 @@ static void http_header_x_netdata_account_id(struct web_client *w, const char *v
128 if(web_client_flag_check(w, WEB_CLIENT_FLAG_CONN_CLOUD) && w->acl & HTTP_ACL_ACLK) {
129 char buffer[UUID_STR_LEN * 2];
130 strncpyz(buffer, v, (len < sizeof(buffer) - 1 ? len : sizeof(buffer) - 1));
132 - (void) uuid_parse_flexi(buffer, w->auth.cloud_account_id); // will not alter w->cloud_account_id if it fails
131 + (void) uuid_parse_flexi(buffer, w->user_auth.cloud_account_id.uuid); // will not alter w->cloud_account_id if it fails
132 }
133 }
134
@@ -138,32 +137,32 @@ static void http_header_x_netdata_role(struct web_client *w, const char *v, size
137 char buffer[100];
138 strncpyz(buffer, v, (len < sizeof(buffer) - 1 ? len : sizeof(buffer) - 1));
139 if (strcasecmp(buffer, "admin") == 0)
141 - w->user_role = HTTP_USER_ROLE_ADMIN;
140 + w->user_auth.user_role = HTTP_USER_ROLE_ADMIN;
141 else if(strcasecmp(buffer, "manager") == 0)
143 - w->user_role = HTTP_USER_ROLE_MANAGER;
142 + w->user_auth.user_role = HTTP_USER_ROLE_MANAGER;
143 else if(strcasecmp(buffer, "troubleshooter") == 0)
145 - w->user_role = HTTP_USER_ROLE_TROUBLESHOOTER;
144 + w->user_auth.user_role = HTTP_USER_ROLE_TROUBLESHOOTER;
145 else if(strcasecmp(buffer, "observer") == 0)
147 - w->user_role = HTTP_USER_ROLE_OBSERVER;
146 + w->user_auth.user_role = HTTP_USER_ROLE_OBSERVER;
147 else if(strcasecmp(buffer, "member") == 0)
149 - w->user_role = HTTP_USER_ROLE_MEMBER;
148 + w->user_auth.user_role = HTTP_USER_ROLE_MEMBER;
149 else if(strcasecmp(buffer, "billing") == 0)
151 - w->user_role = HTTP_USER_ROLE_BILLING;
150 + w->user_auth.user_role = HTTP_USER_ROLE_BILLING;
151 else
153 - w->user_role = HTTP_USER_ROLE_MEMBER;
152 + w->user_auth.user_role = HTTP_USER_ROLE_MEMBER;
153 }
154 }
155
156 static void http_header_x_netdata_permissions(struct web_client *w, const char *v, size_t len __maybe_unused) {
157 if(web_client_flag_check(w, WEB_CLIENT_FLAG_CONN_CLOUD) && w->acl & HTTP_ACL_ACLK) {
158 HTTP_ACCESS access = http_access_from_hex(v);
160 - web_client_set_permissions(w, access, w->user_role, WEB_CLIENT_FLAG_AUTH_CLOUD);
159 + web_client_set_permissions(w, access, w->user_auth.user_role, USER_AUTH_METHOD_CLOUD);
160 }
161 }
162
163 static void http_header_x_netdata_user_name(struct web_client *w, const char *v, size_t len) {
164 if(web_client_flag_check(w, WEB_CLIENT_FLAG_CONN_CLOUD) && w->acl & HTTP_ACL_ACLK) {
166 - strncpyz(w->auth.client_name, v, (len < sizeof(w->auth.client_name) - 1 ? len : sizeof(w->auth.client_name) - 1));
165 + strncpyz(w->user_auth.client_name, v, (len < sizeof(w->user_auth.client_name) - 1 ? len : sizeof(w->user_auth.client_name) - 1));
166 }
167 }
168
@@ -187,112 +186,103 @@ static void http_header_upgrade(struct web_client *w, const char *v, size_t len
186 }
187
188 static void http_header_sec_websocket_key(struct web_client *w, const char *v, size_t len __maybe_unused) {
190 - if(web_client_is_websocket(w)) {
191 - // Store the websocket key for later use in the handshake
192 - freez(w->websocket.key);
193 - w->websocket.key = strdupz(v);
194 - }
189 + // Store the websocket key for later use in the handshake
190 + freez(w->websocket.key);
191 + w->websocket.key = strdupz(v);
192 }
193
194 static void http_header_sec_websocket_version(struct web_client *w, const char *v, size_t len __maybe_unused) {
198 - if(web_client_is_websocket(w)) {
199 - // We only support version 13, which will be checked during handshake
200 - // No need to store this as we only accept one version
201 - if(strcmp(v, "13") != 0) {
202 - netdata_log_debug(D_WEB_CLIENT, "%llu: WebSocket version %s not supported, only version 13 is supported", w->id, v);
203 - web_client_clear_websocket(w);
204 - }
195 + // We only support version 13, which will be checked during handshake
196 + // No need to store this as we only accept one version
197 + if(strcmp(v, "13") != 0) {
198 + netdata_log_debug(D_WEB_CLIENT, "%llu: WebSocket version %s not supported, only version 13 is supported", w->id, v);
199 + web_client_clear_websocket(w);
200 }
201 }
202
203 static void http_header_sec_websocket_protocol(struct web_client *w, const char *v, size_t len __maybe_unused) {
209 - if(web_client_is_websocket(w)) {
210 - // Store the requested protocols for later evaluation during handshake
211 - w->websocket.protocol = WEBSOCKET_PROTOCOL_2id(v);
212 - }
204 + // Store the requested protocols for later evaluation during handshake
205 + w->websocket.protocol = WEBSOCKET_PROTOCOL_2id(v);
206 }
207
208 static void http_header_sec_websocket_extensions(struct web_client *w, const char *v, size_t len __maybe_unused) {
216 - if(web_client_is_websocket(w)) {
217 - // Reset extension flags
218 - w->websocket.ext_flags = WS_EXTENSION_NONE;
219 -
220 - // Check if "permessage-deflate" is requested
221 - if (strstr(v, "permessage-deflate") != NULL) {
222 - // Parse extension parameters
223 - char extension_copy[1024];
224 - strncpy(extension_copy, v, sizeof(extension_copy) - 1);
225 - extension_copy[sizeof(extension_copy) - 1] = '\0';
226 -
227 - char *token, *saveptr;
228 - token = strtok_r(extension_copy, ",", &saveptr);
229 -
230 - while (token) {
231 - // Trim leading/trailing spaces
232 - char *ext = token;
233 - while (*ext && isspace(*ext)) ext++;
234 - char *end = ext + strlen(ext) - 1;
235 - while (end > ext && isspace(*end)) *end-- = '\0';
236 -
237 - // Check if this is permessage-deflate extension
238 - if (strncmp(ext, "permessage-deflate", 18) == 0) {
239 - w->websocket.ext_flags |= WS_EXTENSION_PERMESSAGE_DEFLATE;
240 -
241 - // Parse parameters
242 - char *params = ext + 18;
243 - if (*params == ';') {
244 - params++;
245 -
246 - char *param, *param_saveptr;
247 - param = strtok_r(params, ";", &param_saveptr);
248 -
249 - while (param) {
250 - // Trim leading/trailing spaces
251 - while (*param && isspace(*param)) param++;
252 - end = param + strlen(param) - 1;
253 - while (end > param && isspace(*end)) *end-- = '\0';
254 -
255 - // Client no context takeover
256 - if (strcmp(param, "client_no_context_takeover") == 0)
257 - w->websocket.ext_flags |= WS_EXTENSION_CLIENT_NO_CONTEXT_TAKEOVER;
258 -
259 - // Server no context takeover
260 - else if (strcmp(param, "server_no_context_takeover") == 0)
261 - w->websocket.ext_flags |= WS_EXTENSION_SERVER_NO_CONTEXT_TAKEOVER;
262 -
263 - // Server max window bits
264 - else if (strncmp(param, "server_max_window_bits=", 23) == 0) {
265 - w->websocket.server_max_window_bits = str2u(param + 23);
266 - if(w->websocket.server_max_window_bits >= 8 && w->websocket.server_max_window_bits <= 15)
267 - w->websocket.ext_flags |= WS_EXTENSION_SERVER_MAX_WINDOW_BITS;
268 - }
269 - // Server max window bits without value
270 - else if (strcmp(param, "server_max_window_bits") == 0) {
209 + // Reset extension flags
210 + w->websocket.ext_flags = WS_EXTENSION_NONE;
211 +
212 + // Check if "permessage-deflate" is requested
213 + if (strstr(v, "permessage-deflate") != NULL) {
214 + // Parse extension parameters
215 + char extension_copy[1024];
216 + strncpyz(extension_copy, v, sizeof(extension_copy) - 1);
217 +
218 + char *token, *saveptr;
219 + token = strtok_r(extension_copy, ",", &saveptr);
220 +
221 + while (token) {
222 + // Trim leading/trailing spaces
223 + char *ext = token;
224 + while (*ext && isspace(*ext)) ext++;
225 + char *end = ext + strlen(ext) - 1;
226 + while (end > ext && isspace(*end)) *end-- = '\0';
227 +
228 + // Check if this is permessage-deflate extension
229 + if (strncmp(ext, "permessage-deflate", 18) == 0) {
230 + w->websocket.ext_flags |= WS_EXTENSION_PERMESSAGE_DEFLATE;
231 +
232 + // Parse parameters
233 + char *params = ext + 18;
234 + if (*params == ';') {
235 + params++;
236 +
237 + char *param, *param_saveptr;
238 + param = strtok_r(params, ";", &param_saveptr);
239 +
240 + while (param) {
241 + // Trim leading/trailing spaces
242 + while (*param && isspace(*param)) param++;
243 + end = param + strlen(param) - 1;
244 + while (end > param && isspace(*end)) *end-- = '\0';
245 +
246 + // Client no context takeover
247 + if (strcmp(param, "client_no_context_takeover") == 0)
248 + w->websocket.ext_flags |= WS_EXTENSION_CLIENT_NO_CONTEXT_TAKEOVER;
249 +
250 + // Server no context takeover
251 + else if (strcmp(param, "server_no_context_takeover") == 0)
252 + w->websocket.ext_flags |= WS_EXTENSION_SERVER_NO_CONTEXT_TAKEOVER;
253 +
254 + // Server max window bits
255 + else if (strncmp(param, "server_max_window_bits=", 23) == 0) {
256 + w->websocket.server_max_window_bits = str2u(param + 23);
257 + if(w->websocket.server_max_window_bits >= 8 && w->websocket.server_max_window_bits <= 15)
258 w->websocket.ext_flags |= WS_EXTENSION_SERVER_MAX_WINDOW_BITS;
272 - w->websocket.server_max_window_bits = 0; // Default
273 - }
274 -
275 - // Client max window bits with value
276 - else if (strncmp(param, "client_max_window_bits=", 23) == 0) {
277 - w->websocket.client_max_window_bits = str2u(param + 23);
278 - if(w->websocket.client_max_window_bits >= 8 && w->websocket.client_max_window_bits <= 15)
279 - w->websocket.ext_flags |= WS_EXTENSION_CLIENT_MAX_WINDOW_BITS;
280 - }
281 - // Client max window bits without value
282 - else if (strcmp(param, "client_max_window_bits") == 0) {
283 - w->websocket.ext_flags |= WS_EXTENSION_CLIENT_MAX_WINDOW_BITS;
284 - w->websocket.client_max_window_bits = 0; // Default
285 - }
259 + }
260 + // Server max window bits without value
261 + else if (strcmp(param, "server_max_window_bits") == 0) {
262 + w->websocket.ext_flags |= WS_EXTENSION_SERVER_MAX_WINDOW_BITS;
263 + w->websocket.server_max_window_bits = 0; // Default
264 + }
265
287 - param = strtok_r(NULL, ";", &param_saveptr);
266 + // Client max window bits with value
267 + else if (strncmp(param, "client_max_window_bits=", 23) == 0) {
268 + w->websocket.client_max_window_bits = str2u(param + 23);
269 + if(w->websocket.client_max_window_bits >= 8 && w->websocket.client_max_window_bits <= 15)
270 + w->websocket.ext_flags |= WS_EXTENSION_CLIENT_MAX_WINDOW_BITS;
271 + }
272 + // Client max window bits without value
273 + else if (strcmp(param, "client_max_window_bits") == 0) {
274 + w->websocket.ext_flags |= WS_EXTENSION_CLIENT_MAX_WINDOW_BITS;
275 + w->websocket.client_max_window_bits = 0; // Default
276 }
289 - }
277
291 - break; // Found and parsed permessage-deflate
278 + param = strtok_r(NULL, ";", &param_saveptr);
279 + }
280 }
281
294 - token = strtok_r(NULL, ",", &saveptr);
282 + break; // Found and parsed permessage-deflate
283 }
284 +
285 + token = strtok_r(NULL, ",", &saveptr);
286 }
287
288 netdata_log_debug(D_WEB_CLIENT, "%llu: Client requested WebSocket extensions: %s, "
src/web/api/maps/contexts_options.c
+36 -4
@@ -9,10 +9,22 @@ static struct {
9 } contexts_options[] = {
10 {"minify" , 0 , CONTEXTS_OPTION_MINIFY}
11 , {"debug" , 0 , CONTEXTS_OPTION_DEBUG}
12 - , {"config" , 0 , CONTEXTS_OPTION_ALERTS_WITH_CONFIGURATIONS}
13 - , {"instances" , 0 , CONTEXTS_OPTION_ALERTS_WITH_INSTANCES}
14 - , {"values" , 0 , CONTEXTS_OPTION_ALERTS_WITH_VALUES}
15 - , {"summary" , 0 , CONTEXTS_OPTION_ALERTS_WITH_SUMMARY}
12 + , {"config" , 0 , CONTEXTS_OPTION_CONFIGURATIONS}
13 + , {"instances" , 0 , CONTEXTS_OPTION_INSTANCES}
14 + , {"values" , 0 , CONTEXTS_OPTION_VALUES}
15 + , {"summary" , 0 , CONTEXTS_OPTION_SUMMARY}
16 + , {"mcp" , 0 , CONTEXTS_OPTION_MCP}
17 + , {"dimensions" , 0 , CONTEXTS_OPTION_DIMENSIONS}
18 + , {"labels" , 0 , CONTEXTS_OPTION_LABELS}
19 + , {"priorities" , 0 , CONTEXTS_OPTION_PRIORITIES}
20 + , {"titles" , 0 , CONTEXTS_OPTION_TITLES}
21 + , {"retention" , 0 , CONTEXTS_OPTION_RETENTION}
22 + , {"liveness" , 0 , CONTEXTS_OPTION_LIVENESS}
23 + , {"family" , 0 , CONTEXTS_OPTION_FAMILY}
24 + , {"units" , 0 , CONTEXTS_OPTION_UNITS}
25 + , {"rfc3339" , 0 , CONTEXTS_OPTION_RFC3339}
26 + , {"long-json-keys" , 0 , CONTEXTS_OPTION_JSON_LONG_KEYS}
27 + , {"long-keys" , 0 , CONTEXTS_OPTION_JSON_LONG_KEYS}
28 , {NULL , 0 , 0}
29 };
30
@@ -56,3 +68,23 @@ void contexts_options_init(void) {
68 for(size_t i = 0; contexts_options[i].name ; i++)
69 contexts_options[i].hash = simple_hash(contexts_options[i].name);
70 }
71 +
72 +// Map RRDR_OPTIONS to CONTEXTS_OPTIONS for options that are common between both
73 +ALWAYS_INLINE
74 +CONTEXTS_OPTIONS rrdr_options_to_contexts_options(RRDR_OPTIONS rrdr_options) {
75 + CONTEXTS_OPTIONS contexts_options = 0;
76 +
77 + if(rrdr_options & RRDR_OPTION_MINIFY)
78 + contexts_options |= CONTEXTS_OPTION_MINIFY;
79 +
80 + if(rrdr_options & RRDR_OPTION_DEBUG)
81 + contexts_options |= CONTEXTS_OPTION_DEBUG;
82 +
83 + if(rrdr_options & RRDR_OPTION_RFC3339)
84 + contexts_options |= CONTEXTS_OPTION_RFC3339;
85 +
86 + if(rrdr_options & RRDR_OPTION_LONG_JSON_KEYS)
87 + contexts_options |= CONTEXTS_OPTION_JSON_LONG_KEYS;
88 +
89 + return contexts_options;
90 +}
src/web/api/maps/contexts_options.h
+19 -4
@@ -4,14 +4,26 @@
4 #define NETDATA_CONTEXTS_OPTIONS_H
5
6 #include "libnetdata/libnetdata.h"
7 +#include "rrdr_options.h"
8
9 typedef enum contexts_options {
10 CONTEXTS_OPTION_MINIFY = (1 << 0), // remove JSON spaces and newlines from JSON output
11 CONTEXTS_OPTION_DEBUG = (1 << 1), // show the request
11 - CONTEXTS_OPTION_ALERTS_WITH_CONFIGURATIONS = (1 << 2), // include alert configurations (used by /api/v2/alert_transitions)
12 - CONTEXTS_OPTION_ALERTS_WITH_INSTANCES = (1 << 3), // include alert instances (used by /api/v2/alerts)
13 - CONTEXTS_OPTION_ALERTS_WITH_VALUES = (1 << 4), // include alert latest values (used by /api/v2/alerts)
14 - CONTEXTS_OPTION_ALERTS_WITH_SUMMARY = (1 << 5), // include alerts summary counters (used by /api/v2/alerts)
12 + CONTEXTS_OPTION_CONFIGURATIONS = (1 << 2), // include alert configurations (used by /api/v2/alert_transitions)
13 + CONTEXTS_OPTION_INSTANCES = (1 << 3), // include alert/context instances (used by /api/v2/alerts)
14 + CONTEXTS_OPTION_VALUES = (1 << 4), // include alert latest values (used by /api/v2/alerts)
15 + CONTEXTS_OPTION_SUMMARY = (1 << 5), // include alerts summary counters (used by /api/v2/alerts)
16 + CONTEXTS_OPTION_MCP = (1 << 6), // MCP output format
17 + CONTEXTS_OPTION_DIMENSIONS = (1 << 7), // include context dimensions
18 + CONTEXTS_OPTION_LABELS = (1 << 8), // include context labels
19 + CONTEXTS_OPTION_PRIORITIES = (1 << 9), // include context priorities
20 + CONTEXTS_OPTION_TITLES = (1 << 10), // include context titles
21 + CONTEXTS_OPTION_RETENTION = (1 << 11), // include first_entry and last_entry
22 + CONTEXTS_OPTION_LIVENESS = (1 << 12), // include live status
23 + CONTEXTS_OPTION_FAMILY = (1 << 13), // include family
24 + CONTEXTS_OPTION_UNITS = (1 << 14), // include units
25 + CONTEXTS_OPTION_RFC3339 = (1 << 15), // Return timestamps in RFC3339 format
26 + CONTEXTS_OPTION_JSON_LONG_KEYS = (1 << 16), // Use long JSON keys instead of short ones
27 } CONTEXTS_OPTIONS;
28
29 CONTEXTS_OPTIONS contexts_options_str_to_id(char *o);
@@ -19,4 +31,7 @@ void contexts_options_to_buffer_json_array(BUFFER *wb, const char *key, CONTEXTS
31
32 void contexts_options_init(void);
33
34 +// Map RRDR_OPTIONS to CONTEXTS_OPTIONS for options that are common between both
35 +CONTEXTS_OPTIONS rrdr_options_to_contexts_options(RRDR_OPTIONS rrdr_options);
36 +
37 #endif //NETDATA_CONTEXTS_OPTIONS_H
src/web/api/maps/datasource_formats.c
+2 -2
@@ -18,7 +18,7 @@ static struct {
18 , {NULL, 0, 0}
19 };
20
21 -inline DATASOURCE_FORMAT google_data_format_str_to_id(char *name) {
21 +inline DATASOURCE_FORMAT google_data_format_str_to_id(const char *name) {
22 uint32_t hash = simple_hash(name);
23 int i;
24
@@ -57,7 +57,7 @@ static struct {
57 , {NULL, 0, 0}
58 };
59
60 -DATASOURCE_FORMAT datasource_format_str_to_id(char *name) {
60 +DATASOURCE_FORMAT datasource_format_str_to_id(const char *name) {
61 uint32_t hash = simple_hash(name);
62 int i;
63
src/web/api/maps/datasource_formats.h
+2 -2
@@ -22,10 +22,10 @@ typedef enum {
22 DATASOURCE_JSON2,
23 } DATASOURCE_FORMAT;
24
25 -DATASOURCE_FORMAT datasource_format_str_to_id(char *name);
25 +DATASOURCE_FORMAT datasource_format_str_to_id(const char *name);
26 const char *rrdr_format_to_string(DATASOURCE_FORMAT format);
27
28 -DATASOURCE_FORMAT google_data_format_str_to_id(char *name);
28 +DATASOURCE_FORMAT google_data_format_str_to_id(const char *name);
29
30 void datasource_formats_init(void);
31
src/web/api/maps/rrdr_options.c
+11 -1
@@ -48,6 +48,12 @@ static struct {
48 , {"minify" , 0 , RRDR_OPTION_MINIFY}
49 , {"group-by-labels" , 0 , RRDR_OPTION_GROUP_BY_LABELS}
50 , {"label-quotes" , 0 , RRDR_OPTION_LABEL_QUOTES}
51 + , {"minimal-stats" , 0 , RRDR_OPTION_MINIMAL_STATS}
52 + , {"minimal" , 0 , RRDR_OPTION_MINIMAL_STATS}
53 + , {"long-json-keys" , 0 , RRDR_OPTION_LONG_JSON_KEYS}
54 + , {"long-keys" , 0 , RRDR_OPTION_LONG_JSON_KEYS}
55 + , {"mcp-info" , 0 , RRDR_OPTION_MCP_INFO}
56 + , {"rfc3339" , 0 , RRDR_OPTION_RFC3339}
57 , {NULL , 0 , 0}
58 };
59
@@ -68,7 +74,11 @@ RRDR_OPTIONS rrdr_options_parse_one(const char *o) {
74 return ret;
75 }
76
71 -RRDR_OPTIONS rrdr_options_parse(char *o) {
77 +RRDR_OPTIONS rrdr_options_parse(const char *options_str) {
78 + char src[strlen(options_str) + 1];
79 + strcatz(src, 0, options_str, sizeof(src));
80 + char *o = src;
81 +
82 RRDR_OPTIONS ret = 0;
83 char *tok;
84
src/web/api/maps/rrdr_options.h
+38 -32
@@ -6,47 +6,53 @@
6 #include "libnetdata/libnetdata.h"
7
8 typedef enum rrdr_options {
9 - RRDR_OPTION_NONZERO = (1 << 0), // don't output dimensions with just zero values
10 - RRDR_OPTION_REVERSED = (1 << 1), // output the rows in reverse order (oldest to newest)
11 - RRDR_OPTION_ABSOLUTE = (1 << 2), // values positive, for DATASOURCE_SSV before summing
12 - RRDR_OPTION_DIMS_MIN2MAX = (1 << 3), // when adding dimensions, use max - min, instead of sum
13 - RRDR_OPTION_DIMS_AVERAGE = (1 << 4), // when adding dimensions, use average, instead of sum
14 - RRDR_OPTION_DIMS_MIN = (1 << 5), // when adding dimensions, use minimum, instead of sum
15 - RRDR_OPTION_DIMS_MAX = (1 << 6), // when adding dimensions, use maximum, instead of sum
16 - RRDR_OPTION_SECONDS = (1 << 7), // output seconds, instead of dates
17 - RRDR_OPTION_MILLISECONDS = (1 << 8), // output milliseconds, instead of dates
18 - RRDR_OPTION_NULL2ZERO = (1 << 9), // do not show nulls, convert them to zeros
19 - RRDR_OPTION_OBJECTSROWS = (1 << 10), // each row of values should be an object, not an array
20 - RRDR_OPTION_GOOGLE_JSON = (1 << 11), // comply with google JSON/JSONP specs
21 - RRDR_OPTION_JSON_WRAP = (1 << 12), // wrap the response in a JSON header with info about the result
22 - RRDR_OPTION_LABEL_QUOTES = (1 << 13), // in CSV output, wrap header labels in double quotes
23 - RRDR_OPTION_PERCENTAGE = (1 << 14), // give values as percentage of total
24 - RRDR_OPTION_NOT_ALIGNED = (1 << 15), // do not align charts for persistent timeframes
25 - RRDR_OPTION_DISPLAY_ABS = (1 << 16), // for badges, display the absolute value, but calculate colors with sign
26 - RRDR_OPTION_MATCH_IDS = (1 << 17), // when filtering dimensions, match only IDs
27 - RRDR_OPTION_MATCH_NAMES = (1 << 18), // when filtering dimensions, match only names
28 - RRDR_OPTION_NATURAL_POINTS = (1 << 19), // return the natural points of the database
29 - RRDR_OPTION_VIRTUAL_POINTS = (1 << 20), // return virtual points
30 - RRDR_OPTION_ANOMALY_BIT = (1 << 21), // Return the anomaly bit stored in each collected_number
31 - RRDR_OPTION_RETURN_RAW = (1 << 22), // Return raw data for aggregating across multiple nodes
32 - RRDR_OPTION_RETURN_JWAR = (1 << 23), // Return anomaly rates in jsonwrap
33 - RRDR_OPTION_SELECTED_TIER = (1 << 24), // Use the selected tier for the query
34 - RRDR_OPTION_ALL_DIMENSIONS = (1 << 25), // Return the full dimensions list
35 - RRDR_OPTION_SHOW_DETAILS = (1 << 26), // v2 returns detailed object tree
36 - RRDR_OPTION_DEBUG = (1 << 27), // v2 returns request description
37 - RRDR_OPTION_MINIFY = (1 << 28), // remove JSON spaces and newlines from JSON output
38 - RRDR_OPTION_GROUP_BY_LABELS = (1 << 29), // v2 returns flattened labels per dimension of the chart
9 + RRDR_OPTION_NONZERO = (1ULL << 0), // don't output dimensions with just zero values
10 + RRDR_OPTION_REVERSED = (1ULL << 1), // output the rows in reverse order (oldest to newest)
11 + RRDR_OPTION_ABSOLUTE = (1ULL << 2), // values positive, for DATASOURCE_SSV before summing
12 + RRDR_OPTION_DIMS_MIN2MAX = (1ULL << 3), // when adding dimensions, use max - min, instead of sum
13 + RRDR_OPTION_DIMS_AVERAGE = (1ULL << 4), // when adding dimensions, use average, instead of sum
14 + RRDR_OPTION_DIMS_MIN = (1ULL << 5), // when adding dimensions, use minimum, instead of sum
15 + RRDR_OPTION_DIMS_MAX = (1ULL << 6), // when adding dimensions, use maximum, instead of sum
16 + RRDR_OPTION_SECONDS = (1ULL << 7), // output seconds, instead of dates
17 + RRDR_OPTION_MILLISECONDS = (1ULL << 8), // output milliseconds, instead of dates
18 + RRDR_OPTION_NULL2ZERO = (1ULL << 9), // do not show nulls, convert them to zeros
19 + RRDR_OPTION_OBJECTSROWS = (1ULL << 10), // each row of values should be an object, not an array
20 + RRDR_OPTION_GOOGLE_JSON = (1ULL << 11), // comply with google JSON/JSONP specs
21 + RRDR_OPTION_JSON_WRAP = (1ULL << 12), // wrap the response in a JSON header with info about the result
22 + RRDR_OPTION_LABEL_QUOTES = (1ULL << 13), // in CSV output, wrap header labels in double quotes
23 + RRDR_OPTION_PERCENTAGE = (1ULL << 14), // give values as percentage of total
24 + RRDR_OPTION_NOT_ALIGNED = (1ULL << 15), // do not align charts for persistent timeframes
25 + RRDR_OPTION_DISPLAY_ABS = (1ULL << 16), // for badges, display the absolute value, but calculate colors with sign
26 + RRDR_OPTION_MATCH_IDS = (1ULL << 17), // when filtering dimensions, match only IDs
27 + RRDR_OPTION_MATCH_NAMES = (1ULL << 18), // when filtering dimensions, match only names
28 + RRDR_OPTION_NATURAL_POINTS = (1ULL << 19), // return the natural points of the database
29 + RRDR_OPTION_VIRTUAL_POINTS = (1ULL << 20), // return virtual points
30 + RRDR_OPTION_ANOMALY_BIT = (1ULL << 21), // Return the anomaly bit stored in each collected_number
31 + RRDR_OPTION_RETURN_RAW = (1ULL << 22), // Return raw data for aggregating across multiple nodes
32 + RRDR_OPTION_RETURN_JWAR = (1ULL << 23), // Return anomaly rates in jsonwrap
33 + RRDR_OPTION_SELECTED_TIER = (1ULL << 24), // Use the selected tier for the query
34 + RRDR_OPTION_ALL_DIMENSIONS = (1ULL << 25), // Return the full dimensions list
35 + RRDR_OPTION_SHOW_DETAILS = (1ULL << 26), // v2 returns detailed object tree
36 + RRDR_OPTION_DEBUG = (1ULL << 27), // v2 returns request description
37 + RRDR_OPTION_MINIFY = (1ULL << 28), // remove JSON spaces and newlines from JSON output
38 + RRDR_OPTION_GROUP_BY_LABELS = (1ULL << 29), // v2 returns flattened labels per dimension of the chart
39 + RRDR_OPTION_MINIMAL_STATS = (1ULL << 30), // Remove "totals" and statistics fields (qr, sl, ex) from response
40 + RRDR_OPTION_LONG_JSON_KEYS = (1ULL << 31), // Use descriptive long JSON keys instead of cryptic short ones
41 + RRDR_OPTION_MCP_INFO = (1ULL << 32), // Include "info" nodes in JSON response sections
42 + RRDR_OPTION_RFC3339 = (1ULL << 33), // Return timestamps in RFC3339 format instead of epoch seconds
43
44 // internal ones - not to be exposed to the API
41 - RRDR_OPTION_INTERNAL_AR = (1 << 31), // internal use only, to let the formatters know we want to render the anomaly rate
45 + RRDR_OPTION_INTERNAL_AR = (1ULL << 63), // internal use only, to let the formatters know we want to render the anomaly rate
46 } RRDR_OPTIONS;
47
48 +static_assert(sizeof(RRDR_OPTIONS) == 8, "RRDR_OPTIONS must be 8 bytes");
49 +
50 void rrdr_options_to_buffer(BUFFER *wb, RRDR_OPTIONS options);
51 void rrdr_options_to_buffer_json_array(BUFFER *wb, const char *key, RRDR_OPTIONS options);
52 void web_client_api_request_data_vX_options_to_string(char *buf, size_t size, RRDR_OPTIONS options);
53 void rrdr_options_init(void);
54
49 -RRDR_OPTIONS rrdr_options_parse(char *o);
55 +RRDR_OPTIONS rrdr_options_parse(const char *options_str);
56 RRDR_OPTIONS rrdr_options_parse_one(const char *o);
57
58 #endif //NETDATA_RRDR_OPTIONS_H
src/web/api/netdata-swagger.json
+576 -23
@@ -2,7 +2,7 @@
2 "openapi": "3.0.0",
3 "info": {
4 "title": "Netdata API",
5 - "description": "Real-time performance and health monitoring.",
5 + "description": "Real-time performance and health monitoring.\n\n## API Versions\n\nNetdata provides three API versions:\n- **v1**: The original API, focused on single-node operations\n- **v2**: Multi-node API with advanced grouping and aggregation capabilities\n- **v3**: The latest API version that combines v1 and v2 endpoints and may include additional features\n\n### v3 API Endpoints\n\nThe v3 API provides the following endpoints (most reuse v1 or v2 implementations):\n- `/api/v3/data` - Multi-dimensional data queries (uses v2 implementation)\n- `/api/v3/weights` - Metric scoring/correlation (uses v2 implementation)\n- `/api/v3/contexts` - Context metadata (uses v2 implementation)\n- `/api/v3/nodes` - Node information (uses v2 implementation)\n- `/api/v3/q` - Full-text search (uses v2 implementation)\n- `/api/v3/alerts` - Alert information (uses v2 implementation)\n- `/api/v3/alert_transitions` - Alert state transitions (uses v2 implementation)\n- `/api/v3/alert_config` - Alert configuration (uses v2 implementation)\n- `/api/v3/functions` - Available functions (uses v2 implementation)\n- `/api/v3/function` - Execute functions (uses v1 implementation)\n- `/api/v3/info` - Agent information (uses v2 implementation)\n- `/api/v3/node_instances` - Node instance information (uses v2 implementation)\n- `/api/v3/stream_path` - Streaming topology (v3 specific)\n- `/api/v3/versions` - Version information (uses v2 implementation)\n- `/api/v3/badge.svg` - Dynamic badges (uses v1 implementation)\n- `/api/v3/allmetrics` - Export metrics (uses v1 implementation)\n- `/api/v3/context` - Single context info (uses v1 implementation)\n- `/api/v3/variable` - Variable information (uses v1 implementation)\n- `/api/v3/config` - Dynamic configuration (uses v1 implementation)\n- `/api/v3/settings` - Agent settings (v3 specific)\n- `/api/v3/me` - Current user information (v3 specific)\n- `/api/v3/claim` - Agent claiming (v3 specific)\n- Additional management and streaming endpoints\n\nNote: This documentation currently focuses on v1 and v2 endpoints. The v3 endpoints listed above generally accept the same parameters as their v1/v2 counterparts.\n",
6 "version": "v1-rolling",
7 "contact": {
8 "name": "Netdata Agent API",
@@ -84,6 +84,9 @@
84 },
85 {
86 "$ref": "#/components/parameters/filterContexts"
87 + },
88 + {
89 + "$ref": "#/components/parameters/contextsQueryOptions"
90 }
91 ],
92 "responses": {
@@ -118,6 +121,63 @@
121 }
122 }
123 },
124 + "/api/v3/nodes": {
125 + "get": {
126 + "operationId": "getNodes3",
127 + "tags": [
128 + "nodes"
129 + ],
130 + "summary": "Nodes Info v3",
131 + "description": "Get a list of all nodes hosted by this Netdata Agent.\nThis is the same as `/api/v2/nodes` but may support additional features in the future.\n",
132 + "parameters": [
133 + {
134 + "$ref": "#/components/parameters/scopeNodes"
135 + },
136 + {
137 + "$ref": "#/components/parameters/scopeContexts"
138 + },
139 + {
140 + "$ref": "#/components/parameters/filterNodes"
141 + },
142 + {
143 + "$ref": "#/components/parameters/filterContexts"
144 + },
145 + {
146 + "$ref": "#/components/parameters/contextsQueryOptions"
147 + }
148 + ],
149 + "responses": {
150 + "200": {
151 + "description": "OK",
152 + "content": {
153 + "application/json": {
154 + "schema": {
155 + "description": "`/api/v3/nodes` response for all nodes hosted by a Netdata Agent.\n",
156 + "type": "object",
157 + "properties": {
158 + "api": {
159 + "$ref": "#/components/schemas/api"
160 + },
161 + "agents": {
162 + "$ref": "#/components/schemas/agents"
163 + },
164 + "versions": {
165 + "$ref": "#/components/schemas/versions"
166 + },
167 + "nodes": {
168 + "type": "array",
169 + "items": {
170 + "$ref": "#/components/schemas/nodeFull"
171 + }
172 + }
173 + }
174 + }
175 + }
176 + }
177 + }
178 + }
179 + }
180 + },
181 "/api/v2/contexts": {
182 "get": {
183 "operationId": "getContexts2",
@@ -138,6 +198,48 @@
198 },
199 {
200 "$ref": "#/components/parameters/filterContexts"
201 + },
202 + {
203 + "$ref": "#/components/parameters/contextsQueryOptions"
204 + }
205 + ],
206 + "responses": {
207 + "200": {
208 + "description": "OK",
209 + "content": {
210 + "application/json": {
211 + "schema": {
212 + "$ref": "#/components/schemas/contexts2"
213 + }
214 + }
215 + }
216 + }
217 + }
218 + }
219 + },
220 + "/api/v3/contexts": {
221 + "get": {
222 + "operationId": "getContexts3",
223 + "tags": [
224 + "contexts"
225 + ],
226 + "summary": "Contexts Info v3",
227 + "description": "Get a list of all contexts, across all nodes, hosted by this Netdata Agent.\nThis is the same as `/api/v2/contexts` but may support additional features in the future.\n",
228 + "parameters": [
229 + {
230 + "$ref": "#/components/parameters/scopeNodes"
231 + },
232 + {
233 + "$ref": "#/components/parameters/scopeContexts"
234 + },
235 + {
236 + "$ref": "#/components/parameters/filterNodes"
237 + },
238 + {
239 + "$ref": "#/components/parameters/filterContexts"
240 + },
241 + {
242 + "$ref": "#/components/parameters/contextsQueryOptions"
243 }
244 ],
245 "responses": {
@@ -184,6 +286,58 @@
286 },
287 {
288 "$ref": "#/components/parameters/filterContexts"
289 + },
290 + {
291 + "$ref": "#/components/parameters/contextsQueryOptions"
292 + }
293 + ],
294 + "responses": {
295 + "200": {
296 + "description": "OK",
297 + "content": {
298 + "application/json": {
299 + "schema": {
300 + "$ref": "#/components/schemas/contexts2"
301 + }
302 + }
303 + }
304 + }
305 + }
306 + }
307 + },
308 + "/api/v3/q": {
309 + "get": {
310 + "operationId": "q3",
311 + "tags": [
312 + "contexts"
313 + ],
314 + "summary": "Full Text Search v3",
315 + "description": "Get a list of contexts, across all nodes, hosted by this Netdata Agent, matching a string expression.\nThis is the same as `/api/v2/q` but may support additional features in the future.\n",
316 + "parameters": [
317 + {
318 + "name": "q",
319 + "in": "query",
320 + "description": "The strings to search for, formatted as a simple pattern",
321 + "required": true,
322 + "schema": {
323 + "type": "string",
324 + "format": "simple pattern"
325 + }
326 + },
327 + {
328 + "$ref": "#/components/parameters/scopeNodes"
329 + },
330 + {
331 + "$ref": "#/components/parameters/scopeContexts"
332 + },
333 + {
334 + "$ref": "#/components/parameters/filterNodes"
335 + },
336 + {
337 + "$ref": "#/components/parameters/filterContexts"
338 + },
339 + {
340 + "$ref": "#/components/parameters/contextsQueryOptions"
341 }
342 ],
343 "responses": {
@@ -626,7 +780,8 @@
780 "avg",
781 "average",
782 "sum",
629 - "percentage"
783 + "percentage",
784 + "extremes"
785 ],
786 "default": "average"
787 }
@@ -637,6 +792,15 @@
792 {
793 "$ref": "#/components/parameters/scopeContexts"
794 },
795 + {
796 + "$ref": "#/components/parameters/scopeInstances"
797 + },
798 + {
799 + "$ref": "#/components/parameters/scopeLabels"
800 + },
801 + {
802 + "$ref": "#/components/parameters/scopeDimensions"
803 + },
804 {
805 "$ref": "#/components/parameters/filterNodes"
806 },
@@ -682,6 +846,194 @@
846 {
847 "$ref": "#/components/parameters/dataFormat2"
848 },
849 + {
850 + "$ref": "#/components/parameters/cardinalityLimit"
851 + },
852 + {
853 + "$ref": "#/components/parameters/timeoutMS"
854 + },
855 + {
856 + "$ref": "#/components/parameters/callback"
857 + },
858 + {
859 + "$ref": "#/components/parameters/filename"
860 + },
861 + {
862 + "$ref": "#/components/parameters/tqx"
863 + }
864 + ],
865 + "responses": {
866 + "200": {
867 + "description": "The call was successful. The response includes the data in the format requested.\n",
868 + "content": {
869 + "application/json": {
870 + "schema": {
871 + "oneOf": [
872 + {
873 + "$ref": "#/components/schemas/jsonwrap2"
874 + },
875 + {
876 + "$ref": "#/components/schemas/data_json_formats2"
877 + }
878 + ]
879 + }
880 + },
881 + "text/plain": {
882 + "schema": {
883 + "type": "string",
884 + "format": "according to the format requested."
885 + }
886 + },
887 + "text/html": {
888 + "schema": {
889 + "type": "string",
890 + "format": "html"
891 + }
892 + },
893 + "application/x-javascript": {
894 + "schema": {
895 + "type": "string",
896 + "format": "javascript"
897 + }
898 + }
899 + }
900 + },
901 + "400": {
902 + "description": "Bad request - the body will include a message stating what is wrong.\n"
903 + },
904 + "500": {
905 + "description": "Internal server error. This usually means the server is out of memory.\n"
906 + }
907 + }
908 + }
909 + },
910 + "/api/v3/data": {
911 + "get": {
912 + "operationId": "dataQuery3",
913 + "tags": [
914 + "data"
915 + ],
916 + "summary": "Data Query v3",
917 + "description": "Multi-node, multi-context, multi-instance, multi-dimension data queries, with time and metric aggregation.\nThis is the same as `/api/v2/data` but may support additional features in the future.\n",
918 + "parameters": [
919 + {
920 + "name": "group_by",
921 + "in": "query",
922 + "description": "A comma separated list of the groupings required.\nAll possible values can be combined together, except `selected`. If `selected` is given in the list, all others are ignored.\nThe order they are placed in the list is currently ignored.\nThis parameter is also accepted as `group_by[0]` and `group_by[1]` when multiple grouping passes are required.\n",
923 + "required": false,
924 + "schema": {
925 + "type": "array",
926 + "items": {
927 + "type": "string",
928 + "enum": [
929 + "dimension",
930 + "instance",
931 + "percentage-of-instance",
932 + "label",
933 + "node",
934 + "context",
935 + "units",
936 + "selected"
937 + ]
938 + },
939 + "default": [
940 + "dimension"
941 + ]
942 + }
943 + },
944 + {
945 + "name": "group_by_label",
946 + "in": "query",
947 + "description": "A comma separated list of the label keys to group by their values. The order of the labels in the list is respected.\nThis parameter is also accepted as `group_by_label[0]` and `group_by_label[1]` when multiple grouping passes are required.\n",
948 + "required": false,
949 + "schema": {
950 + "type": "string",
951 + "format": "comma separated list of label keys to group by",
952 + "default": ""
953 + }
954 + },
955 + {
956 + "name": "aggregation",
957 + "in": "query",
958 + "description": "The aggregation function to apply when grouping metrics together.\nWhen option `raw` is given, `average` and `avg` behave like `sum` and the caller is expected to calculate the average.\nThis parameter is also accepted as `aggregation[0]` and `aggregation[1]` when multiple grouping passes are required.\n",
959 + "required": false,
960 + "schema": {
961 + "type": "string",
962 + "enum": [
963 + "min",
964 + "max",
965 + "avg",
966 + "average",
967 + "sum",
968 + "percentage",
969 + "extremes"
970 + ],
971 + "default": "average"
972 + }
973 + },
974 + {
975 + "$ref": "#/components/parameters/scopeNodes"
976 + },
977 + {
978 + "$ref": "#/components/parameters/scopeContexts"
979 + },
980 + {
981 + "$ref": "#/components/parameters/scopeInstances"
982 + },
983 + {
984 + "$ref": "#/components/parameters/scopeLabels"
985 + },
986 + {
987 + "$ref": "#/components/parameters/scopeDimensions"
988 + },
989 + {
990 + "$ref": "#/components/parameters/filterNodes"
991 + },
992 + {
993 + "$ref": "#/components/parameters/filterContexts"
994 + },
995 + {
996 + "$ref": "#/components/parameters/filterInstances"
997 + },
998 + {
999 + "$ref": "#/components/parameters/filterLabels"
1000 + },
1001 + {
1002 + "$ref": "#/components/parameters/filterAlerts"
1003 + },
1004 + {
1005 + "$ref": "#/components/parameters/filterDimensions"
1006 + },
1007 + {
1008 + "$ref": "#/components/parameters/after"
1009 + },
1010 + {
1011 + "$ref": "#/components/parameters/before"
1012 + },
1013 + {
1014 + "$ref": "#/components/parameters/points"
1015 + },
1016 + {
1017 + "$ref": "#/components/parameters/tier"
1018 + },
1019 + {
1020 + "$ref": "#/components/parameters/dataQueryOptions"
1021 + },
1022 + {
1023 + "$ref": "#/components/parameters/dataTimeGroup2"
1024 + },
1025 + {
1026 + "$ref": "#/components/parameters/dataTimeGroupOptions2"
1027 + },
1028 + {
1029 + "$ref": "#/components/parameters/dataTimeResampling2"
1030 + },
1031 + {
1032 + "$ref": "#/components/parameters/dataFormat2"
1033 + },
1034 + {
1035 + "$ref": "#/components/parameters/cardinalityLimit"
1036 + },
1037 {
1038 "$ref": "#/components/parameters/timeoutMS"
1039 },
@@ -895,10 +1247,10 @@
1247 "schema": {
1248 "type": "string",
1249 "enum": [
898 - "yes",
899 - "no"
1250 + true,
1251 + false
1252 ],
901 - "default": "no"
1253 + "default": false
1254 }
1255 },
1256 {
@@ -909,10 +1261,10 @@
1261 "schema": {
1262 "type": "string",
1263 "enum": [
912 - "yes",
913 - "no"
1264 + true,
1265 + false
1266 ],
915 - "default": "yes"
1267 + "default": true
1268 }
1269 },
1270 {
@@ -923,10 +1275,10 @@
1275 "schema": {
1276 "type": "string",
1277 "enum": [
926 - "yes",
927 - "no"
1278 + true,
1279 + false
1280 ],
929 - "default": "yes"
1281 + "default": true
1282 }
1283 },
1284 {
@@ -937,10 +1289,10 @@
1289 "schema": {
1290 "type": "string",
1291 "enum": [
940 - "yes",
941 - "no"
1292 + true,
1293 + false
1294 ],
943 - "default": "yes"
1295 + "default": true
1296 }
1297 },
1298 {
@@ -951,10 +1303,10 @@
1303 "schema": {
1304 "type": "string",
1305 "enum": [
954 - "yes",
955 - "no"
1306 + true,
1307 + false
1308 ],
957 - "default": "yes"
1309 + "default": true
1310 }
1311 },
1312 {
@@ -1257,6 +1609,15 @@
1609 {
1610 "$ref": "#/components/parameters/scopeContexts"
1611 },
1612 + {
1613 + "$ref": "#/components/parameters/scopeInstances"
1614 + },
1615 + {
1616 + "$ref": "#/components/parameters/scopeLabels"
1617 + },
1618 + {
1619 + "$ref": "#/components/parameters/scopeDimensions"
1620 + },
1621 {
1622 "$ref": "#/components/parameters/filterNodes"
1623 },
@@ -1304,6 +1665,114 @@
1665 },
1666 {
1667 "$ref": "#/components/parameters/dataTimeGroupOptions2"
1668 + },
1669 + {
1670 + "$ref": "#/components/parameters/cardinalityLimit"
1671 + }
1672 + ],
1673 + "responses": {
1674 + "200": {
1675 + "description": "JSON object with weights for each context, chart and dimension.",
1676 + "content": {
1677 + "application/json": {
1678 + "schema": {
1679 + "$ref": "#/components/schemas/weights2"
1680 + }
1681 + }
1682 + }
1683 + },
1684 + "400": {
1685 + "description": "The given parameters are invalid."
1686 + },
1687 + "403": {
1688 + "description": "metrics correlations are not enabled on this Netdata Agent."
1689 + },
1690 + "404": {
1691 + "description": "No charts could be found, or the method that correlated the metrics did not produce any result.\n"
1692 + },
1693 + "504": {
1694 + "description": "Timeout - the query took too long and has been cancelled."
1695 + }
1696 + }
1697 + }
1698 + },
1699 + "/api/v3/weights": {
1700 + "get": {
1701 + "operationId": "weights3",
1702 + "tags": [
1703 + "weights"
1704 + ],
1705 + "summary": "Score or weight all or some of the metrics, across all nodes, according to various algorithms.",
1706 + "description": "This endpoint goes through all metrics and scores them according to an algorithm.\nThis is the same as `/api/v2/weights` but may support additional features in the future.\n",
1707 + "parameters": [
1708 + {
1709 + "$ref": "#/components/parameters/weightMethods"
1710 + },
1711 + {
1712 + "$ref": "#/components/parameters/scopeNodes"
1713 + },
1714 + {
1715 + "$ref": "#/components/parameters/scopeContexts"
1716 + },
1717 + {
1718 + "$ref": "#/components/parameters/scopeInstances"
1719 + },
1720 + {
1721 + "$ref": "#/components/parameters/scopeLabels"
1722 + },
1723 + {
1724 + "$ref": "#/components/parameters/scopeDimensions"
1725 + },
1726 + {
1727 + "$ref": "#/components/parameters/filterNodes"
1728 + },
1729 + {
1730 + "$ref": "#/components/parameters/filterContexts"
1731 + },
1732 + {
1733 + "$ref": "#/components/parameters/filterInstances"
1734 + },
1735 + {
1736 + "$ref": "#/components/parameters/filterLabels"
1737 + },
1738 + {
1739 + "$ref": "#/components/parameters/filterAlerts"
1740 + },
1741 + {
1742 + "$ref": "#/components/parameters/filterDimensions"
1743 + },
1744 + {
1745 + "$ref": "#/components/parameters/baselineAfter"
1746 + },
1747 + {
1748 + "$ref": "#/components/parameters/baselineBefore"
1749 + },
1750 + {
1751 + "$ref": "#/components/parameters/after"
1752 + },
1753 + {
1754 + "$ref": "#/components/parameters/before"
1755 + },
1756 + {
1757 + "$ref": "#/components/parameters/tier"
1758 + },
1759 + {
1760 + "$ref": "#/components/parameters/points"
1761 + },
1762 + {
1763 + "$ref": "#/components/parameters/timeoutMS"
1764 + },
1765 + {
1766 + "$ref": "#/components/parameters/dataQueryOptions"
1767 + },
1768 + {
1769 + "$ref": "#/components/parameters/dataTimeGroup2"
1770 + },
1771 + {
1772 + "$ref": "#/components/parameters/dataTimeGroupOptions2"
1773 + },
1774 + {
1775 + "$ref": "#/components/parameters/cardinalityLimit"
1776 }
1777 ],
1778 "responses": {
@@ -1872,6 +2341,39 @@
2341 "default": "*"
2342 }
2343 },
2344 + "scopeInstances": {
2345 + "name": "scope_instances",
2346 + "in": "query",
2347 + "description": "A simple pattern limiting the instances scope of the query. The scope controls both data and metadata response. This limits which instances are even considered for the query, including their visibility in the query response metadata. The simple pattern is checked against the instance `id`, the instance `name`, the fully qualified name of the instance `id` and `name`, like `instance@machine_guid`, where `instance` is either its `id` or `name`. Both positive and negative simple pattern expressions are supported.\n",
2348 + "required": false,
2349 + "schema": {
2350 + "type": "string",
2351 + "format": "simple pattern",
2352 + "default": "*"
2353 + }
2354 + },
2355 + "scopeLabels": {
2356 + "name": "scope_labels",
2357 + "in": "query",
2358 + "description": "A simple pattern limiting the labels scope of the query. The scope controls both data and metadata response. This limits which instances are even considered for the query based on their labels. The simple pattern is checked against `name:value` of all the labels of all the eligible instances (as filtered by scope nodes, scope contexts, and scope instances). Only instances having labels that match this pattern will be included in the query. Both positive and negative simple pattern expressions are supported.\n",
2359 + "required": false,
2360 + "schema": {
2361 + "type": "string",
2362 + "format": "simple pattern",
2363 + "default": "*"
2364 + }
2365 + },
2366 + "scopeDimensions": {
2367 + "name": "scope_dimensions",
2368 + "in": "query",
2369 + "description": "A simple pattern limiting the dimensions scope of the query. The scope controls both data and metadata response. This limits which dimensions are even considered for the query, including their visibility in the query response metadata. The simple pattern is checked against both the dimension `id` and `name`. Both positive and negative simple pattern expressions are supported.\n",
2370 + "required": false,
2371 + "schema": {
2372 + "type": "string",
2373 + "format": "simple pattern",
2374 + "default": "*"
2375 + }
2376 + },
2377 "filterNodes": {
2378 "name": "nodes",
2379 "in": "query",
@@ -1992,7 +2494,7 @@
2494 "dataQueryOptions": {
2495 "name": "options",
2496 "in": "query",
1995 - "description": "Options that affect data generation.\n* `jsonwrap` - Wrap the output in a JSON object with metadata about the query.\n* `raw` - change the output so that it is aggregatable across multiple such queries. Supported by `/api/v2` data queries and `json2` format.\n* `minify` - Remove unnecessary spaces and newlines from the output.\n* `debug` - Provide additional information in `jsonwrap` output to help tracing issues.\n* `nonzero` - Do not return dimensions that all their values are zero, to improve the visual appearance of charts. They will still be returned if all the dimensions are entirely zero.\n* `null2zero` - Replace `null` values with `0`.\n* `absolute` or `abs` - Traditionally Netdata returns select dimensions negative to improve visual appearance. This option turns this feature off.\n* `display-absolute` - Only used by badges, to do color calculation using the signed value, but render the value without a sign.\n* `flip` or `reversed` - Order the timestamps array in reverse order (newest to oldest).\n* `min2max` - When flattening multi-dimensional data into a single metric format, use `max - min` instead of `sum`. This is EOL - use `/api/v2` to control aggregation across dimensions.\n* `percentage` - Convert all values into a percentage vs the row total. When enabled, Netdata will query all dimensions, even the ones that have not been selected or are hidden, to find the row total, in order to calculate the percentage of each dimension selected.\n* `seconds` - Output timestamps in seconds instead of dates.\n* `milliseconds` or `ms` - Output timestamps in milliseconds instead of dates.\n* `unaligned` - by default queries are aligned to the the view, so that as time passes past data returned do not change. When a data query will not be used for visualization, `unaligned` can be given to avoid aligning the query time-frame for visual precision.\n* `match-ids`, `match-names`. By default filters match both IDs and names when they are available. Setting either of the two options will disable the other.\n* `anomaly-bit` - query the anomaly information instead of metric values. This is EOL, use `/api/v2` and `json2` format which always returns this information and many more.\n* `jw-anomaly-rates` - return anomaly rates as a separate result set in the same `json` format response. This is EOL, use `/api/v2` and `json2` format which always returns information and many more. \n* `details` - `/api/v2/data` returns in `jsonwrap` the full tree of dimensions that have been matched by the query.\n* `group-by-labels` - `/api/v2/data` returns in `jsonwrap` flattened labels per output dimension. These are used to identify the instances that have been aggregated into each dimension, making it possible to provide a map, like Netdata does for Kubernetes.\n* `natural-points` - return timestamps as found in the database. The result is again fixed-step, but the query engine attempts to align them with the timestamps found in the database.\n* `virtual-points` - return timestamps independent of the database alignment. This is needed aggregating data across multiple Netdata Agents, to ensure that their outputs do not need to be interpolated to be merged.\n* `selected-tier` - use data exclusively from the selected tier given with the `tier` parameter. This option is set automatically when the `tier` parameter is set.\n* `all-dimensions` - In `/api/v1` `jsonwrap` include metadata for all candidate metrics examined. In `/api/v2` this is standard behavior and no option is needed.\n* `label-quotes` - In `csv` output format, enclose each header label in quotes.\n* `objectrows` - Each row of value should be an object, not an array (only for `json` format).\n* `google_json` - Comply with google JSON/JSONP specs (only for `json` format).\n",
2497 + "description": "Options that affect data generation.\n* `jsonwrap` - Wrap the output in a JSON object with metadata about the query.\n* `raw` - change the output so that it is aggregatable across multiple such queries. Supported by `/api/v2` data queries and `json2` format.\n* `minify` - Remove unnecessary spaces and newlines from the output.\n* `debug` - Provide additional information in `jsonwrap` output to help tracing issues.\n* `nonzero` - Do not return dimensions that all their values are zero, to improve the visual appearance of charts. They will still be returned if all the dimensions are entirely zero.\n* `null2zero` - Replace `null` values with `0`.\n* `absolute` or `abs` - Traditionally Netdata returns select dimensions negative to improve visual appearance. This option turns this feature off.\n* `display-absolute` - Only used by badges, to do color calculation using the signed value, but render the value without a sign.\n* `flip` or `reversed` - Order the timestamps array in reverse order (newest to oldest).\n* `min2max` - When flattening multi-dimensional data into a single metric format, use `max - min` instead of `sum`. This is EOL - use `/api/v2` to control aggregation across dimensions.\n* `percentage` - Convert all values into a percentage vs the row total. When enabled, Netdata will query all dimensions, even the ones that have not been selected or are hidden, to find the row total, in order to calculate the percentage of each dimension selected.\n* `seconds` - Output timestamps in seconds instead of dates.\n* `milliseconds` or `ms` - Output timestamps in milliseconds instead of dates.\n* `unaligned` - by default queries are aligned to the the view, so that as time passes past data returned do not change. When a data query will not be used for visualization, `unaligned` can be given to avoid aligning the query time-frame for visual precision.\n* `match-ids`, `match-names`. By default filters match both IDs and names when they are available. Setting either of the two options will disable the other.\n* `anomaly-bit` - query the anomaly information instead of metric values. This is EOL, use `/api/v2` and `json2` format which always returns this information and many more.\n* `jw-anomaly-rates` - return anomaly rates as a separate result set in the same `json` format response. This is EOL, use `/api/v2` and `json2` format which always returns information and many more. \n* `details` - `/api/v2/data` returns in `jsonwrap` the full tree of dimensions that have been matched by the query.\n* `group-by-labels` - `/api/v2/data` returns in `jsonwrap` flattened labels per output dimension. These are used to identify the instances that have been aggregated into each dimension, making it possible to provide a map, like Netdata does for Kubernetes.\n* `natural-points` - return timestamps as found in the database. The result is again fixed-step, but the query engine attempts to align them with the timestamps found in the database.\n* `virtual-points` - return timestamps independent of the database alignment. This is needed aggregating data across multiple Netdata Agents, to ensure that their outputs do not need to be interpolated to be merged.\n* `selected-tier` - use data exclusively from the selected tier given with the `tier` parameter. This option is set automatically when the `tier` parameter is set.\n* `all-dimensions` - In `/api/v1` `jsonwrap` include metadata for all candidate metrics examined. In `/api/v2` this is standard behavior and no option is needed.\n* `label-quotes` - In `csv` output format, enclose each header label in quotes.\n* `objectrows` - Each row of value should be an object, not an array (only for `json` format).\n* `google_json` - Comply with google JSON/JSONP specs (only for `json` format).\n* `minimal-stats` or `minimal` - Reduce the amount of statistics returned in `jsonwrap` format to save bandwidth.\n* `long-json-keys` or `long-keys` - Use descriptive key names in JSON output instead of abbreviated ones.\n* `mcp-info` - Include additional metadata useful for the Model Context Protocol (MCP) integration.\n* `rfc3339` - Return timestamps in RFC3339 format (e.g., \"2023-01-01T00:00:00Z\") instead of Unix timestamps.\n",
2498 "required": false,
2499 "allowEmptyValue": false,
2500 "schema": {
@@ -2029,7 +2531,13 @@
2531 "all-dimensions",
2532 "label-quotes",
2533 "objectrows",
2032 - "google_json"
2534 + "google_json",
2535 + "minimal-stats",
2536 + "minimal",
2537 + "long-json-keys",
2538 + "long-keys",
2539 + "mcp-info",
2540 + "rfc3339"
2541 ]
2542 },
2543 "default": [
@@ -2085,7 +2593,9 @@
2593 "trimmed-median10",
2594 "trimmed-median15",
2595 "trimmed-median20",
2088 - "trimmed-median25"
2596 + "trimmed-median25",
2597 + "ema",
2598 + "extremes"
2599 ],
2600 "default": "average"
2601 }
@@ -2137,7 +2647,9 @@
2647 "trimmed-median10",
2648 "trimmed-median15",
2649 "trimmed-median20",
2140 - "trimmed-median25"
2650 + "trimmed-median25",
2651 + "ema",
2652 + "extremes"
2653 ],
2654 "default": "average"
2655 }
@@ -2160,6 +2672,46 @@
2672 "type": "string"
2673 }
2674 },
2675 + "cardinalityLimit": {
2676 + "name": "cardinality_limit",
2677 + "in": "query",
2678 + "description": "Limits the number of unique items (contexts, instances, dimensions) returned in the query result. This is useful for preventing excessive memory usage and response sizes when queries match a large number of metrics. The query engine will return the most relevant items up to this limit.\n",
2679 + "required": false,
2680 + "schema": {
2681 + "type": "integer",
2682 + "format": "int64",
2683 + "minimum": 1,
2684 + "default": 10000
2685 + }
2686 + },
2687 + "contextsQueryOptions": {
2688 + "name": "options",
2689 + "in": "query",
2690 + "description": "Options that affect the output of contexts metadata queries.\n* `minify` - Remove unnecessary spaces and newlines from the output to reduce bandwidth.\n* `debug` - Provide additional debugging information in the response.\n* `config` - Include alert configuration information (used by /api/v2/alert_transitions).\n* `instances` - Include alert/context instance information (used by /api/v2/alerts).\n* `values` - Include latest metric values (used by /api/v2/alerts).\n* `summary` - Include summary counters (used by /api/v2/alerts).\n* `mcp` - Format output for Model Context Protocol (MCP) integration.\n* `dimensions` - Include dimension information for each context.\n* `labels` - Include label information for each context.\n* `priorities` - Include priority information for each context.\n* `titles` - Include title information for each context.\n* `rfc3339` - Return timestamps in RFC3339 format (e.g., \"2023-01-01T00:00:00Z\") instead of Unix timestamps.\n\nNote: Some options like `retention`, `liveness`, `family`, and `units` are automatically included by specific endpoints and cannot be controlled via this parameter.\n",
2691 + "required": false,
2692 + "allowEmptyValue": false,
2693 + "schema": {
2694 + "type": "array",
2695 + "items": {
2696 + "type": "string",
2697 + "enum": [
2698 + "minify",
2699 + "debug",
2700 + "config",
2701 + "instances",
2702 + "values",
2703 + "summary",
2704 + "mcp",
2705 + "dimensions",
2706 + "labels",
2707 + "priorities",
2708 + "titles",
2709 + "rfc3339"
2710 + ]
2711 + },
2712 + "default": []
2713 + }
2714 + },
2715 "dataTimeResampling1": {
2716 "name": "gtime",
2717 "in": "query",
@@ -2296,7 +2848,7 @@
2848 "contextOptions1": {
2849 "name": "options",
2850 "in": "query",
2299 - "description": "Options that affect data generation.",
2851 + "description": "Options that affect data generation.\n* `full` or `all` - Include all available information.\n* `charts` - Include chart information.\n* `dimensions` - Include dimension information.\n* `labels` - Include label information.\n* `uuids` - Include UUIDs in the response.\n* `queue` - Include queue information.\n* `flags` - Include internal flags.\n* `deleted` - Include deleted items.\n* `deepscan` - Perform a deep scan for contexts.\n* `rfc3339` - Return timestamps in RFC3339 format instead of Unix timestamps.\n",
2852 "required": false,
2853 "schema": {
2854 "type": "array",
@@ -2312,7 +2864,8 @@
2864 "queue",
2865 "flags",
2866 "deleted",
2315 - "deepscan"
2867 + "deepscan",
2868 + "rfc3339"
2869 ]
2870 }
2871 }
src/web/api/netdata-swagger.yaml
+406 -2
@@ -1,7 +1,44 @@
1 openapi: 3.0.0
2 info:
3 title: Netdata API
4 - description: Real-time performance and health monitoring.
4 + description: |
5 + Real-time performance and health monitoring.
6 +
7 + ## API Versions
8 +
9 + Netdata provides three API versions:
10 + - **v1**: The original API, focused on single-node operations
11 + - **v2**: Multi-node API with advanced grouping and aggregation capabilities
12 + - **v3**: The latest API version that combines v1 and v2 endpoints and may include additional features
13 +
14 + ### v3 API Endpoints
15 +
16 + The v3 API provides the following endpoints (most reuse v1 or v2 implementations):
17 + - `/api/v3/data` - Multi-dimensional data queries (uses v2 implementation)
18 + - `/api/v3/weights` - Metric scoring/correlation (uses v2 implementation)
19 + - `/api/v3/contexts` - Context metadata (uses v2 implementation)
20 + - `/api/v3/nodes` - Node information (uses v2 implementation)
21 + - `/api/v3/q` - Full-text search (uses v2 implementation)
22 + - `/api/v3/alerts` - Alert information (uses v2 implementation)
23 + - `/api/v3/alert_transitions` - Alert state transitions (uses v2 implementation)
24 + - `/api/v3/alert_config` - Alert configuration (uses v2 implementation)
25 + - `/api/v3/functions` - Available functions (uses v2 implementation)
26 + - `/api/v3/function` - Execute functions (uses v1 implementation)
27 + - `/api/v3/info` - Agent information (uses v2 implementation)
28 + - `/api/v3/node_instances` - Node instance information (uses v2 implementation)
29 + - `/api/v3/stream_path` - Streaming topology (v3 specific)
30 + - `/api/v3/versions` - Version information (uses v2 implementation)
31 + - `/api/v3/badge.svg` - Dynamic badges (uses v1 implementation)
32 + - `/api/v3/allmetrics` - Export metrics (uses v1 implementation)
33 + - `/api/v3/context` - Single context info (uses v1 implementation)
34 + - `/api/v3/variable` - Variable information (uses v1 implementation)
35 + - `/api/v3/config` - Dynamic configuration (uses v1 implementation)
36 + - `/api/v3/settings` - Agent settings (v3 specific)
37 + - `/api/v3/me` - Current user information (v3 specific)
38 + - `/api/v3/claim` - Agent claiming (v3 specific)
39 + - Additional management and streaming endpoints
40 +
41 + Note: This documentation currently focuses on v1 and v2 endpoints. The v3 endpoints listed above generally accept the same parameters as their v1/v2 counterparts.
42 version: "v1-rolling"
43 contact:
44 name: Netdata Agent API
@@ -47,6 +84,7 @@ paths:
84 - $ref: '#/components/parameters/scopeContexts'
85 - $ref: '#/components/parameters/filterNodes'
86 - $ref: '#/components/parameters/filterContexts'
87 + - $ref: '#/components/parameters/contextsQueryOptions'
88 responses:
89 "200":
90 description: OK
@@ -67,6 +105,41 @@ paths:
105 type: array
106 items:
107 $ref: '#/components/schemas/nodeFull'
108 + /api/v3/nodes:
109 + get:
110 + operationId: getNodes3
111 + tags:
112 + - nodes
113 + summary: Nodes Info v3
114 + description: |
115 + Get a list of all nodes hosted by this Netdata Agent.
116 + This is the same as `/api/v2/nodes` but may support additional features in the future.
117 + parameters:
118 + - $ref: '#/components/parameters/scopeNodes'
119 + - $ref: '#/components/parameters/scopeContexts'
120 + - $ref: '#/components/parameters/filterNodes'
121 + - $ref: '#/components/parameters/filterContexts'
122 + - $ref: '#/components/parameters/contextsQueryOptions'
123 + responses:
124 + "200":
125 + description: OK
126 + content:
127 + application/json:
128 + schema:
129 + description: |
130 + `/api/v3/nodes` response for all nodes hosted by a Netdata Agent.
131 + type: object
132 + properties:
133 + api:
134 + $ref: '#/components/schemas/api'
135 + agents:
136 + $ref: '#/components/schemas/agents'
137 + versions:
138 + $ref: '#/components/schemas/versions'
139 + nodes:
140 + type: array
141 + items:
142 + $ref: '#/components/schemas/nodeFull'
143 /api/v2/contexts:
144 get:
145 operationId: getContexts2
@@ -80,6 +153,29 @@ paths:
153 - $ref: '#/components/parameters/scopeContexts'
154 - $ref: '#/components/parameters/filterNodes'
155 - $ref: '#/components/parameters/filterContexts'
156 + - $ref: '#/components/parameters/contextsQueryOptions'
157 + responses:
158 + "200":
159 + description: OK
160 + content:
161 + application/json:
162 + schema:
163 + $ref: '#/components/schemas/contexts2'
164 + /api/v3/contexts:
165 + get:
166 + operationId: getContexts3
167 + tags:
168 + - contexts
169 + summary: Contexts Info v3
170 + description: |
171 + Get a list of all contexts, across all nodes, hosted by this Netdata Agent.
172 + This is the same as `/api/v2/contexts` but may support additional features in the future.
173 + parameters:
174 + - $ref: '#/components/parameters/scopeNodes'
175 + - $ref: '#/components/parameters/scopeContexts'
176 + - $ref: '#/components/parameters/filterNodes'
177 + - $ref: '#/components/parameters/filterContexts'
178 + - $ref: '#/components/parameters/contextsQueryOptions'
179 responses:
180 "200":
181 description: OK
@@ -107,6 +203,36 @@ paths:
203 - $ref: '#/components/parameters/scopeContexts'
204 - $ref: '#/components/parameters/filterNodes'
205 - $ref: '#/components/parameters/filterContexts'
206 + - $ref: '#/components/parameters/contextsQueryOptions'
207 + responses:
208 + "200":
209 + description: OK
210 + content:
211 + application/json:
212 + schema:
213 + $ref: '#/components/schemas/contexts2'
214 + /api/v3/q:
215 + get:
216 + operationId: q3
217 + tags:
218 + - contexts
219 + summary: Full Text Search v3
220 + description: |
221 + Get a list of contexts, across all nodes, hosted by this Netdata Agent, matching a string expression.
222 + This is the same as `/api/v2/q` but may support additional features in the future.
223 + parameters:
224 + - name: q
225 + in: query
226 + description: The strings to search for, formatted as a simple pattern
227 + required: true
228 + schema:
229 + type: string
230 + format: simple pattern
231 + - $ref: '#/components/parameters/scopeNodes'
232 + - $ref: '#/components/parameters/scopeContexts'
233 + - $ref: '#/components/parameters/filterNodes'
234 + - $ref: '#/components/parameters/filterContexts'
235 + - $ref: '#/components/parameters/contextsQueryOptions'
236 responses:
237 "200":
238 description: OK
@@ -407,9 +533,127 @@ paths:
533 - average
534 - sum
535 - percentage
536 + - extremes
537 + default: average
538 + - $ref: '#/components/parameters/scopeNodes'
539 + - $ref: '#/components/parameters/scopeContexts'
540 + - $ref: '#/components/parameters/scopeInstances'
541 + - $ref: '#/components/parameters/scopeLabels'
542 + - $ref: '#/components/parameters/scopeDimensions'
543 + - $ref: '#/components/parameters/filterNodes'
544 + - $ref: '#/components/parameters/filterContexts'
545 + - $ref: '#/components/parameters/filterInstances'
546 + - $ref: '#/components/parameters/filterLabels'
547 + - $ref: '#/components/parameters/filterAlerts'
548 + - $ref: '#/components/parameters/filterDimensions'
549 + - $ref: '#/components/parameters/after'
550 + - $ref: '#/components/parameters/before'
551 + - $ref: '#/components/parameters/points'
552 + - $ref: '#/components/parameters/tier'
553 + - $ref: '#/components/parameters/dataQueryOptions'
554 + - $ref: '#/components/parameters/dataTimeGroup2'
555 + - $ref: '#/components/parameters/dataTimeGroupOptions2'
556 + - $ref: '#/components/parameters/dataTimeResampling2'
557 + - $ref: '#/components/parameters/dataFormat2'
558 + - $ref: '#/components/parameters/cardinalityLimit'
559 + - $ref: '#/components/parameters/timeoutMS'
560 + - $ref: '#/components/parameters/callback'
561 + - $ref: '#/components/parameters/filename'
562 + - $ref: '#/components/parameters/tqx'
563 + responses:
564 + "200":
565 + description: |
566 + The call was successful. The response includes the data in the format requested.
567 + content:
568 + application/json:
569 + schema:
570 + oneOf:
571 + - $ref: '#/components/schemas/jsonwrap2'
572 + - $ref: '#/components/schemas/data_json_formats2'
573 + text/plain:
574 + schema:
575 + type: string
576 + format: according to the format requested.
577 + text/html:
578 + schema:
579 + type: string
580 + format: html
581 + application/x-javascript:
582 + schema:
583 + type: string
584 + format: javascript
585 + "400":
586 + description: |
587 + Bad request - the body will include a message stating what is wrong.
588 + "500":
589 + description: |
590 + Internal server error. This usually means the server is out of memory.
591 + /api/v3/data:
592 + get:
593 + operationId: dataQuery3
594 + tags:
595 + - data
596 + summary: Data Query v3
597 + description: |
598 + Multi-node, multi-context, multi-instance, multi-dimension data queries, with time and metric aggregation.
599 + This is the same as `/api/v2/data` but may support additional features in the future.
600 + parameters:
601 + - name: group_by
602 + in: query
603 + description: |
604 + A comma separated list of the groupings required.
605 + All possible values can be combined together, except `selected`. If `selected` is given in the list, all others are ignored.
606 + The order they are placed in the list is currently ignored.
607 + This parameter is also accepted as `group_by[0]` and `group_by[1]` when multiple grouping passes are required.
608 + required: false
609 + schema:
610 + type: array
611 + items:
612 + type: string
613 + enum:
614 + - dimension
615 + - instance
616 + - percentage-of-instance
617 + - label
618 + - node
619 + - context
620 + - units
621 + - selected
622 + default:
623 + - dimension
624 + - name: group_by_label
625 + in: query
626 + description: |
627 + A comma separated list of the label keys to group by their values. The order of the labels in the list is respected.
628 + This parameter is also accepted as `group_by_label[0]` and `group_by_label[1]` when multiple grouping passes are required.
629 + required: false
630 + schema:
631 + type: string
632 + format: comma separated list of label keys to group by
633 + default: ""
634 + - name: aggregation
635 + in: query
636 + description: |
637 + The aggregation function to apply when grouping metrics together.
638 + When option `raw` is given, `average` and `avg` behave like `sum` and the caller is expected to calculate the average.
639 + This parameter is also accepted as `aggregation[0]` and `aggregation[1]` when multiple grouping passes are required.
640 + required: false
641 + schema:
642 + type: string
643 + enum:
644 + - min
645 + - max
646 + - avg
647 + - average
648 + - sum
649 + - percentage
650 + - extremes
651 default: average
652 - $ref: '#/components/parameters/scopeNodes'
653 - $ref: '#/components/parameters/scopeContexts'
654 + - $ref: '#/components/parameters/scopeInstances'
655 + - $ref: '#/components/parameters/scopeLabels'
656 + - $ref: '#/components/parameters/scopeDimensions'
657 - $ref: '#/components/parameters/filterNodes'
658 - $ref: '#/components/parameters/filterContexts'
659 - $ref: '#/components/parameters/filterInstances'
@@ -425,6 +669,7 @@ paths:
669 - $ref: '#/components/parameters/dataTimeGroupOptions2'
670 - $ref: '#/components/parameters/dataTimeResampling2'
671 - $ref: '#/components/parameters/dataFormat2'
672 + - $ref: '#/components/parameters/cardinalityLimit'
673 - $ref: '#/components/parameters/timeoutMS'
674 - $ref: '#/components/parameters/callback'
675 - $ref: '#/components/parameters/filename'
@@ -813,6 +1058,9 @@ paths:
1058 - $ref: '#/components/parameters/weightMethods'
1059 - $ref: '#/components/parameters/scopeNodes'
1060 - $ref: '#/components/parameters/scopeContexts'
1061 + - $ref: '#/components/parameters/scopeInstances'
1062 + - $ref: '#/components/parameters/scopeLabels'
1063 + - $ref: '#/components/parameters/scopeDimensions'
1064 - $ref: '#/components/parameters/filterNodes'
1065 - $ref: '#/components/parameters/filterContexts'
1066 - $ref: '#/components/parameters/filterInstances'
@@ -829,6 +1077,56 @@ paths:
1077 - $ref: '#/components/parameters/dataQueryOptions'
1078 - $ref: '#/components/parameters/dataTimeGroup2'
1079 - $ref: '#/components/parameters/dataTimeGroupOptions2'
1080 + - $ref: '#/components/parameters/cardinalityLimit'
1081 + responses:
1082 + "200":
1083 + description: JSON object with weights for each context, chart and dimension.
1084 + content:
1085 + application/json:
1086 + schema:
1087 + $ref: "#/components/schemas/weights2"
1088 + "400":
1089 + description: The given parameters are invalid.
1090 + "403":
1091 + description: metrics correlations are not enabled on this Netdata Agent.
1092 + "404":
1093 + description: |
1094 + No charts could be found, or the method that correlated the metrics did not produce any result.
1095 + "504":
1096 + description: Timeout - the query took too long and has been cancelled.
1097 + /api/v3/weights:
1098 + get:
1099 + operationId: weights3
1100 + tags:
1101 + - weights
1102 + summary: Score or weight all or some of the metrics, across all nodes, according to various algorithms.
1103 + description: |
1104 + This endpoint goes through all metrics and scores them according to an algorithm.
1105 + This is the same as `/api/v2/weights` but may support additional features in the future.
1106 + parameters:
1107 + - $ref: '#/components/parameters/weightMethods'
1108 + - $ref: '#/components/parameters/scopeNodes'
1109 + - $ref: '#/components/parameters/scopeContexts'
1110 + - $ref: '#/components/parameters/scopeInstances'
1111 + - $ref: '#/components/parameters/scopeLabels'
1112 + - $ref: '#/components/parameters/scopeDimensions'
1113 + - $ref: '#/components/parameters/filterNodes'
1114 + - $ref: '#/components/parameters/filterContexts'
1115 + - $ref: '#/components/parameters/filterInstances'
1116 + - $ref: '#/components/parameters/filterLabels'
1117 + - $ref: '#/components/parameters/filterAlerts'
1118 + - $ref: '#/components/parameters/filterDimensions'
1119 + - $ref: '#/components/parameters/baselineAfter'
1120 + - $ref: '#/components/parameters/baselineBefore'
1121 + - $ref: '#/components/parameters/after'
1122 + - $ref: '#/components/parameters/before'
1123 + - $ref: '#/components/parameters/tier'
1124 + - $ref: '#/components/parameters/points'
1125 + - $ref: '#/components/parameters/timeoutMS'
1126 + - $ref: '#/components/parameters/dataQueryOptions'
1127 + - $ref: '#/components/parameters/dataTimeGroup2'
1128 + - $ref: '#/components/parameters/dataTimeGroupOptions2'
1129 + - $ref: '#/components/parameters/cardinalityLimit'
1130 responses:
1131 "200":
1132 description: JSON object with weights for each context, chart and dimension.
@@ -1202,6 +1500,36 @@ components:
1500 type: string
1501 format: simple pattern
1502 default: "*"
1503 + scopeInstances:
1504 + name: scope_instances
1505 + in: query
1506 + description: |
1507 + A simple pattern limiting the instances scope of the query. The scope controls both data and metadata response. This limits which instances are even considered for the query, including their visibility in the query response metadata. The simple pattern is checked against the instance `id`, the instance `name`, the fully qualified name of the instance `id` and `name`, like `instance@machine_guid`, where `instance` is either its `id` or `name`. Both positive and negative simple pattern expressions are supported.
1508 + required: false
1509 + schema:
1510 + type: string
1511 + format: simple pattern
1512 + default: "*"
1513 + scopeLabels:
1514 + name: scope_labels
1515 + in: query
1516 + description: |
1517 + A simple pattern limiting the labels scope of the query. The scope controls both data and metadata response. This limits which instances are even considered for the query based on their labels. The simple pattern is checked against `name:value` of all the labels of all the eligible instances (as filtered by scope nodes, scope contexts, and scope instances). Only instances having labels that match this pattern will be included in the query. Both positive and negative simple pattern expressions are supported.
1518 + required: false
1519 + schema:
1520 + type: string
1521 + format: simple pattern
1522 + default: "*"
1523 + scopeDimensions:
1524 + name: scope_dimensions
1525 + in: query
1526 + description: |
1527 + A simple pattern limiting the dimensions scope of the query. The scope controls both data and metadata response. This limits which dimensions are even considered for the query, including their visibility in the query response metadata. The simple pattern is checked against both the dimension `id` and `name`. Both positive and negative simple pattern expressions are supported.
1528 + required: false
1529 + schema:
1530 + type: string
1531 + format: simple pattern
1532 + default: "*"
1533 filterNodes:
1534 name: nodes
1535 in: query
@@ -1339,6 +1667,10 @@ components:
1667 * `label-quotes` - In `csv` output format, enclose each header label in quotes.
1668 * `objectrows` - Each row of value should be an object, not an array (only for `json` format).
1669 * `google_json` - Comply with google JSON/JSONP specs (only for `json` format).
1670 + * `minimal-stats` or `minimal` - Reduce the amount of statistics returned in `jsonwrap` format to save bandwidth.
1671 + * `long-json-keys` or `long-keys` - Use descriptive key names in JSON output instead of abbreviated ones.
1672 + * `mcp-info` - Include additional metadata useful for the Model Context Protocol (MCP) integration.
1673 + * `rfc3339` - Return timestamps in RFC3339 format (e.g., "2023-01-01T00:00:00Z") instead of Unix timestamps.
1674 required: false
1675 allowEmptyValue: false
1676 schema:
@@ -1376,6 +1708,12 @@ components:
1708 - label-quotes
1709 - objectrows
1710 - google_json
1711 + - minimal-stats
1712 + - minimal
1713 + - long-json-keys
1714 + - long-keys
1715 + - mcp-info
1716 + - rfc3339
1717 default:
1718 - seconds
1719 - jsonwrap
@@ -1428,6 +1766,8 @@ components:
1766 - trimmed-median15
1767 - trimmed-median20
1768 - trimmed-median25
1769 + - ema
1770 + - extremes
1771 default: average
1772 dataTimeGroup2:
1773 name: time_group
@@ -1478,6 +1818,8 @@ components:
1818 - trimmed-median15
1819 - trimmed-median20
1820 - trimmed-median25
1821 + - ema
1822 + - extremes
1823 default: average
1824 dataTimeGroupOptions1:
1825 name: group_options
@@ -1495,6 +1837,56 @@ components:
1837 required: false
1838 schema:
1839 type: string
1840 + cardinalityLimit:
1841 + name: cardinality_limit
1842 + in: query
1843 + description: |
1844 + Limits the number of unique items (contexts, instances, dimensions) returned in the query result. This is useful for preventing excessive memory usage and response sizes when queries match a large number of metrics. The query engine will return the most relevant items up to this limit.
1845 + required: false
1846 + schema:
1847 + type: integer
1848 + format: int64
1849 + minimum: 1
1850 + default: 10000
1851 + contextsQueryOptions:
1852 + name: options
1853 + in: query
1854 + description: |
1855 + Options that affect the output of contexts metadata queries.
1856 + * `minify` - Remove unnecessary spaces and newlines from the output to reduce bandwidth.
1857 + * `debug` - Provide additional debugging information in the response.
1858 + * `config` - Include alert configuration information (used by /api/v2/alert_transitions).
1859 + * `instances` - Include alert/context instance information (used by /api/v2/alerts).
1860 + * `values` - Include latest metric values (used by /api/v2/alerts).
1861 + * `summary` - Include summary counters (used by /api/v2/alerts).
1862 + * `mcp` - Format output for Model Context Protocol (MCP) integration.
1863 + * `dimensions` - Include dimension information for each context.
1864 + * `labels` - Include label information for each context.
1865 + * `priorities` - Include priority information for each context.
1866 + * `titles` - Include title information for each context.
1867 + * `rfc3339` - Return timestamps in RFC3339 format (e.g., "2023-01-01T00:00:00Z") instead of Unix timestamps.
1868 +
1869 + Note: Some options like `retention`, `liveness`, `family`, and `units` are automatically included by specific endpoints and cannot be controlled via this parameter.
1870 + required: false
1871 + allowEmptyValue: false
1872 + schema:
1873 + type: array
1874 + items:
1875 + type: string
1876 + enum:
1877 + - minify
1878 + - debug
1879 + - config
1880 + - instances
1881 + - values
1882 + - summary
1883 + - mcp
1884 + - dimensions
1885 + - labels
1886 + - priorities
1887 + - titles
1888 + - rfc3339
1889 + default: []
1890 dataTimeResampling1:
1891 name: gtime
1892 in: query
@@ -1618,7 +2010,18 @@ components:
2010 contextOptions1:
2011 name: options
2012 in: query
1621 - description: Options that affect data generation.
2013 + description: |
2014 + Options that affect data generation.
2015 + * `full` or `all` - Include all available information.
2016 + * `charts` - Include chart information.
2017 + * `dimensions` - Include dimension information.
2018 + * `labels` - Include label information.
2019 + * `uuids` - Include UUIDs in the response.
2020 + * `queue` - Include queue information.
2021 + * `flags` - Include internal flags.
2022 + * `deleted` - Include deleted items.
2023 + * `deepscan` - Perform a deep scan for contexts.
2024 + * `rfc3339` - Return timestamps in RFC3339 format instead of Unix timestamps.
2025 required: false
2026 schema:
2027 type: array
@@ -1635,6 +2038,7 @@ components:
2038 - flags
2039 - deleted
2040 - deepscan
2041 + - rfc3339
2042 chart:
2043 name: chart
2044 in: query
src/web/api/queries/extremes/extremes.c new
+6
@@ -0,0 +1,6 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "extremes.h"
4 +
5 +// ----------------------------------------------------------------------------
6 +// extremes
src/web/api/queries/extremes/extremes.h new
+101
@@ -0,0 +1,101 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_API_QUERY_EXTREMES_H
4 +#define NETDATA_API_QUERY_EXTREMES_H
5 +
6 +#include "../query.h"
7 +#include "../rrdr.h"
8 +
9 +struct tg_extremes {
10 + NETDATA_DOUBLE min; // for negative values
11 + NETDATA_DOUBLE max; // for positive values
12 + size_t pos_count; // count of positive values
13 + size_t neg_count; // count of negative values
14 + size_t zero_count; // count of zero values
15 +};
16 +
17 +static inline void tg_extremes_create(RRDR *r, const char *options __maybe_unused) {
18 + r->time_grouping.data = onewayalloc_callocz(r->internal.owa, 1, sizeof(struct tg_extremes));
19 +}
20 +
21 +// resets when switches dimensions
22 +// so, clear everything to restart
23 +static inline void tg_extremes_reset(RRDR *r) {
24 + struct tg_extremes *g = (struct tg_extremes *)r->time_grouping.data;
25 + g->min = 0;
26 + g->max = 0;
27 + g->pos_count = 0;
28 + g->neg_count = 0;
29 + g->zero_count = 0;
30 +}
31 +
32 +static inline void tg_extremes_free(RRDR *r) {
33 + onewayalloc_freez(r->internal.owa, r->time_grouping.data);
34 + r->time_grouping.data = NULL;
35 +}
36 +
37 +static inline void tg_extremes_add(RRDR *r, NETDATA_DOUBLE value) {
38 + struct tg_extremes *g = (struct tg_extremes *)r->time_grouping.data;
39 +
40 + if (value > 0) {
41 + // For positive values, track the maximum
42 + if (!g->pos_count || value > g->max) {
43 + g->max = value;
44 + }
45 + g->pos_count++;
46 + }
47 + else if (value < 0) {
48 + // For negative values, track the minimum
49 + if (!g->neg_count || value < g->min) {
50 + g->min = value;
51 + }
52 + g->neg_count++;
53 + }
54 + else {
55 + // It's a zero
56 + g->zero_count++;
57 + }
58 +}
59 +
60 +static inline NETDATA_DOUBLE tg_extremes_flush(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr) {
61 + struct tg_extremes *g = (struct tg_extremes *)r->time_grouping.data;
62 +
63 + NETDATA_DOUBLE value;
64 +
65 + if (unlikely(!g->pos_count && !g->neg_count && !g->zero_count)) {
66 + // No values at all
67 + value = 0.0;
68 + *rrdr_value_options_ptr |= RRDR_VALUE_EMPTY;
69 + }
70 + else if (g->pos_count && g->neg_count) {
71 + // If we have both positive and negative values,
72 + // return the one with the greatest absolute value
73 + if (fabsndd(g->max) > fabsndd(g->min))
74 + value = g->max;
75 + else
76 + value = g->min;
77 + }
78 + else if (g->pos_count) {
79 + // Only positive values, return the maximum
80 + value = g->max;
81 + }
82 + else if (g->neg_count) {
83 + // Only negative values, return the minimum
84 + value = g->min;
85 + }
86 + else {
87 + // Only zeros
88 + value = 0.0;
89 + }
90 +
91 + // Reset the state for the next calculation
92 + g->min = 0.0;
93 + g->max = 0.0;
94 + g->pos_count = 0;
95 + g->neg_count = 0;
96 + g->zero_count = 0;
97 +
98 + return value;
99 +}
100 +
101 +#endif //NETDATA_API_QUERY_EXTREMES_H
\ No newline at end of file
src/web/api/queries/query-group-by.c new
+1383
@@ -0,0 +1,1383 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "query-internal.h"
4 +
5 +RRDR_GROUP_BY group_by_parse(const char *group_by_txt) {
6 + char src[strlen(group_by_txt) + 1];
7 + strcatz(src, 0, group_by_txt, sizeof(src));
8 + char *s = src;
9 +
10 + RRDR_GROUP_BY group_by = RRDR_GROUP_BY_NONE;
11 +
12 + while(s) {
13 + char *key = strsep_skip_consecutive_separators(&s, ",| ");
14 + if (!key || !*key) continue;
15 +
16 + if (strcmp(key, "selected") == 0)
17 + group_by |= RRDR_GROUP_BY_SELECTED;
18 +
19 + if (strcmp(key, "dimension") == 0)
20 + group_by |= RRDR_GROUP_BY_DIMENSION;
21 +
22 + if (strcmp(key, "instance") == 0)
23 + group_by |= RRDR_GROUP_BY_INSTANCE;
24 +
25 + if (strcmp(key, "percentage-of-instance") == 0)
26 + group_by |= RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE;
27 +
28 + if (strcmp(key, "label") == 0)
29 + group_by |= RRDR_GROUP_BY_LABEL;
30 +
31 + if (strcmp(key, "node") == 0)
32 + group_by |= RRDR_GROUP_BY_NODE;
33 +
34 + if (strcmp(key, "context") == 0)
35 + group_by |= RRDR_GROUP_BY_CONTEXT;
36 +
37 + if (strcmp(key, "units") == 0)
38 + group_by |= RRDR_GROUP_BY_UNITS;
39 + }
40 +
41 + if((group_by & RRDR_GROUP_BY_SELECTED) && (group_by & ~RRDR_GROUP_BY_SELECTED)) {
42 + internal_error(true, "group-by given by query has 'selected' together with more groupings");
43 + group_by = RRDR_GROUP_BY_SELECTED; // remove all other groupings
44 + }
45 +
46 + if(group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
47 + group_by = RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE; // remove all other groupings
48 +
49 + return group_by;
50 +}
51 +
52 +void buffer_json_group_by_to_array(BUFFER *wb, RRDR_GROUP_BY group_by) {
53 + if(group_by == RRDR_GROUP_BY_NONE)
54 + buffer_json_add_array_item_string(wb, "none");
55 + else {
56 + if (group_by & RRDR_GROUP_BY_DIMENSION)
57 + buffer_json_add_array_item_string(wb, "dimension");
58 +
59 + if (group_by & RRDR_GROUP_BY_INSTANCE)
60 + buffer_json_add_array_item_string(wb, "instance");
61 +
62 + if (group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
63 + buffer_json_add_array_item_string(wb, "percentage-of-instance");
64 +
65 + if (group_by & RRDR_GROUP_BY_LABEL)
66 + buffer_json_add_array_item_string(wb, "label");
67 +
68 + if (group_by & RRDR_GROUP_BY_NODE)
69 + buffer_json_add_array_item_string(wb, "node");
70 +
71 + if (group_by & RRDR_GROUP_BY_CONTEXT)
72 + buffer_json_add_array_item_string(wb, "context");
73 +
74 + if (group_by & RRDR_GROUP_BY_UNITS)
75 + buffer_json_add_array_item_string(wb, "units");
76 +
77 + if (group_by & RRDR_GROUP_BY_SELECTED)
78 + buffer_json_add_array_item_string(wb, "selected");
79 + }
80 +}
81 +
82 +RRDR_GROUP_BY_FUNCTION group_by_aggregate_function_parse(const char *s) {
83 + if(strcmp(s, "average") == 0)
84 + return RRDR_GROUP_BY_FUNCTION_AVERAGE;
85 +
86 + if(strcmp(s, "avg") == 0)
87 + return RRDR_GROUP_BY_FUNCTION_AVERAGE;
88 +
89 + if(strcmp(s, "min") == 0)
90 + return RRDR_GROUP_BY_FUNCTION_MIN;
91 +
92 + if(strcmp(s, "max") == 0)
93 + return RRDR_GROUP_BY_FUNCTION_MAX;
94 +
95 + if(strcmp(s, "sum") == 0)
96 + return RRDR_GROUP_BY_FUNCTION_SUM;
97 +
98 + if(strcmp(s, "percentage") == 0)
99 + return RRDR_GROUP_BY_FUNCTION_PERCENTAGE;
100 +
101 + if(strcmp(s, "extremes") == 0)
102 + return RRDR_GROUP_BY_FUNCTION_EXTREMES;
103 +
104 + return RRDR_GROUP_BY_FUNCTION_AVERAGE;
105 +}
106 +
107 +const char *group_by_aggregate_function_to_string(RRDR_GROUP_BY_FUNCTION group_by_function) {
108 + switch(group_by_function) {
109 + default:
110 + case RRDR_GROUP_BY_FUNCTION_AVERAGE:
111 + return "average";
112 +
113 + case RRDR_GROUP_BY_FUNCTION_MIN:
114 + return "min";
115 +
116 + case RRDR_GROUP_BY_FUNCTION_MAX:
117 + return "max";
118 +
119 + case RRDR_GROUP_BY_FUNCTION_SUM:
120 + return "sum";
121 +
122 + case RRDR_GROUP_BY_FUNCTION_PERCENTAGE:
123 + return "percentage";
124 +
125 + case RRDR_GROUP_BY_FUNCTION_EXTREMES:
126 + return "extremes";
127 + }
128 +}
129 +
130 +// ----------------------------------------------------------------------------
131 +// group by
132 +
133 +struct group_by_label_key {
134 + DICTIONARY *values;
135 +};
136 +
137 +static void group_by_label_key_insert_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
138 + // add the key to our r->label_keys global keys dictionary
139 + DICTIONARY *label_keys = data;
140 + dictionary_set(label_keys, dictionary_acquired_item_name(item), NULL, 0);
141 +
142 + // create a dictionary for the values of this key
143 + struct group_by_label_key *k = value;
144 + k->values = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
145 +}
146 +
147 +static void group_by_label_key_delete_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
148 + struct group_by_label_key *k = value;
149 + dictionary_destroy(k->values);
150 +}
151 +
152 +static int rrdlabels_traversal_cb_to_group_by_label_key(const char *name, const char *value, RRDLABEL_SRC ls __maybe_unused, void *data) {
153 + DICTIONARY *dl = data;
154 + struct group_by_label_key *k = dictionary_set(dl, name, NULL, sizeof(struct group_by_label_key));
155 + dictionary_set(k->values, value, NULL, 0);
156 + return 1;
157 +}
158 +
159 +void rrdr_json_group_by_labels(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
160 + if(!r->label_keys || !r->dl)
161 + return;
162 +
163 + buffer_json_member_add_object(wb, key);
164 +
165 + void *t;
166 + dfe_start_read(r->label_keys, t) {
167 + buffer_json_member_add_array(wb, t_dfe.name);
168 +
169 + for(size_t d = 0; d < r->d ;d++) {
170 + if(!rrdr_dimension_should_be_exposed(r->od[d], options))
171 + continue;
172 +
173 + struct group_by_label_key *k = dictionary_get(r->dl[d], t_dfe.name);
174 + if(k) {
175 + buffer_json_add_array_item_array(wb);
176 + void *tt;
177 + dfe_start_read(k->values, tt) {
178 + buffer_json_add_array_item_string(wb, tt_dfe.name);
179 + }
180 + dfe_done(tt);
181 + buffer_json_array_close(wb);
182 + }
183 + else
184 + buffer_json_add_array_item_string(wb, NULL);
185 + }
186 +
187 + buffer_json_array_close(wb);
188 + }
189 + dfe_done(t);
190 +
191 + buffer_json_object_close(wb); // key
192 +}
193 +
194 +static void rrd2rrdr_set_timestamps(RRDR *r) {
195 + QUERY_TARGET *qt = r->internal.qt;
196 +
197 + internal_fatal(qt->window.points != r->n, "QUERY: mismatch to the number of points in qt and r");
198 +
199 + r->view.group = qt->window.group;
200 + r->view.update_every = (int) query_view_update_every(qt);
201 + r->view.before = qt->window.before;
202 + r->view.after = qt->window.after;
203 +
204 + r->time_grouping.points_wanted = qt->window.points;
205 + r->time_grouping.resampling_group = qt->window.resampling_group;
206 + r->time_grouping.resampling_divisor = qt->window.resampling_divisor;
207 +
208 + r->rows = qt->window.points;
209 +
210 + size_t points_wanted = qt->window.points;
211 + time_t after_wanted = qt->window.after;
212 + time_t before_wanted = qt->window.before; (void)before_wanted;
213 +
214 + time_t view_update_every = r->view.update_every;
215 + time_t query_granularity = (time_t)(r->view.update_every / r->view.group);
216 +
217 + size_t rrdr_line = 0;
218 + time_t first_point_end_time = after_wanted + view_update_every - query_granularity;
219 + time_t now_end_time = first_point_end_time;
220 +
221 + while (rrdr_line < points_wanted) {
222 + r->t[rrdr_line++] = now_end_time;
223 + now_end_time += view_update_every;
224 + }
225 +
226 + internal_fatal(r->t[0] != first_point_end_time, "QUERY: wrong first timestamp in the query");
227 + internal_error(r->t[points_wanted - 1] != before_wanted,
228 + "QUERY: wrong last timestamp in the query, expected %ld, found %ld",
229 + before_wanted, r->t[points_wanted - 1]);
230 +}
231 +
232 +static void query_group_by_make_dimension_key(BUFFER *key, RRDR_GROUP_BY group_by, size_t group_by_id, QUERY_TARGET *qt, QUERY_NODE *qn, QUERY_CONTEXT *qc, QUERY_INSTANCE *qi, QUERY_DIMENSION *qd __maybe_unused, QUERY_METRIC *qm, bool query_has_percentage_of_group) {
233 + buffer_flush(key);
234 + if(unlikely(!query_has_percentage_of_group && qm->status & RRDR_DIMENSION_HIDDEN)) {
235 + buffer_strcat(key, "__hidden_dimensions__");
236 + }
237 + else if(unlikely(group_by & RRDR_GROUP_BY_SELECTED)) {
238 + buffer_strcat(key, "selected");
239 + }
240 + else {
241 + if (group_by & RRDR_GROUP_BY_DIMENSION) {
242 + buffer_fast_strcat(key, "|", 1);
243 + buffer_strcat(key, query_metric_name(qt, qm));
244 + }
245 +
246 + if (group_by & (RRDR_GROUP_BY_INSTANCE|RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)) {
247 + buffer_fast_strcat(key, "|", 1);
248 + buffer_strcat(key, string2str(query_instance_id_fqdn(qi, qt->request.version)));
249 + }
250 +
251 + if (group_by & RRDR_GROUP_BY_LABEL) {
252 + RRDLABELS *labels = rrdinstance_acquired_labels(qi->ria);
253 + for (size_t l = 0; l < qt->group_by[group_by_id].used; l++) {
254 + buffer_fast_strcat(key, "|", 1);
255 + rrdlabels_get_value_to_buffer_or_unset(labels, key, qt->group_by[group_by_id].label_keys[l], "[unset]");
256 + }
257 + }
258 +
259 + if (group_by & RRDR_GROUP_BY_NODE) {
260 + buffer_fast_strcat(key, "|", 1);
261 + buffer_strcat(key, qn->rrdhost->machine_guid);
262 + }
263 +
264 + if (group_by & RRDR_GROUP_BY_CONTEXT) {
265 + buffer_fast_strcat(key, "|", 1);
266 + buffer_strcat(key, rrdcontext_acquired_id(qc->rca));
267 + }
268 +
269 + if (group_by & RRDR_GROUP_BY_UNITS) {
270 + buffer_fast_strcat(key, "|", 1);
271 + buffer_strcat(key, query_target_has_percentage_units(qt) ? "%" : rrdinstance_acquired_units(qi->ria));
272 + }
273 + }
274 +}
275 +
276 +static void query_group_by_make_dimension_id(BUFFER *key, RRDR_GROUP_BY group_by, size_t group_by_id, QUERY_TARGET *qt, QUERY_NODE *qn, QUERY_CONTEXT *qc, QUERY_INSTANCE *qi, QUERY_DIMENSION *qd __maybe_unused, QUERY_METRIC *qm, bool query_has_percentage_of_group) {
277 + buffer_flush(key);
278 + if(unlikely(!query_has_percentage_of_group && qm->status & RRDR_DIMENSION_HIDDEN)) {
279 + buffer_strcat(key, "__hidden_dimensions__");
280 + }
281 + else if(unlikely(group_by & RRDR_GROUP_BY_SELECTED)) {
282 + buffer_strcat(key, "selected");
283 + }
284 + else {
285 + if (group_by & RRDR_GROUP_BY_DIMENSION) {
286 + buffer_strcat(key, query_metric_name(qt, qm));
287 + }
288 +
289 + if (group_by & (RRDR_GROUP_BY_INSTANCE|RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)) {
290 + if (buffer_strlen(key) != 0)
291 + buffer_fast_strcat(key, ",", 1);
292 +
293 + if (group_by & RRDR_GROUP_BY_NODE)
294 + buffer_strcat(key, rrdinstance_acquired_id(qi->ria));
295 + else
296 + buffer_strcat(key, string2str(query_instance_id_fqdn(qi, qt->request.version)));
297 + }
298 +
299 + if (group_by & RRDR_GROUP_BY_LABEL) {
300 + RRDLABELS *labels = rrdinstance_acquired_labels(qi->ria);
301 + for (size_t l = 0; l < qt->group_by[group_by_id].used; l++) {
302 + if (buffer_strlen(key) != 0)
303 + buffer_fast_strcat(key, ",", 1);
304 + rrdlabels_get_value_to_buffer_or_unset(labels, key, qt->group_by[group_by_id].label_keys[l], "[unset]");
305 + }
306 + }
307 +
308 + if (group_by & RRDR_GROUP_BY_NODE) {
309 + if (buffer_strlen(key) != 0)
310 + buffer_fast_strcat(key, ",", 1);
311 +
312 + buffer_strcat(key, qn->rrdhost->machine_guid);
313 + }
314 +
315 + if (group_by & RRDR_GROUP_BY_CONTEXT) {
316 + if (buffer_strlen(key) != 0)
317 + buffer_fast_strcat(key, ",", 1);
318 +
319 + buffer_strcat(key, rrdcontext_acquired_id(qc->rca));
320 + }
321 +
322 + if (group_by & RRDR_GROUP_BY_UNITS) {
323 + if (buffer_strlen(key) != 0)
324 + buffer_fast_strcat(key, ",", 1);
325 +
326 + buffer_strcat(key, query_target_has_percentage_units(qt) ? "%" : rrdinstance_acquired_units(qi->ria));
327 + }
328 + }
329 +}
330 +
331 +static void query_group_by_make_dimension_name(BUFFER *key, RRDR_GROUP_BY group_by, size_t group_by_id, QUERY_TARGET *qt, QUERY_NODE *qn, QUERY_CONTEXT *qc, QUERY_INSTANCE *qi, QUERY_DIMENSION *qd __maybe_unused, QUERY_METRIC *qm, bool query_has_percentage_of_group) {
332 + buffer_flush(key);
333 + if(unlikely(!query_has_percentage_of_group && qm->status & RRDR_DIMENSION_HIDDEN)) {
334 + buffer_strcat(key, "__hidden_dimensions__");
335 + }
336 + else if(unlikely(group_by & RRDR_GROUP_BY_SELECTED)) {
337 + buffer_strcat(key, "selected");
338 + }
339 + else {
340 + if (group_by & RRDR_GROUP_BY_DIMENSION) {
341 + buffer_strcat(key, query_metric_name(qt, qm));
342 + }
343 +
344 + if (group_by & (RRDR_GROUP_BY_INSTANCE|RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)) {
345 + if (buffer_strlen(key) != 0)
346 + buffer_fast_strcat(key, ",", 1);
347 +
348 + if (group_by & RRDR_GROUP_BY_NODE)
349 + buffer_strcat(key, rrdinstance_acquired_name(qi->ria));
350 + else
351 + buffer_strcat(key, string2str(query_instance_name_fqdn(qi, qt->request.version)));
352 + }
353 +
354 + if (group_by & RRDR_GROUP_BY_LABEL) {
355 + RRDLABELS *labels = rrdinstance_acquired_labels(qi->ria);
356 + for (size_t l = 0; l < qt->group_by[group_by_id].used; l++) {
357 + if (buffer_strlen(key) != 0)
358 + buffer_fast_strcat(key, ",", 1);
359 + rrdlabels_get_value_to_buffer_or_unset(labels, key, qt->group_by[group_by_id].label_keys[l], "[unset]");
360 + }
361 + }
362 +
363 + if (group_by & RRDR_GROUP_BY_NODE) {
364 + if (buffer_strlen(key) != 0)
365 + buffer_fast_strcat(key, ",", 1);
366 +
367 + buffer_strcat(key, rrdhost_hostname(qn->rrdhost));
368 + }
369 +
370 + if (group_by & RRDR_GROUP_BY_CONTEXT) {
371 + if (buffer_strlen(key) != 0)
372 + buffer_fast_strcat(key, ",", 1);
373 +
374 + buffer_strcat(key, rrdcontext_acquired_id(qc->rca));
375 + }
376 +
377 + if (group_by & RRDR_GROUP_BY_UNITS) {
378 + if (buffer_strlen(key) != 0)
379 + buffer_fast_strcat(key, ",", 1);
380 +
381 + buffer_strcat(key, query_target_has_percentage_units(qt) ? "%" : rrdinstance_acquired_units(qi->ria));
382 + }
383 + }
384 +}
385 +
386 +struct rrdr_group_by_entry {
387 + size_t priority;
388 + size_t count;
389 + STRING *id;
390 + STRING *name;
391 + STRING *units;
392 + RRDR_DIMENSION_FLAGS od;
393 + DICTIONARY *dl;
394 +};
395 +
396 +RRDR *rrd2rrdr_group_by_initialize(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
397 + RRDR *r_tmp = NULL;
398 + RRDR_OPTIONS options = qt->window.options;
399 +
400 + if(qt->request.version < 2) {
401 + // v1 query
402 + RRDR *r = rrdr_create(owa, qt, qt->query.used, qt->window.points);
403 + if(unlikely(!r)) {
404 + internal_error(true, "QUERY: cannot create RRDR for %s, after=%ld, before=%ld, dimensions=%u, points=%zu",
405 + qt->id, qt->window.after, qt->window.before, qt->query.used, qt->window.points);
406 + return NULL;
407 + }
408 + r->group_by.r = NULL;
409 +
410 + for(size_t d = 0; d < qt->query.used ; d++) {
411 + QUERY_METRIC *qm = query_metric(qt, d);
412 + QUERY_DIMENSION *qd = query_dimension(qt, qm->link.query_dimension_id);
413 + r->di[d] = rrdmetric_acquired_id_dup(qd->rma);
414 + r->dn[d] = rrdmetric_acquired_name_dup(qd->rma);
415 + }
416 +
417 + rrd2rrdr_set_timestamps(r);
418 + return r;
419 + }
420 + // v2 query
421 +
422 + // parse all the group-by label keys
423 + for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
424 + if (qt->request.group_by[g].group_by & RRDR_GROUP_BY_LABEL &&
425 + qt->request.group_by[g].group_by_label && *qt->request.group_by[g].group_by_label)
426 + qt->group_by[g].used = quoted_strings_splitter_query_group_by_label(
427 + qt->request.group_by[g].group_by_label, qt->group_by[g].label_keys,
428 + GROUP_BY_MAX_LABEL_KEYS);
429 +
430 + if (!qt->group_by[g].used)
431 + qt->request.group_by[g].group_by &= ~RRDR_GROUP_BY_LABEL;
432 + }
433 +
434 + // make sure there are valid group-by methods
435 + for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
436 + if(!(qt->request.group_by[g].group_by & SUPPORTED_GROUP_BY_METHODS))
437 + qt->request.group_by[g].group_by = (g == 0) ? RRDR_GROUP_BY_DIMENSION : RRDR_GROUP_BY_NONE;
438 + }
439 +
440 + bool query_has_percentage_of_group = query_target_has_percentage_of_group(qt);
441 +
442 + // merge all group-by options to upper levels,
443 + // so that the top level has all the groupings of the inner levels,
444 + // and each subsequent level has all the groupings of its inner levels.
445 + for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES - 1 ;g++) {
446 + if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_NONE)
447 + continue;
448 +
449 + if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_SELECTED) {
450 + for (size_t r = g + 1; r < MAX_QUERY_GROUP_BY_PASSES; r++)
451 + qt->request.group_by[r].group_by = RRDR_GROUP_BY_NONE;
452 + }
453 + else {
454 + for (size_t r = g + 1; r < MAX_QUERY_GROUP_BY_PASSES; r++) {
455 + if (qt->request.group_by[r].group_by == RRDR_GROUP_BY_NONE)
456 + continue;
457 +
458 + if (qt->request.group_by[r].group_by != RRDR_GROUP_BY_SELECTED) {
459 + if(qt->request.group_by[r].group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
460 + qt->request.group_by[g].group_by |= RRDR_GROUP_BY_INSTANCE;
461 + else
462 + qt->request.group_by[g].group_by |= qt->request.group_by[r].group_by;
463 +
464 + if(qt->request.group_by[r].group_by & RRDR_GROUP_BY_LABEL) {
465 + for (size_t lr = 0; lr < qt->group_by[r].used; lr++) {
466 + bool found = false;
467 + for (size_t lg = 0; lg < qt->group_by[g].used; lg++) {
468 + if (strcmp(qt->group_by[g].label_keys[lg], qt->group_by[r].label_keys[lr]) == 0) {
469 + found = true;
470 + break;
471 + }
472 + }
473 +
474 + if (!found && qt->group_by[g].used < GROUP_BY_MAX_LABEL_KEYS * MAX_QUERY_GROUP_BY_PASSES)
475 + qt->group_by[g].label_keys[qt->group_by[g].used++] = qt->group_by[r].label_keys[lr];
476 + }
477 + }
478 + }
479 + }
480 + }
481 + }
482 +
483 + int added = 0;
484 + RRDR *first_r = NULL, *last_r = NULL;
485 + BUFFER *key = buffer_create(0, NULL);
486 + struct rrdr_group_by_entry *entries = onewayalloc_mallocz(owa, qt->query.used * sizeof(struct rrdr_group_by_entry));
487 + DICTIONARY *groups = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
488 + DICTIONARY *label_keys = NULL;
489 +
490 + for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
491 + RRDR_GROUP_BY group_by = qt->request.group_by[g].group_by;
492 + RRDR_GROUP_BY_FUNCTION aggregation_method = qt->request.group_by[g].aggregation;
493 +
494 + if(group_by == RRDR_GROUP_BY_NONE)
495 + break;
496 +
497 + memset(entries, 0, qt->query.used * sizeof(struct rrdr_group_by_entry));
498 + dictionary_flush(groups);
499 + added = 0;
500 +
501 + size_t hidden_dimensions = 0;
502 + bool final_grouping = (g == MAX_QUERY_GROUP_BY_PASSES - 1 || qt->request.group_by[g + 1].group_by == RRDR_GROUP_BY_NONE) ? true : false;
503 +
504 + if (final_grouping && (options & RRDR_OPTION_GROUP_BY_LABELS))
505 + label_keys = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
506 +
507 + QUERY_INSTANCE *last_qi = NULL;
508 + size_t priority = 0;
509 + time_t update_every_max = 0;
510 + for (size_t d = 0; d < qt->query.used; d++) {
511 + QUERY_METRIC *qm = query_metric(qt, d);
512 + QUERY_DIMENSION *qd = query_dimension(qt, qm->link.query_dimension_id);
513 + QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
514 + QUERY_CONTEXT *qc = query_context(qt, qm->link.query_context_id);
515 + QUERY_NODE *qn = query_node(qt, qm->link.query_node_id);
516 +
517 + if (qi != last_qi) {
518 + last_qi = qi;
519 +
520 + time_t update_every = rrdinstance_acquired_update_every(qi->ria);
521 + if (update_every > update_every_max)
522 + update_every_max = update_every;
523 + }
524 +
525 + priority = qd->priority;
526 +
527 + if(qm->status & RRDR_DIMENSION_HIDDEN)
528 + hidden_dimensions++;
529 +
530 + // --------------------------------------------------------------------
531 + // generate the group by key
532 +
533 + query_group_by_make_dimension_key(key, group_by, g, qt, qn, qc, qi, qd, qm, query_has_percentage_of_group);
534 +
535 + // lookup the key in the dictionary
536 +
537 + int pos = -1;
538 + int *set = dictionary_set(groups, buffer_tostring(key), &pos, sizeof(pos));
539 + if (*set == -1) {
540 + // the key just added to the dictionary
541 +
542 + *set = pos = added++;
543 +
544 + // ----------------------------------------------------------------
545 + // generate the dimension id
546 +
547 + query_group_by_make_dimension_id(key, group_by, g, qt, qn, qc, qi, qd, qm, query_has_percentage_of_group);
548 + entries[pos].id = string_strdupz(buffer_tostring(key));
549 +
550 + // ----------------------------------------------------------------
551 + // generate the dimension name
552 +
553 + query_group_by_make_dimension_name(key, group_by, g, qt, qn, qc, qi, qd, qm, query_has_percentage_of_group);
554 + entries[pos].name = string_strdupz(buffer_tostring(key));
555 +
556 + // add the rest of the info
557 + entries[pos].units = rrdinstance_acquired_units_dup(qi->ria);
558 + entries[pos].priority = priority;
559 +
560 + if (label_keys) {
561 + entries[pos].dl = dictionary_create_advanced(
562 + DICT_OPTION_SINGLE_THREADED | DICT_OPTION_FIXED_SIZE | DICT_OPTION_DONT_OVERWRITE_VALUE,
563 + NULL, sizeof(struct group_by_label_key));
564 + dictionary_register_insert_callback(entries[pos].dl, group_by_label_key_insert_cb, label_keys);
565 + dictionary_register_delete_callback(entries[pos].dl, group_by_label_key_delete_cb, label_keys);
566 + }
567 + } else {
568 + // the key found in the dictionary
569 + pos = *set;
570 + }
571 +
572 + entries[pos].count++;
573 +
574 + if (unlikely(priority < entries[pos].priority))
575 + entries[pos].priority = priority;
576 +
577 + if(g > 0)
578 + last_r->dgbs[qm->grouped_as.slot] = pos;
579 + else
580 + qm->grouped_as.first_slot = pos;
581 +
582 + qm->grouped_as.slot = pos;
583 + qm->grouped_as.id = entries[pos].id;
584 + qm->grouped_as.name = entries[pos].name;
585 + qm->grouped_as.units = entries[pos].units;
586 +
587 + // copy the dimension flags decided by the query target
588 + // we need this, because if a dimension is explicitly selected
589 + // the query target adds to it the non-zero flag
590 + qm->status |= RRDR_DIMENSION_GROUPED;
591 +
592 + if(query_has_percentage_of_group)
593 + // when the query has percentage of group
594 + // there will be no hidden dimensions in the final query,
595 + // so we have to remove the hidden flag from all dimensions
596 + entries[pos].od |= qm->status & ~RRDR_DIMENSION_HIDDEN;
597 + else
598 + entries[pos].od |= qm->status;
599 +
600 + if (entries[pos].dl)
601 + rrdlabels_walkthrough_read(rrdinstance_acquired_labels(qi->ria),
602 + rrdlabels_traversal_cb_to_group_by_label_key, entries[pos].dl);
603 + }
604 +
605 + RRDR *r = rrdr_create(owa, qt, added, qt->window.points);
606 + if (!r) {
607 + internal_error(true,
608 + "QUERY: cannot create group by RRDR for %s, after=%ld, before=%ld, dimensions=%d, points=%zu",
609 + qt->id, qt->window.after, qt->window.before, added, qt->window.points);
610 + goto cleanup;
611 + }
612 + // prevent double free at cleanup in case of error
613 + added = 0;
614 +
615 + // link this RRDR
616 + if(!last_r)
617 + first_r = last_r = r;
618 + else
619 + last_r->group_by.r = r;
620 +
621 + last_r = r;
622 +
623 + rrd2rrdr_set_timestamps(r);
624 +
625 + if(r->d) {
626 + r->dp = onewayalloc_callocz(owa, r->d, sizeof(*r->dp));
627 + r->dview = onewayalloc_callocz(owa, r->d, sizeof(*r->dview));
628 + r->dgbc = onewayalloc_callocz(owa, r->d, sizeof(*r->dgbc));
629 + r->dqp = onewayalloc_callocz(owa, r->d, sizeof(STORAGE_POINT));
630 +
631 + if(!final_grouping)
632 + // this is where we are going to store the slot in the next RRDR
633 + // that we are going to group by the dimension of this RRDR
634 + r->dgbs = onewayalloc_callocz(owa, r->d, sizeof(*r->dgbs));
635 +
636 + if (label_keys) {
637 + r->dl = onewayalloc_callocz(owa, r->d, sizeof(DICTIONARY *));
638 + r->label_keys = label_keys;
639 + label_keys = NULL;
640 + }
641 +
642 + if(r->n) {
643 + r->gbc = onewayalloc_callocz(owa, r->n * r->d, sizeof(*r->gbc));
644 +
645 + if(hidden_dimensions && ((group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE) || (aggregation_method == RRDR_GROUP_BY_FUNCTION_PERCENTAGE)))
646 + // this is where we are going to group the hidden dimensions
647 + r->vh = onewayalloc_mallocz(owa, r->n * r->d * sizeof(*r->vh));
648 + }
649 + }
650 +
651 + // zero r (dimension options, names, and ids)
652 + // this is required, because group-by may lead to empty dimensions
653 + for (size_t d = 0; d < r->d; d++) {
654 + r->di[d] = entries[d].id;
655 + r->dn[d] = entries[d].name;
656 +
657 + r->od[d] = entries[d].od;
658 + r->du[d] = entries[d].units;
659 + r->dp[d] = entries[d].priority;
660 + r->dgbc[d] = entries[d].count;
661 +
662 + if (r->dl)
663 + r->dl[d] = entries[d].dl;
664 + }
665 +
666 + // initialize partial trimming
667 + r->partial_data_trimming.max_update_every = update_every_max * 2;
668 + r->partial_data_trimming.expected_after =
669 + (!query_target_aggregatable(qt) &&
670 + qt->window.before >= qt->window.now - r->partial_data_trimming.max_update_every) ?
671 + qt->window.before - r->partial_data_trimming.max_update_every :
672 + qt->window.before;
673 + r->partial_data_trimming.trimmed_after = qt->window.before;
674 +
675 + // make all values empty
676 + if(r->n && r->d) {
677 + for (size_t i = 0; i != r->n; i++) {
678 + NETDATA_DOUBLE *cn = &r->v[i * r->d];
679 + RRDR_VALUE_FLAGS *co = &r->o[i * r->d];
680 + NETDATA_DOUBLE *ar = &r->ar[i * r->d];
681 + NETDATA_DOUBLE *vh = r->vh ? &r->vh[i * r->d] : NULL;
682 +
683 + for (size_t d = 0; d < r->d; d++) {
684 + cn[d] = NAN;
685 + ar[d] = 0.0;
686 + co[d] = RRDR_VALUE_EMPTY;
687 +
688 + if (vh)
689 + vh[d] = NAN;
690 + }
691 + }
692 + }
693 + }
694 +
695 + if(!first_r || !last_r)
696 + goto cleanup;
697 +
698 + r_tmp = rrdr_create(owa, qt, 1, qt->window.points);
699 + if (!r_tmp) {
700 + internal_error(true,
701 + "QUERY: cannot create group by temporary RRDR for %s, after=%ld, before=%ld, dimensions=%d, points=%zu",
702 + qt->id, qt->window.after, qt->window.before, 1, qt->window.points);
703 + goto cleanup;
704 + }
705 + rrd2rrdr_set_timestamps(r_tmp);
706 + r_tmp->group_by.r = first_r;
707 +
708 +cleanup:
709 + if(!first_r || !last_r || !r_tmp) {
710 + if(r_tmp) {
711 + r_tmp->group_by.r = NULL;
712 + rrdr_free(owa, r_tmp);
713 + }
714 +
715 + if(first_r) {
716 + RRDR *r = first_r;
717 + while (r) {
718 + r_tmp = r->group_by.r;
719 + r->group_by.r = NULL;
720 + rrdr_free(owa, r);
721 + r = r_tmp;
722 + }
723 + }
724 +
725 + if(entries && added) {
726 + for (int d = 0; d < added; d++) {
727 + string_freez(entries[d].id);
728 + string_freez(entries[d].name);
729 + string_freez(entries[d].units);
730 + dictionary_destroy(entries[d].dl);
731 + }
732 + }
733 + dictionary_destroy(label_keys);
734 +
735 + first_r = last_r = r_tmp = NULL;
736 + }
737 +
738 + buffer_free(key);
739 + onewayalloc_freez(owa, entries);
740 + dictionary_destroy(groups);
741 +
742 + return r_tmp;
743 +}
744 +
745 +void rrd2rrdr_group_by_add_metric(RRDR *r_dst, size_t d_dst, RRDR *r_tmp, size_t d_tmp,
746 + RRDR_GROUP_BY_FUNCTION group_by_aggregate_function,
747 + STORAGE_POINT *query_points, size_t pass __maybe_unused) {
748 + if(!r_tmp || r_dst == r_tmp || !(r_tmp->od[d_tmp] & RRDR_DIMENSION_QUERIED))
749 + return;
750 +
751 + internal_fatal(r_dst->n != r_tmp->n, "QUERY: group-by source and destination do not have the same number of rows");
752 + internal_fatal(d_dst >= r_dst->d, "QUERY: group-by destination dimension number exceeds destination RRDR size");
753 + internal_fatal(d_tmp >= r_tmp->d, "QUERY: group-by source dimension number exceeds source RRDR size");
754 + internal_fatal(!r_dst->dqp, "QUERY: group-by destination is not properly prepared (missing dqp array)");
755 + internal_fatal(!r_dst->gbc, "QUERY: group-by destination is not properly prepared (missing gbc array)");
756 +
757 + bool hidden_dimension_on_percentage_of_group = (r_tmp->od[d_tmp] & RRDR_DIMENSION_HIDDEN) && r_dst->vh;
758 +
759 + if(!hidden_dimension_on_percentage_of_group) {
760 + r_dst->od[d_dst] |= r_tmp->od[d_tmp];
761 + storage_point_merge_to(r_dst->dqp[d_dst], *query_points);
762 + }
763 +
764 + // do the group_by
765 + for(size_t i = 0; i != rrdr_rows(r_tmp) ; i++) {
766 +
767 + size_t idx_tmp = i * r_tmp->d + d_tmp;
768 + NETDATA_DOUBLE n_tmp = r_tmp->v[ idx_tmp ];
769 + RRDR_VALUE_FLAGS o_tmp = r_tmp->o[ idx_tmp ];
770 + NETDATA_DOUBLE ar_tmp = r_tmp->ar[ idx_tmp ];
771 +
772 + if(o_tmp & RRDR_VALUE_EMPTY)
773 + continue;
774 +
775 + size_t idx_dst = i * r_dst->d + d_dst;
776 + NETDATA_DOUBLE *cn = (hidden_dimension_on_percentage_of_group) ? &r_dst->vh[ idx_dst ] : &r_dst->v[ idx_dst ];
777 + RRDR_VALUE_FLAGS *co = &r_dst->o[ idx_dst ];
778 + NETDATA_DOUBLE *ar = &r_dst->ar[ idx_dst ];
779 + uint32_t *gbc = &r_dst->gbc[ idx_dst ];
780 +
781 + switch(group_by_aggregate_function) {
782 + default:
783 + case RRDR_GROUP_BY_FUNCTION_AVERAGE:
784 + case RRDR_GROUP_BY_FUNCTION_SUM:
785 + case RRDR_GROUP_BY_FUNCTION_PERCENTAGE:
786 + if(isnan(*cn))
787 + *cn = n_tmp;
788 + else
789 + *cn += n_tmp;
790 + break;
791 +
792 + case RRDR_GROUP_BY_FUNCTION_MIN:
793 + if(isnan(*cn) || n_tmp < *cn)
794 + *cn = n_tmp;
795 + break;
796 +
797 + case RRDR_GROUP_BY_FUNCTION_MAX:
798 + if(isnan(*cn) || n_tmp > *cn)
799 + *cn = n_tmp;
800 + break;
801 +
802 + case RRDR_GROUP_BY_FUNCTION_EXTREMES:
803 + // For extremes, we need to keep track of the value with the maximum absolute value
804 + if(isnan(*cn) || fabsndd(n_tmp) > fabsndd(*cn))
805 + *cn = n_tmp;
806 + break;
807 + }
808 +
809 + if(!hidden_dimension_on_percentage_of_group) {
810 + *co &= ~RRDR_VALUE_EMPTY;
811 + *co |= (o_tmp & (RRDR_VALUE_RESET | RRDR_VALUE_PARTIAL));
812 + *ar += ar_tmp;
813 + (*gbc)++;
814 + }
815 + }
816 +}
817 +
818 +void rrdr2rrdr_group_by_partial_trimming(RRDR *r) {
819 + time_t trimmable_after = r->partial_data_trimming.expected_after;
820 +
821 + // find the point just before the trimmable ones
822 + ssize_t i = (ssize_t)r->n - 1;
823 + for( ; i >= 0 ;i--) {
824 + if (r->t[i] < trimmable_after)
825 + break;
826 + }
827 +
828 + if(unlikely(i < 0))
829 + return;
830 +
831 + // internal_error(true, "Found trimmable index %zd (from 0 to %zu)", i, r->n - 1);
832 +
833 + size_t last_row_gbc = 0;
834 + for (; i < (ssize_t)r->n; i++) {
835 + size_t row_gbc = 0;
836 + for (size_t d = 0; d < r->d; d++) {
837 + if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
838 + continue;
839 +
840 + row_gbc += r->gbc[ i * r->d + d ];
841 + }
842 +
843 + // internal_error(true, "GBC of index %zd is %zu", i, row_gbc);
844 +
845 + if (unlikely(r->t[i] >= trimmable_after && (row_gbc < last_row_gbc || !row_gbc))) {
846 + // discard the rest of the points
847 + // internal_error(true, "Discarding points %zd to %zu", i, r->n - 1);
848 + r->partial_data_trimming.trimmed_after = r->t[i];
849 + r->rows = i;
850 + break;
851 + }
852 + else
853 + last_row_gbc = row_gbc;
854 + }
855 +}
856 +
857 +void rrdr2rrdr_group_by_calculate_percentage_of_group(RRDR *r) {
858 + if(!r->vh)
859 + return;
860 +
861 + if(query_target_aggregatable(r->internal.qt) && query_has_group_by_aggregation_percentage(r->internal.qt))
862 + return;
863 +
864 + for(size_t i = 0; i < r->n ;i++) {
865 + NETDATA_DOUBLE *cn = &r->v[ i * r->d ];
866 + NETDATA_DOUBLE *ch = &r->vh[ i * r->d ];
867 +
868 + for(size_t d = 0; d < r->d ;d++) {
869 + NETDATA_DOUBLE n = cn[d];
870 + NETDATA_DOUBLE h = ch[d];
871 +
872 + if(isnan(n))
873 + cn[d] = 0.0;
874 +
875 + else if(isnan(h))
876 + cn[d] = 100.0;
877 +
878 + else
879 + cn[d] = n * 100.0 / (n + h);
880 + }
881 + }
882 +}
883 +
884 +
885 +void rrd2rrdr_convert_values_to_percentage_of_total(RRDR *r) {
886 + if(!(r->internal.qt->window.options & RRDR_OPTION_PERCENTAGE) || query_target_aggregatable(r->internal.qt))
887 + return;
888 +
889 + size_t global_min_max_values = 0;
890 + NETDATA_DOUBLE global_min = NAN, global_max = NAN;
891 +
892 + for(size_t i = 0; i != r->n ;i++) {
893 + NETDATA_DOUBLE *cn = &r->v[ i * r->d ];
894 + RRDR_VALUE_FLAGS *co = &r->o[ i * r->d ];
895 +
896 + NETDATA_DOUBLE total = 0;
897 + for (size_t d = 0; d < r->d; d++) {
898 + if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
899 + continue;
900 +
901 + if(co[d] & RRDR_VALUE_EMPTY)
902 + continue;
903 +
904 + total += cn[d];
905 + }
906 +
907 + if(total == 0.0)
908 + total = 1.0;
909 +
910 + for (size_t d = 0; d < r->d; d++) {
911 + if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
912 + continue;
913 +
914 + if(co[d] & RRDR_VALUE_EMPTY)
915 + continue;
916 +
917 + NETDATA_DOUBLE n = cn[d];
918 + n = cn[d] = n * 100.0 / total;
919 +
920 + if(unlikely(!global_min_max_values++))
921 + global_min = global_max = n;
922 + else {
923 + if(n < global_min)
924 + global_min = n;
925 + if(n > global_max)
926 + global_max = n;
927 + }
928 + }
929 + }
930 +
931 + r->view.min = global_min;
932 + r->view.max = global_max;
933 +
934 + if(!r->dview)
935 + // v1 query
936 + return;
937 +
938 + // v2 query
939 +
940 + for (size_t d = 0; d < r->d; d++) {
941 + if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
942 + continue;
943 +
944 + size_t count = 0;
945 + NETDATA_DOUBLE min = 0.0, max = 0.0, sum = 0.0, ars = 0.0;
946 + for(size_t i = 0; i != r->rows ;i++) { // we use r->rows to respect trimming
947 + size_t idx = i * r->d + d;
948 +
949 + RRDR_VALUE_FLAGS o = r->o[ idx ];
950 +
951 + if (o & RRDR_VALUE_EMPTY)
952 + continue;
953 +
954 + NETDATA_DOUBLE ar = r->ar[ idx ];
955 + ars += ar;
956 +
957 + NETDATA_DOUBLE n = r->v[ idx ];
958 + sum += n;
959 +
960 + if(!count++)
961 + min = max = n;
962 + else {
963 + if(n < min)
964 + min = n;
965 + if(n > max)
966 + max = n;
967 + }
968 + }
969 +
970 + r->dview[d] = (STORAGE_POINT) {
971 + .sum = sum,
972 + .count = count,
973 + .min = min,
974 + .max = max,
975 + .anomaly_count = (size_t)(ars * (NETDATA_DOUBLE)count),
976 + };
977 + }
978 +}
979 +
980 +RRDR *rrd2rrdr_group_by_finalize(RRDR *r_tmp) {
981 + QUERY_TARGET *qt = r_tmp->internal.qt;
982 +
983 + if(!r_tmp->group_by.r) {
984 + // v1 query
985 + rrd2rrdr_convert_values_to_percentage_of_total(r_tmp);
986 + return r_tmp;
987 + }
988 + // v2 query
989 +
990 + // do the additional passes on RRDRs
991 + RRDR *last_r = r_tmp->group_by.r;
992 + rrdr2rrdr_group_by_calculate_percentage_of_group(last_r);
993 +
994 + RRDR *r = last_r->group_by.r;
995 + size_t pass = 0;
996 + while(r) {
997 + pass++;
998 + for(size_t d = 0; d < last_r->d ;d++) {
999 + rrd2rrdr_group_by_add_metric(r, last_r->dgbs[d], last_r, d,
1000 + qt->request.group_by[pass].aggregation,
1001 + &last_r->dqp[d], pass);
1002 + }
1003 + rrdr2rrdr_group_by_calculate_percentage_of_group(r);
1004 +
1005 + last_r = r;
1006 + r = last_r->group_by.r;
1007 + }
1008 +
1009 + // free all RRDRs except the last one
1010 + r = r_tmp;
1011 + while(r != last_r) {
1012 + r_tmp = r->group_by.r;
1013 + r->group_by.r = NULL;
1014 + rrdr_free(r->internal.owa, r);
1015 + r = r_tmp;
1016 + }
1017 + r = last_r;
1018 +
1019 + // find the final aggregation
1020 + RRDR_GROUP_BY_FUNCTION aggregation = qt->request.group_by[0].aggregation;
1021 + for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++)
1022 + if(qt->request.group_by[g].group_by != RRDR_GROUP_BY_NONE)
1023 + aggregation = qt->request.group_by[g].aggregation;
1024 +
1025 + if(!query_target_aggregatable(qt) && r->partial_data_trimming.expected_after < qt->window.before)
1026 + rrdr2rrdr_group_by_partial_trimming(r);
1027 +
1028 + // apply averaging, remove RRDR_VALUE_EMPTY, find the non-zero dimensions, min and max
1029 + size_t global_min_max_values = 0;
1030 + size_t dimensions_nonzero = 0;
1031 + NETDATA_DOUBLE global_min = NAN, global_max = NAN;
1032 + for (size_t d = 0; d < r->d; d++) {
1033 + if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
1034 + continue;
1035 +
1036 + size_t points_nonzero = 0;
1037 + NETDATA_DOUBLE min = 0, max = 0, sum = 0, ars = 0;
1038 + size_t count = 0;
1039 +
1040 + for(size_t i = 0; i != r->n ;i++) {
1041 + size_t idx = i * r->d + d;
1042 +
1043 + NETDATA_DOUBLE *cn = &r->v[ idx ];
1044 + RRDR_VALUE_FLAGS *co = &r->o[ idx ];
1045 + NETDATA_DOUBLE *ar = &r->ar[ idx ];
1046 + uint32_t gbc = r->gbc[ idx ];
1047 +
1048 + if(likely(gbc)) {
1049 + *co &= ~RRDR_VALUE_EMPTY;
1050 +
1051 + if(gbc != r->dgbc[d])
1052 + *co |= RRDR_VALUE_PARTIAL;
1053 +
1054 + NETDATA_DOUBLE n;
1055 +
1056 + sum += *cn;
1057 + ars += *ar;
1058 +
1059 + if(aggregation == RRDR_GROUP_BY_FUNCTION_AVERAGE && !query_target_aggregatable(qt))
1060 + n = (*cn /= gbc);
1061 + else
1062 + n = *cn;
1063 +
1064 + if(!query_target_aggregatable(qt))
1065 + *ar /= gbc;
1066 +
1067 + if(islessgreater(n, 0.0))
1068 + points_nonzero++;
1069 +
1070 + if(unlikely(!count))
1071 + min = max = n;
1072 + else {
1073 + if(n < min)
1074 + min = n;
1075 +
1076 + if(n > max)
1077 + max = n;
1078 + }
1079 +
1080 + if(unlikely(!global_min_max_values++))
1081 + global_min = global_max = n;
1082 + else {
1083 + if(n < global_min)
1084 + global_min = n;
1085 +
1086 + if(n > global_max)
1087 + global_max = n;
1088 + }
1089 +
1090 + count += gbc;
1091 + }
1092 + }
1093 +
1094 + if(points_nonzero) {
1095 + r->od[d] |= RRDR_DIMENSION_NONZERO;
1096 + dimensions_nonzero++;
1097 + }
1098 +
1099 + r->dview[d] = (STORAGE_POINT) {
1100 + .sum = sum,
1101 + .count = count,
1102 + .min = min,
1103 + .max = max,
1104 + .anomaly_count = (size_t)(ars * RRDR_DVIEW_ANOMALY_COUNT_MULTIPLIER / 100.0),
1105 + };
1106 + }
1107 +
1108 + r->view.min = global_min;
1109 + r->view.max = global_max;
1110 +
1111 + if(!dimensions_nonzero && (qt->window.options & RRDR_OPTION_NONZERO)) {
1112 + // all dimensions are zero
1113 + // remove the nonzero option
1114 + qt->window.options &= ~RRDR_OPTION_NONZERO;
1115 + }
1116 +
1117 + rrd2rrdr_convert_values_to_percentage_of_total(r);
1118 +
1119 + // update query instance counts in query host and query context
1120 + {
1121 + size_t h = 0, c = 0, i = 0;
1122 + for(; h < qt->nodes.used ; h++) {
1123 + QUERY_NODE *qn = &qt->nodes.array[h];
1124 +
1125 + for(; c < qt->contexts.used ;c++) {
1126 + QUERY_CONTEXT *qc = &qt->contexts.array[c];
1127 +
1128 + if(!rrdcontext_acquired_belongs_to_host(qc->rca, qn->rrdhost))
1129 + break;
1130 +
1131 + for(; i < qt->instances.used ;i++) {
1132 + QUERY_INSTANCE *qi = &qt->instances.array[i];
1133 +
1134 + if(!rrdinstance_acquired_belongs_to_context(qi->ria, qc->rca))
1135 + break;
1136 +
1137 + if(qi->metrics.queried) {
1138 + qc->instances.queried++;
1139 + qn->instances.queried++;
1140 + }
1141 + else if(qi->metrics.failed) {
1142 + qc->instances.failed++;
1143 + qn->instances.failed++;
1144 + }
1145 + }
1146 + }
1147 + }
1148 + }
1149 +
1150 + return r;
1151 +}
1152 +
1153 +static int compare_contributions(const void *a, const void *b) {
1154 + const struct { size_t dim_idx; NETDATA_DOUBLE contribution; } *da = a;
1155 + const struct { size_t dim_idx; NETDATA_DOUBLE contribution; } *db = b;
1156 +
1157 + if (da->contribution > db->contribution) return -1;
1158 + if (da->contribution < db->contribution) return 1;
1159 + return 0;
1160 +}
1161 +
1162 +RRDR *rrd2rrdr_cardinality_limit(RRDR *r) {
1163 + QUERY_TARGET *qt = r->internal.qt;
1164 +
1165 + if(!qt || qt->request.cardinality_limit == 0 || r->d <= qt->request.cardinality_limit)
1166 + return r;
1167 +
1168 + ONEWAYALLOC *owa = r->internal.owa;
1169 +
1170 + // Calculate contribution of each dimension using dview statistics (sum of values)
1171 + NETDATA_DOUBLE *contributions = onewayalloc_mallocz(owa, r->d * sizeof(NETDATA_DOUBLE));
1172 +
1173 + // Count queried dimensions and get their contributions from dview
1174 + size_t queried_count = 0;
1175 + for (size_t d = 0; d < r->d; d++) {
1176 + contributions[d] = 0.0;
1177 +
1178 + if (!(r->od[d] & RRDR_DIMENSION_QUERIED))
1179 + continue;
1180 +
1181 + queried_count++;
1182 +
1183 + // Use the sum from dview if available, otherwise fall back to manual calculation
1184 + if(r->dview && !isnan(r->dview[d].sum)) {
1185 + contributions[d] = fabsndd(r->dview[d].sum);
1186 + } else {
1187 + // Fallback: calculate manually from values
1188 + for(size_t i = 0; i < r->rows; i++) {
1189 + size_t idx = i * r->d + d;
1190 +
1191 + if(r->o[idx] & RRDR_VALUE_EMPTY)
1192 + continue;
1193 +
1194 + NETDATA_DOUBLE value = r->v[idx];
1195 + if(!isnan(value))
1196 + contributions[d] += fabsndd(value);
1197 + }
1198 + }
1199 + }
1200 +
1201 + // If we don't need to reduce, return original
1202 + if(queried_count <= qt->request.cardinality_limit) {
1203 + onewayalloc_freez(owa, contributions);
1204 + return r;
1205 + }
1206 +
1207 + // Create array of dimension indices sorted by contribution (descending)
1208 + struct {
1209 + size_t dim_idx;
1210 + NETDATA_DOUBLE contribution;
1211 + } *sorted_dims = onewayalloc_mallocz(owa, queried_count * sizeof(*sorted_dims));
1212 +
1213 + size_t sorted_idx = 0;
1214 + for (size_t d = 0; d < r->d; d++) {
1215 + if (r->od[d] & RRDR_DIMENSION_QUERIED) {
1216 + sorted_dims[sorted_idx].dim_idx = d;
1217 + sorted_dims[sorted_idx].contribution = contributions[d];
1218 + sorted_idx++;
1219 + }
1220 + }
1221 +
1222 + // Sort by contribution (descending)
1223 + qsort(sorted_dims, queried_count, sizeof(*sorted_dims), compare_contributions);
1224 +
1225 + // Create new RRDR with limited dimensions
1226 + size_t new_d = qt->request.cardinality_limit;
1227 + size_t remaining_count = queried_count - (qt->request.cardinality_limit - 1);
1228 + if(remaining_count > 0)
1229 + new_d = qt->request.cardinality_limit; // Keep one slot for "remaining N dimensions"
1230 + else
1231 + new_d = queried_count; // No remaining dimensions needed
1232 +
1233 + RRDR *new_r = rrdr_create(owa, qt, new_d, r->n);
1234 + if (!new_r) {
1235 + internal_error(true, "QUERY: cannot create cardinality limited RRDR");
1236 + onewayalloc_freez(owa, contributions);
1237 + onewayalloc_freez(owa, sorted_dims);
1238 + return r;
1239 + }
1240 +
1241 + // Copy basic metadata from original RRDR
1242 + new_r->view = r->view;
1243 + new_r->time_grouping = r->time_grouping;
1244 + new_r->partial_data_trimming = r->partial_data_trimming;
1245 + new_r->rows = r->rows;
1246 +
1247 + // Copy timestamps
1248 + memcpy(new_r->t, r->t, r->n * sizeof(time_t));
1249 +
1250 + // Setup arrays for new RRDR
1251 + if(new_r->d) {
1252 + new_r->dp = onewayalloc_callocz(owa, new_r->d, sizeof(*new_r->dp));
1253 + new_r->dview = onewayalloc_callocz(owa, new_r->d, sizeof(*new_r->dview));
1254 +
1255 + if(new_r->n) {
1256 + // Initialize all values as empty
1257 + for (size_t i = 0; i < new_r->n; i++) {
1258 + for (size_t d = 0; d < new_r->d; d++) {
1259 + size_t idx = i * new_r->d + d;
1260 + new_r->v[idx] = NAN;
1261 + new_r->ar[idx] = 0.0;
1262 + new_r->o[idx] = RRDR_VALUE_EMPTY;
1263 + }
1264 + }
1265 + }
1266 + }
1267 +
1268 + // Copy top dimensions
1269 + size_t kept_dimensions = (remaining_count > 0) ? qt->request.cardinality_limit - 1 : queried_count;
1270 +
1271 + for (size_t i = 0; i < kept_dimensions; i++) {
1272 + size_t src_d = sorted_dims[i].dim_idx;
1273 +
1274 + // Copy metadata
1275 + new_r->di[i] = string_dup(r->di[src_d]);
1276 + new_r->dn[i] = string_dup(r->dn[src_d]);
1277 + new_r->od[i] = r->od[src_d];
1278 + new_r->du[i] = string_dup(r->du[src_d]);
1279 + new_r->dp[i] = r->dp[src_d];
1280 +
1281 + // Copy data
1282 + for (size_t row = 0; row < r->rows; row++) {
1283 + size_t src_idx = row * r->d + src_d;
1284 + size_t dst_idx = row * new_r->d + i;
1285 +
1286 + new_r->v[dst_idx] = r->v[src_idx];
1287 + new_r->ar[dst_idx] = r->ar[src_idx];
1288 + new_r->o[dst_idx] = r->o[src_idx];
1289 + }
1290 +
1291 + // Copy dview stats
1292 + if(r->dview)
1293 + new_r->dview[i] = r->dview[src_d];
1294 + }
1295 +
1296 + // Create "remaining N dimensions" if needed
1297 + if (remaining_count > 0) {
1298 + size_t remaining_idx = kept_dimensions;
1299 +
1300 + char remaining_name[256];
1301 + snprintfz(remaining_name, sizeof(remaining_name), "remaining %zu dimension%s",
1302 + remaining_count, remaining_count == 1 ? "" : "s");
1303 +
1304 + new_r->di[remaining_idx] = string_strdupz(remaining_name);
1305 + new_r->dn[remaining_idx] = string_strdupz(remaining_name);
1306 + new_r->od[remaining_idx] = RRDR_DIMENSION_QUERIED | RRDR_DIMENSION_NONZERO;
1307 +
1308 + // Use the units from the first remaining dimension
1309 + if(kept_dimensions < queried_count) {
1310 + size_t first_remaining_d = sorted_dims[kept_dimensions].dim_idx;
1311 + new_r->du[remaining_idx] = string_dup(r->du[first_remaining_d]);
1312 + new_r->dp[remaining_idx] = r->dp[first_remaining_d];
1313 + }
1314 +
1315 + // Aggregate remaining dimensions
1316 + NETDATA_DOUBLE sum = 0.0, min = NAN, max = NAN, ars = 0.0;
1317 + size_t count = 0;
1318 +
1319 + for (size_t row = 0; row < r->rows; row++) {
1320 + size_t dst_idx = row * new_r->d + remaining_idx;
1321 + NETDATA_DOUBLE aggregated_value = 0.0;
1322 + NETDATA_DOUBLE aggregated_ar = 0.0;
1323 + RRDR_VALUE_FLAGS aggregated_flags = RRDR_VALUE_NOTHING;
1324 + bool has_values = false;
1325 +
1326 + for (size_t i = kept_dimensions; i < queried_count; i++) {
1327 + size_t src_d = sorted_dims[i].dim_idx;
1328 + size_t src_idx = row * r->d + src_d;
1329 +
1330 + if(!(r->o[src_idx] & RRDR_VALUE_EMPTY)) {
1331 + NETDATA_DOUBLE value = r->v[src_idx];
1332 + if(!isnan(value)) {
1333 + aggregated_value += value;
1334 + aggregated_ar += r->ar[src_idx];
1335 + aggregated_flags |= (r->o[src_idx] & (RRDR_VALUE_RESET | RRDR_VALUE_PARTIAL));
1336 + has_values = true;
1337 + }
1338 + }
1339 + }
1340 +
1341 + if(has_values) {
1342 + new_r->v[dst_idx] = aggregated_value;
1343 + new_r->ar[dst_idx] = aggregated_ar;
1344 + new_r->o[dst_idx] = aggregated_flags & ~RRDR_VALUE_EMPTY;
1345 +
1346 + // Update statistics for dview
1347 + sum += aggregated_value;
1348 + ars += aggregated_ar;
1349 + if(count == 0) {
1350 + min = max = aggregated_value;
1351 + } else {
1352 + if(aggregated_value < min) min = aggregated_value;
1353 + if(aggregated_value > max) max = aggregated_value;
1354 + }
1355 + count++;
1356 + } else {
1357 + new_r->v[dst_idx] = NAN;
1358 + new_r->ar[dst_idx] = 0.0;
1359 + new_r->o[dst_idx] = RRDR_VALUE_EMPTY;
1360 + }
1361 + }
1362 +
1363 + // Set dview for remaining dimension
1364 + if(new_r->dview) {
1365 + new_r->dview[remaining_idx] = (STORAGE_POINT) {
1366 + .sum = sum,
1367 + .count = count,
1368 + .min = min,
1369 + .max = max,
1370 + .anomaly_count = (size_t)(ars * RRDR_DVIEW_ANOMALY_COUNT_MULTIPLIER / 100.0),
1371 + };
1372 + }
1373 + }
1374 +
1375 + // Cleanup
1376 + onewayalloc_freez(owa, contributions);
1377 + onewayalloc_freez(owa, sorted_dims);
1378 +
1379 + // Free the original RRDR
1380 + rrdr_free(owa, r);
1381 +
1382 + return new_r;
1383 +}
src/web/api/queries/query-group-over-time.c new
+837
@@ -0,0 +1,837 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "query-internal.h"
4 +
5 +#include "average/average.h"
6 +#include "countif/countif.h"
7 +#include "incremental_sum/incremental_sum.h"
8 +#include "max/max.h"
9 +#include "median/median.h"
10 +#include "min/min.h"
11 +#include "sum/sum.h"
12 +#include "stddev/stddev.h"
13 +#include "ses/ses.h"
14 +#include "des/des.h"
15 +#include "percentile/percentile.h"
16 +#include "trimmed_mean/trimmed_mean.h"
17 +#include "extremes/extremes.h"
18 +
19 +// ----------------------------------------------------------------------------
20 +
21 +static struct {
22 + const char *name;
23 + uint32_t hash;
24 + RRDR_TIME_GROUPING value;
25 + RRDR_TIME_GROUPING add_flush;
26 +
27 + // One time initialization for the module.
28 + // This is called once, when netdata starts.
29 + void (*init)(void);
30 +
31 + // Allocate all required structures for a query.
32 + // This is called once for each netdata query.
33 + void (*create)(struct rrdresult *r, const char *options);
34 +
35 + // Cleanup collected values, but don't destroy the structures.
36 + // This is called when the query engine switches dimensions,
37 + // as part of the same query (so same chart, switching metric).
38 + void (*reset)(struct rrdresult *r);
39 +
40 + // Free all resources allocated for the query.
41 + void (*free)(struct rrdresult *r);
42 +
43 + // Add a single value into the calculation.
44 + // The module may decide to cache it, or use it in the fly.
45 + void (*add)(struct rrdresult *r, NETDATA_DOUBLE value);
46 +
47 + // Generate a single result for the values added so far.
48 + // More values and points may be requested later.
49 + // It is up to the module to reset its internal structures
50 + // when flushing it (so for a few modules it may be better to
51 + // continue after a flush as if nothing changed, for others a
52 + // cleanup of the internal structures may be required).
53 + NETDATA_DOUBLE (*flush)(struct rrdresult *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr);
54 +
55 + TIER_QUERY_FETCH tier_query_fetch;
56 +} api_v1_data_groups[] = {
57 + {.name = "average",
58 + .hash = 0,
59 + .value = RRDR_GROUPING_AVERAGE,
60 + .add_flush = RRDR_GROUPING_AVERAGE,
61 + .init = NULL,
62 + .create= tg_average_create,
63 + .reset = tg_average_reset,
64 + .free = tg_average_free,
65 + .add = tg_average_add,
66 + .flush = tg_average_flush,
67 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
68 + },
69 + {.name = "avg", // alias on 'average'
70 + .hash = 0,
71 + .value = RRDR_GROUPING_AVERAGE,
72 + .add_flush = RRDR_GROUPING_AVERAGE,
73 + .init = NULL,
74 + .create= tg_average_create,
75 + .reset = tg_average_reset,
76 + .free = tg_average_free,
77 + .add = tg_average_add,
78 + .flush = tg_average_flush,
79 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
80 + },
81 + {.name = "mean", // alias on 'average'
82 + .hash = 0,
83 + .value = RRDR_GROUPING_AVERAGE,
84 + .add_flush = RRDR_GROUPING_AVERAGE,
85 + .init = NULL,
86 + .create= tg_average_create,
87 + .reset = tg_average_reset,
88 + .free = tg_average_free,
89 + .add = tg_average_add,
90 + .flush = tg_average_flush,
91 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
92 + },
93 + {.name = "trimmed-mean1",
94 + .hash = 0,
95 + .value = RRDR_GROUPING_TRIMMED_MEAN1,
96 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
97 + .init = NULL,
98 + .create= tg_trimmed_mean_create_1,
99 + .reset = tg_trimmed_mean_reset,
100 + .free = tg_trimmed_mean_free,
101 + .add = tg_trimmed_mean_add,
102 + .flush = tg_trimmed_mean_flush,
103 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
104 + },
105 + {.name = "trimmed-mean2",
106 + .hash = 0,
107 + .value = RRDR_GROUPING_TRIMMED_MEAN2,
108 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
109 + .init = NULL,
110 + .create= tg_trimmed_mean_create_2,
111 + .reset = tg_trimmed_mean_reset,
112 + .free = tg_trimmed_mean_free,
113 + .add = tg_trimmed_mean_add,
114 + .flush = tg_trimmed_mean_flush,
115 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
116 + },
117 + {.name = "trimmed-mean3",
118 + .hash = 0,
119 + .value = RRDR_GROUPING_TRIMMED_MEAN3,
120 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
121 + .init = NULL,
122 + .create= tg_trimmed_mean_create_3,
123 + .reset = tg_trimmed_mean_reset,
124 + .free = tg_trimmed_mean_free,
125 + .add = tg_trimmed_mean_add,
126 + .flush = tg_trimmed_mean_flush,
127 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
128 + },
129 + {.name = "trimmed-mean5",
130 + .hash = 0,
131 + .value = RRDR_GROUPING_TRIMMED_MEAN,
132 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
133 + .init = NULL,
134 + .create= tg_trimmed_mean_create_5,
135 + .reset = tg_trimmed_mean_reset,
136 + .free = tg_trimmed_mean_free,
137 + .add = tg_trimmed_mean_add,
138 + .flush = tg_trimmed_mean_flush,
139 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
140 + },
141 + {.name = "trimmed-mean10",
142 + .hash = 0,
143 + .value = RRDR_GROUPING_TRIMMED_MEAN10,
144 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
145 + .init = NULL,
146 + .create= tg_trimmed_mean_create_10,
147 + .reset = tg_trimmed_mean_reset,
148 + .free = tg_trimmed_mean_free,
149 + .add = tg_trimmed_mean_add,
150 + .flush = tg_trimmed_mean_flush,
151 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
152 + },
153 + {.name = "trimmed-mean15",
154 + .hash = 0,
155 + .value = RRDR_GROUPING_TRIMMED_MEAN15,
156 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
157 + .init = NULL,
158 + .create= tg_trimmed_mean_create_15,
159 + .reset = tg_trimmed_mean_reset,
160 + .free = tg_trimmed_mean_free,
161 + .add = tg_trimmed_mean_add,
162 + .flush = tg_trimmed_mean_flush,
163 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
164 + },
165 + {.name = "trimmed-mean20",
166 + .hash = 0,
167 + .value = RRDR_GROUPING_TRIMMED_MEAN20,
168 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
169 + .init = NULL,
170 + .create= tg_trimmed_mean_create_20,
171 + .reset = tg_trimmed_mean_reset,
172 + .free = tg_trimmed_mean_free,
173 + .add = tg_trimmed_mean_add,
174 + .flush = tg_trimmed_mean_flush,
175 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
176 + },
177 + {.name = "trimmed-mean25",
178 + .hash = 0,
179 + .value = RRDR_GROUPING_TRIMMED_MEAN25,
180 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
181 + .init = NULL,
182 + .create= tg_trimmed_mean_create_25,
183 + .reset = tg_trimmed_mean_reset,
184 + .free = tg_trimmed_mean_free,
185 + .add = tg_trimmed_mean_add,
186 + .flush = tg_trimmed_mean_flush,
187 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
188 + },
189 + {.name = "trimmed-mean",
190 + .hash = 0,
191 + .value = RRDR_GROUPING_TRIMMED_MEAN,
192 + .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
193 + .init = NULL,
194 + .create= tg_trimmed_mean_create_5,
195 + .reset = tg_trimmed_mean_reset,
196 + .free = tg_trimmed_mean_free,
197 + .add = tg_trimmed_mean_add,
198 + .flush = tg_trimmed_mean_flush,
199 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
200 + },
201 + {.name = "incremental_sum",
202 + .hash = 0,
203 + .value = RRDR_GROUPING_INCREMENTAL_SUM,
204 + .add_flush = RRDR_GROUPING_INCREMENTAL_SUM,
205 + .init = NULL,
206 + .create= tg_incremental_sum_create,
207 + .reset = tg_incremental_sum_reset,
208 + .free = tg_incremental_sum_free,
209 + .add = tg_incremental_sum_add,
210 + .flush = tg_incremental_sum_flush,
211 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
212 + },
213 + {.name = "incremental-sum",
214 + .hash = 0,
215 + .value = RRDR_GROUPING_INCREMENTAL_SUM,
216 + .add_flush = RRDR_GROUPING_INCREMENTAL_SUM,
217 + .init = NULL,
218 + .create= tg_incremental_sum_create,
219 + .reset = tg_incremental_sum_reset,
220 + .free = tg_incremental_sum_free,
221 + .add = tg_incremental_sum_add,
222 + .flush = tg_incremental_sum_flush,
223 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
224 + },
225 + {.name = "median",
226 + .hash = 0,
227 + .value = RRDR_GROUPING_MEDIAN,
228 + .add_flush = RRDR_GROUPING_MEDIAN,
229 + .init = NULL,
230 + .create= tg_median_create,
231 + .reset = tg_median_reset,
232 + .free = tg_median_free,
233 + .add = tg_median_add,
234 + .flush = tg_median_flush,
235 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
236 + },
237 + {.name = "trimmed-median1",
238 + .hash = 0,
239 + .value = RRDR_GROUPING_TRIMMED_MEDIAN1,
240 + .add_flush = RRDR_GROUPING_MEDIAN,
241 + .init = NULL,
242 + .create= tg_median_create_trimmed_1,
243 + .reset = tg_median_reset,
244 + .free = tg_median_free,
245 + .add = tg_median_add,
246 + .flush = tg_median_flush,
247 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
248 + },
249 + {.name = "trimmed-median2",
250 + .hash = 0,
251 + .value = RRDR_GROUPING_TRIMMED_MEDIAN2,
252 + .add_flush = RRDR_GROUPING_MEDIAN,
253 + .init = NULL,
254 + .create= tg_median_create_trimmed_2,
255 + .reset = tg_median_reset,
256 + .free = tg_median_free,
257 + .add = tg_median_add,
258 + .flush = tg_median_flush,
259 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
260 + },
261 + {.name = "trimmed-median3",
262 + .hash = 0,
263 + .value = RRDR_GROUPING_TRIMMED_MEDIAN3,
264 + .add_flush = RRDR_GROUPING_MEDIAN,
265 + .init = NULL,
266 + .create= tg_median_create_trimmed_3,
267 + .reset = tg_median_reset,
268 + .free = tg_median_free,
269 + .add = tg_median_add,
270 + .flush = tg_median_flush,
271 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
272 + },
273 + {.name = "trimmed-median5",
274 + .hash = 0,
275 + .value = RRDR_GROUPING_TRIMMED_MEDIAN,
276 + .add_flush = RRDR_GROUPING_MEDIAN,
277 + .init = NULL,
278 + .create= tg_median_create_trimmed_5,
279 + .reset = tg_median_reset,
280 + .free = tg_median_free,
281 + .add = tg_median_add,
282 + .flush = tg_median_flush,
283 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
284 + },
285 + {.name = "trimmed-median10",
286 + .hash = 0,
287 + .value = RRDR_GROUPING_TRIMMED_MEDIAN10,
288 + .add_flush = RRDR_GROUPING_MEDIAN,
289 + .init = NULL,
290 + .create= tg_median_create_trimmed_10,
291 + .reset = tg_median_reset,
292 + .free = tg_median_free,
293 + .add = tg_median_add,
294 + .flush = tg_median_flush,
295 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
296 + },
297 + {.name = "trimmed-median15",
298 + .hash = 0,
299 + .value = RRDR_GROUPING_TRIMMED_MEDIAN15,
300 + .add_flush = RRDR_GROUPING_MEDIAN,
301 + .init = NULL,
302 + .create= tg_median_create_trimmed_15,
303 + .reset = tg_median_reset,
304 + .free = tg_median_free,
305 + .add = tg_median_add,
306 + .flush = tg_median_flush,
307 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
308 + },
309 + {.name = "trimmed-median20",
310 + .hash = 0,
311 + .value = RRDR_GROUPING_TRIMMED_MEDIAN20,
312 + .add_flush = RRDR_GROUPING_MEDIAN,
313 + .init = NULL,
314 + .create= tg_median_create_trimmed_20,
315 + .reset = tg_median_reset,
316 + .free = tg_median_free,
317 + .add = tg_median_add,
318 + .flush = tg_median_flush,
319 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
320 + },
321 + {.name = "trimmed-median25",
322 + .hash = 0,
323 + .value = RRDR_GROUPING_TRIMMED_MEDIAN25,
324 + .add_flush = RRDR_GROUPING_MEDIAN,
325 + .init = NULL,
326 + .create= tg_median_create_trimmed_25,
327 + .reset = tg_median_reset,
328 + .free = tg_median_free,
329 + .add = tg_median_add,
330 + .flush = tg_median_flush,
331 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
332 + },
333 + {.name = "trimmed-median",
334 + .hash = 0,
335 + .value = RRDR_GROUPING_TRIMMED_MEDIAN,
336 + .add_flush = RRDR_GROUPING_MEDIAN,
337 + .init = NULL,
338 + .create= tg_median_create_trimmed_5,
339 + .reset = tg_median_reset,
340 + .free = tg_median_free,
341 + .add = tg_median_add,
342 + .flush = tg_median_flush,
343 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
344 + },
345 + {.name = "percentile25",
346 + .hash = 0,
347 + .value = RRDR_GROUPING_PERCENTILE25,
348 + .add_flush = RRDR_GROUPING_PERCENTILE,
349 + .init = NULL,
350 + .create= tg_percentile_create_25,
351 + .reset = tg_percentile_reset,
352 + .free = tg_percentile_free,
353 + .add = tg_percentile_add,
354 + .flush = tg_percentile_flush,
355 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
356 + },
357 + {.name = "percentile50",
358 + .hash = 0,
359 + .value = RRDR_GROUPING_PERCENTILE50,
360 + .add_flush = RRDR_GROUPING_PERCENTILE,
361 + .init = NULL,
362 + .create= tg_percentile_create_50,
363 + .reset = tg_percentile_reset,
364 + .free = tg_percentile_free,
365 + .add = tg_percentile_add,
366 + .flush = tg_percentile_flush,
367 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
368 + },
369 + {.name = "percentile75",
370 + .hash = 0,
371 + .value = RRDR_GROUPING_PERCENTILE75,
372 + .add_flush = RRDR_GROUPING_PERCENTILE,
373 + .init = NULL,
374 + .create= tg_percentile_create_75,
375 + .reset = tg_percentile_reset,
376 + .free = tg_percentile_free,
377 + .add = tg_percentile_add,
378 + .flush = tg_percentile_flush,
379 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
380 + },
381 + {.name = "percentile80",
382 + .hash = 0,
383 + .value = RRDR_GROUPING_PERCENTILE80,
384 + .add_flush = RRDR_GROUPING_PERCENTILE,
385 + .init = NULL,
386 + .create= tg_percentile_create_80,
387 + .reset = tg_percentile_reset,
388 + .free = tg_percentile_free,
389 + .add = tg_percentile_add,
390 + .flush = tg_percentile_flush,
391 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
392 + },
393 + {.name = "percentile90",
394 + .hash = 0,
395 + .value = RRDR_GROUPING_PERCENTILE90,
396 + .add_flush = RRDR_GROUPING_PERCENTILE,
397 + .init = NULL,
398 + .create= tg_percentile_create_90,
399 + .reset = tg_percentile_reset,
400 + .free = tg_percentile_free,
401 + .add = tg_percentile_add,
402 + .flush = tg_percentile_flush,
403 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
404 + },
405 + {.name = "percentile95",
406 + .hash = 0,
407 + .value = RRDR_GROUPING_PERCENTILE,
408 + .add_flush = RRDR_GROUPING_PERCENTILE,
409 + .init = NULL,
410 + .create= tg_percentile_create_95,
411 + .reset = tg_percentile_reset,
412 + .free = tg_percentile_free,
413 + .add = tg_percentile_add,
414 + .flush = tg_percentile_flush,
415 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
416 + },
417 + {.name = "percentile97",
418 + .hash = 0,
419 + .value = RRDR_GROUPING_PERCENTILE97,
420 + .add_flush = RRDR_GROUPING_PERCENTILE,
421 + .init = NULL,
422 + .create= tg_percentile_create_97,
423 + .reset = tg_percentile_reset,
424 + .free = tg_percentile_free,
425 + .add = tg_percentile_add,
426 + .flush = tg_percentile_flush,
427 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
428 + },
429 + {.name = "percentile98",
430 + .hash = 0,
431 + .value = RRDR_GROUPING_PERCENTILE98,
432 + .add_flush = RRDR_GROUPING_PERCENTILE,
433 + .init = NULL,
434 + .create= tg_percentile_create_98,
435 + .reset = tg_percentile_reset,
436 + .free = tg_percentile_free,
437 + .add = tg_percentile_add,
438 + .flush = tg_percentile_flush,
439 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
440 + },
441 + {.name = "percentile99",
442 + .hash = 0,
443 + .value = RRDR_GROUPING_PERCENTILE99,
444 + .add_flush = RRDR_GROUPING_PERCENTILE,
445 + .init = NULL,
446 + .create= tg_percentile_create_99,
447 + .reset = tg_percentile_reset,
448 + .free = tg_percentile_free,
449 + .add = tg_percentile_add,
450 + .flush = tg_percentile_flush,
451 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
452 + },
453 + {.name = "percentile",
454 + .hash = 0,
455 + .value = RRDR_GROUPING_PERCENTILE,
456 + .add_flush = RRDR_GROUPING_PERCENTILE,
457 + .init = NULL,
458 + .create= tg_percentile_create_95,
459 + .reset = tg_percentile_reset,
460 + .free = tg_percentile_free,
461 + .add = tg_percentile_add,
462 + .flush = tg_percentile_flush,
463 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
464 + },
465 + {.name = "min",
466 + .hash = 0,
467 + .value = RRDR_GROUPING_MIN,
468 + .add_flush = RRDR_GROUPING_MIN,
469 + .init = NULL,
470 + .create= tg_min_create,
471 + .reset = tg_min_reset,
472 + .free = tg_min_free,
473 + .add = tg_min_add,
474 + .flush = tg_min_flush,
475 + .tier_query_fetch = TIER_QUERY_FETCH_MIN
476 + },
477 + {.name = "max",
478 + .hash = 0,
479 + .value = RRDR_GROUPING_MAX,
480 + .add_flush = RRDR_GROUPING_MAX,
481 + .init = NULL,
482 + .create= tg_max_create,
483 + .reset = tg_max_reset,
484 + .free = tg_max_free,
485 + .add = tg_max_add,
486 + .flush = tg_max_flush,
487 + .tier_query_fetch = TIER_QUERY_FETCH_MAX
488 + },
489 + {.name = "sum",
490 + .hash = 0,
491 + .value = RRDR_GROUPING_SUM,
492 + .add_flush = RRDR_GROUPING_SUM,
493 + .init = NULL,
494 + .create= tg_sum_create,
495 + .reset = tg_sum_reset,
496 + .free = tg_sum_free,
497 + .add = tg_sum_add,
498 + .flush = tg_sum_flush,
499 + .tier_query_fetch = TIER_QUERY_FETCH_SUM
500 + },
501 +
502 + // standard deviation
503 + {.name = "stddev",
504 + .hash = 0,
505 + .value = RRDR_GROUPING_STDDEV,
506 + .add_flush = RRDR_GROUPING_STDDEV,
507 + .init = NULL,
508 + .create= tg_stddev_create,
509 + .reset = tg_stddev_reset,
510 + .free = tg_stddev_free,
511 + .add = tg_stddev_add,
512 + .flush = tg_stddev_flush,
513 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
514 + },
515 + {.name = "cv", // coefficient variation is calculated by stddev
516 + .hash = 0,
517 + .value = RRDR_GROUPING_CV,
518 + .add_flush = RRDR_GROUPING_CV,
519 + .init = NULL,
520 + .create= tg_stddev_create, // not an error, stddev calculates this too
521 + .reset = tg_stddev_reset, // not an error, stddev calculates this too
522 + .free = tg_stddev_free, // not an error, stddev calculates this too
523 + .add = tg_stddev_add, // not an error, stddev calculates this too
524 + .flush = tg_stddev_coefficient_of_variation_flush,
525 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
526 + },
527 + {.name = "rsd", // alias of 'cv'
528 + .hash = 0,
529 + .value = RRDR_GROUPING_CV,
530 + .add_flush = RRDR_GROUPING_CV,
531 + .init = NULL,
532 + .create= tg_stddev_create, // not an error, stddev calculates this too
533 + .reset = tg_stddev_reset, // not an error, stddev calculates this too
534 + .free = tg_stddev_free, // not an error, stddev calculates this too
535 + .add = tg_stddev_add, // not an error, stddev calculates this too
536 + .flush = tg_stddev_coefficient_of_variation_flush,
537 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
538 + },
539 + {.name = "coefficient-of-variation", // alias of 'cv'
540 + .hash = 0,
541 + .value = RRDR_GROUPING_CV,
542 + .add_flush = RRDR_GROUPING_CV,
543 + .init = NULL,
544 + .create= tg_stddev_create, // not an error, stddev calculates this too
545 + .reset = tg_stddev_reset, // not an error, stddev calculates this too
546 + .free = tg_stddev_free, // not an error, stddev calculates this too
547 + .add = tg_stddev_add, // not an error, stddev calculates this too
548 + .flush = tg_stddev_coefficient_of_variation_flush,
549 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
550 + },
551 +
552 + // single exponential smoothing
553 + {.name = "ses",
554 + .hash = 0,
555 + .value = RRDR_GROUPING_SES,
556 + .add_flush = RRDR_GROUPING_SES,
557 + .init = tg_ses_init,
558 + .create= tg_ses_create,
559 + .reset = tg_ses_reset,
560 + .free = tg_ses_free,
561 + .add = tg_ses_add,
562 + .flush = tg_ses_flush,
563 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
564 + },
565 + {.name = "ema", // alias for 'ses'
566 + .hash = 0,
567 + .value = RRDR_GROUPING_SES,
568 + .add_flush = RRDR_GROUPING_SES,
569 + .init = NULL,
570 + .create= tg_ses_create,
571 + .reset = tg_ses_reset,
572 + .free = tg_ses_free,
573 + .add = tg_ses_add,
574 + .flush = tg_ses_flush,
575 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
576 + },
577 + {.name = "ewma", // alias for ses
578 + .hash = 0,
579 + .value = RRDR_GROUPING_SES,
580 + .add_flush = RRDR_GROUPING_SES,
581 + .init = NULL,
582 + .create= tg_ses_create,
583 + .reset = tg_ses_reset,
584 + .free = tg_ses_free,
585 + .add = tg_ses_add,
586 + .flush = tg_ses_flush,
587 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
588 + },
589 +
590 + // double exponential smoothing
591 + {.name = "des",
592 + .hash = 0,
593 + .value = RRDR_GROUPING_DES,
594 + .add_flush = RRDR_GROUPING_DES,
595 + .init = tg_des_init,
596 + .create= tg_des_create,
597 + .reset = tg_des_reset,
598 + .free = tg_des_free,
599 + .add = tg_des_add,
600 + .flush = tg_des_flush,
601 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
602 + },
603 +
604 + {.name = "countif",
605 + .hash = 0,
606 + .value = RRDR_GROUPING_COUNTIF,
607 + .add_flush = RRDR_GROUPING_COUNTIF,
608 + .init = NULL,
609 + .create= tg_countif_create,
610 + .reset = tg_countif_reset,
611 + .free = tg_countif_free,
612 + .add = tg_countif_add,
613 + .flush = tg_countif_flush,
614 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
615 + },
616 +
617 + {.name = "extremes",
618 + .hash = 0,
619 + .value = RRDR_GROUPING_EXTREMES,
620 + .add_flush = RRDR_GROUPING_EXTREMES,
621 + .init = NULL,
622 + .create= tg_extremes_create,
623 + .reset = tg_extremes_reset,
624 + .free = tg_extremes_free,
625 + .add = tg_extremes_add,
626 + .flush = tg_extremes_flush,
627 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
628 + },
629 +
630 + // terminator
631 + {.name = NULL,
632 + .hash = 0,
633 + .value = RRDR_GROUPING_UNDEFINED,
634 + .add_flush = RRDR_GROUPING_AVERAGE,
635 + .init = NULL,
636 + .create= tg_average_create,
637 + .reset = tg_average_reset,
638 + .free = tg_average_free,
639 + .add = tg_average_add,
640 + .flush = tg_average_flush,
641 + .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
642 + }
643 +};
644 +
645 +void time_grouping_init(void) {
646 + int i;
647 +
648 + for(i = 0; api_v1_data_groups[i].name ; i++) {
649 + api_v1_data_groups[i].hash = simple_hash(api_v1_data_groups[i].name);
650 +
651 + if(api_v1_data_groups[i].init)
652 + api_v1_data_groups[i].init();
653 + }
654 +}
655 +
656 +const char *time_grouping_id2txt(RRDR_TIME_GROUPING group) {
657 + int i;
658 +
659 + for(i = 0; api_v1_data_groups[i].name ; i++) {
660 + if(api_v1_data_groups[i].value == group) {
661 + return api_v1_data_groups[i].name;
662 + }
663 + }
664 +
665 + return "average";
666 +}
667 +
668 +RRDR_TIME_GROUPING time_grouping_txt2id(const char *name) {
669 + int i;
670 +
671 + uint32_t hash = simple_hash(name);
672 + for(i = 0; api_v1_data_groups[i].name ; i++)
673 + if(unlikely(hash == api_v1_data_groups[i].hash && !strcmp(name, api_v1_data_groups[i].name)))
674 + return api_v1_data_groups[i].value;
675 +
676 + return RRDR_GROUPING_AVERAGE;
677 +}
678 +
679 +RRDR_TIME_GROUPING time_grouping_parse(const char *name, RRDR_TIME_GROUPING def) {
680 + int i;
681 +
682 + uint32_t hash = simple_hash(name);
683 + for(i = 0; api_v1_data_groups[i].name ; i++)
684 + if(unlikely(hash == api_v1_data_groups[i].hash && !strcmp(name, api_v1_data_groups[i].name)))
685 + return api_v1_data_groups[i].value;
686 +
687 + return def;
688 +}
689 +
690 +const char *time_grouping_tostring(RRDR_TIME_GROUPING group) {
691 + int i;
692 +
693 + for(i = 0; api_v1_data_groups[i].name ; i++)
694 + if(unlikely(group == api_v1_data_groups[i].value))
695 + return api_v1_data_groups[i].name;
696 +
697 + return "unknown";
698 +}
699 +
700 +void rrdr_set_grouping_function(RRDR *r, RRDR_TIME_GROUPING group_method) {
701 + int i, found = 0;
702 + for(i = 0; !found && api_v1_data_groups[i].name ;i++) {
703 + if(api_v1_data_groups[i].value == group_method) {
704 + r->time_grouping.create = api_v1_data_groups[i].create;
705 + r->time_grouping.reset = api_v1_data_groups[i].reset;
706 + r->time_grouping.free = api_v1_data_groups[i].free;
707 + r->time_grouping.add = api_v1_data_groups[i].add;
708 + r->time_grouping.flush = api_v1_data_groups[i].flush;
709 + r->time_grouping.tier_query_fetch = api_v1_data_groups[i].tier_query_fetch;
710 + r->time_grouping.add_flush = api_v1_data_groups[i].add_flush;
711 + found = 1;
712 + }
713 + }
714 + if(!found) {
715 + errno_clear();
716 + internal_error(true, "QUERY: grouping method %u not found. Using 'average'", (unsigned int)group_method);
717 + r->time_grouping.create = tg_average_create;
718 + r->time_grouping.reset = tg_average_reset;
719 + r->time_grouping.free = tg_average_free;
720 + r->time_grouping.add = tg_average_add;
721 + r->time_grouping.flush = tg_average_flush;
722 + r->time_grouping.tier_query_fetch = TIER_QUERY_FETCH_AVERAGE;
723 + r->time_grouping.add_flush = RRDR_GROUPING_AVERAGE;
724 + }
725 +}
726 +
727 +ALWAYS_INLINE_HOT_FLATTEN
728 +void time_grouping_add(RRDR *r, NETDATA_DOUBLE value, const RRDR_TIME_GROUPING add_flush) {
729 + switch(add_flush) {
730 + case RRDR_GROUPING_AVERAGE:
731 + tg_average_add(r, value);
732 + break;
733 +
734 + case RRDR_GROUPING_MAX:
735 + tg_max_add(r, value);
736 + break;
737 +
738 + case RRDR_GROUPING_MIN:
739 + tg_min_add(r, value);
740 + break;
741 +
742 + case RRDR_GROUPING_MEDIAN:
743 + tg_median_add(r, value);
744 + break;
745 +
746 + case RRDR_GROUPING_STDDEV:
747 + case RRDR_GROUPING_CV:
748 + tg_stddev_add(r, value);
749 + break;
750 +
751 + case RRDR_GROUPING_SUM:
752 + tg_sum_add(r, value);
753 + break;
754 +
755 + case RRDR_GROUPING_COUNTIF:
756 + tg_countif_add(r, value);
757 + break;
758 +
759 + case RRDR_GROUPING_EXTREMES:
760 + tg_extremes_add(r, value);
761 + break;
762 +
763 + case RRDR_GROUPING_TRIMMED_MEAN:
764 + tg_trimmed_mean_add(r, value);
765 + break;
766 +
767 + case RRDR_GROUPING_PERCENTILE:
768 + tg_percentile_add(r, value);
769 + break;
770 +
771 + case RRDR_GROUPING_SES:
772 + tg_ses_add(r, value);
773 + break;
774 +
775 + case RRDR_GROUPING_DES:
776 + tg_des_add(r, value);
777 + break;
778 +
779 + case RRDR_GROUPING_INCREMENTAL_SUM:
780 + tg_incremental_sum_add(r, value);
781 + break;
782 +
783 + default:
784 + r->time_grouping.add(r, value);
785 + break;
786 + }
787 +}
788 +
789 +ALWAYS_INLINE_HOT_FLATTEN
790 +NETDATA_DOUBLE time_grouping_flush(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr, const RRDR_TIME_GROUPING add_flush) {
791 + switch(add_flush) {
792 + case RRDR_GROUPING_AVERAGE:
793 + return tg_average_flush(r, rrdr_value_options_ptr);
794 +
795 + case RRDR_GROUPING_MAX:
796 + return tg_max_flush(r, rrdr_value_options_ptr);
797 +
798 + case RRDR_GROUPING_MIN:
799 + return tg_min_flush(r, rrdr_value_options_ptr);
800 +
801 + case RRDR_GROUPING_MEDIAN:
802 + return tg_median_flush(r, rrdr_value_options_ptr);
803 +
804 + case RRDR_GROUPING_STDDEV:
805 + return tg_stddev_flush(r, rrdr_value_options_ptr);
806 +
807 + case RRDR_GROUPING_CV:
808 + return tg_stddev_coefficient_of_variation_flush(r, rrdr_value_options_ptr);
809 +
810 + case RRDR_GROUPING_SUM:
811 + return tg_sum_flush(r, rrdr_value_options_ptr);
812 +
813 + case RRDR_GROUPING_COUNTIF:
814 + return tg_countif_flush(r, rrdr_value_options_ptr);
815 +
816 + case RRDR_GROUPING_EXTREMES:
817 + return tg_extremes_flush(r, rrdr_value_options_ptr);
818 +
819 + case RRDR_GROUPING_TRIMMED_MEAN:
820 + return tg_trimmed_mean_flush(r, rrdr_value_options_ptr);
821 +
822 + case RRDR_GROUPING_PERCENTILE:
823 + return tg_percentile_flush(r, rrdr_value_options_ptr);
824 +
825 + case RRDR_GROUPING_SES:
826 + return tg_ses_flush(r, rrdr_value_options_ptr);
827 +
828 + case RRDR_GROUPING_DES:
829 + return tg_des_flush(r, rrdr_value_options_ptr);
830 +
831 + case RRDR_GROUPING_INCREMENTAL_SUM:
832 + return tg_incremental_sum_flush(r, rrdr_value_options_ptr);
833 +
834 + default:
835 + return r->time_grouping.flush(r, rrdr_value_options_ptr);
836 + }
837 +}
src/web/api/queries/query-internal.h new
+109
@@ -0,0 +1,109 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_QUERY_INTERNAL_H
4 +#define NETDATA_QUERY_INTERNAL_H
5 +
6 +#include "query.h"
7 +#include "web/api/formatters/rrd2json.h"
8 +#include "rrdr.h"
9 +
10 +#define QUERY_PLAN_MIN_POINTS 10
11 +#define POINTS_TO_EXPAND_QUERY 5
12 +
13 +typedef struct query_point {
14 + STORAGE_POINT sp;
15 + NETDATA_DOUBLE value;
16 + bool added;
17 +#ifdef NETDATA_INTERNAL_CHECKS
18 + size_t id;
19 +#endif
20 +} QUERY_POINT;
21 +
22 +#ifdef NETDATA_INTERNAL_CHECKS
23 +#define QUERY_POINT_EMPTY (QUERY_POINT){ \
24 + .sp = STORAGE_POINT_UNSET, \
25 + .value = NAN, \
26 + .added = false, \
27 + .id = 0, \
28 +}
29 +#else
30 +#define QUERY_POINT_EMPTY (QUERY_POINT){ \
31 + .sp = STORAGE_POINT_UNSET, \
32 + .value = NAN, \
33 + .added = false, \
34 +}
35 +#endif
36 +
37 +#ifdef NETDATA_INTERNAL_CHECKS
38 +#define query_point_set_id(point, point_id) (point).id = point_id
39 +#else
40 +#define query_point_set_id(point, point_id) debug_dummy()
41 +#endif
42 +
43 +typedef struct query_engine_ops {
44 + // configuration
45 + RRDR *r;
46 + QUERY_METRIC *qm;
47 + time_t view_update_every;
48 + time_t query_granularity;
49 + TIER_QUERY_FETCH tier_query_fetch;
50 +
51 + // query planer
52 + size_t current_plan;
53 + time_t current_plan_expire_time;
54 + time_t plan_expanded_after;
55 + time_t plan_expanded_before;
56 +
57 + // storage queries
58 + size_t tier;
59 + struct query_metric_tier *tier_ptr;
60 + struct storage_engine_query_handle *seqh;
61 +
62 + // aggregating points over time
63 + size_t group_points_non_zero;
64 + size_t group_points_added;
65 + STORAGE_POINT group_point; // aggregates min, max, sum, count, anomaly count for each group point
66 + STORAGE_POINT query_point; // aggregates min, max, sum, count, anomaly count across the whole query
67 + RRDR_VALUE_FLAGS group_value_flags;
68 +
69 + // statistics
70 + size_t db_total_points_read;
71 + size_t db_points_read_per_tier[RRD_STORAGE_TIERS];
72 +
73 + struct {
74 + time_t expanded_after;
75 + time_t expanded_before;
76 + struct storage_engine_query_handle handle;
77 + bool initialized;
78 + bool finalized;
79 + } plans[QUERY_PLANS_MAX];
80 +
81 + struct query_engine_ops *next;
82 +} QUERY_ENGINE_OPS;
83 +
84 +// query planner
85 +#define query_plan_should_switch_plan(ops, now) ((now) >= (ops)->current_plan_expire_time)
86 +bool query_planer_next_plan(QUERY_ENGINE_OPS *ops, time_t now, time_t last_point_end_time);
87 +void query_planer_finalize_remaining_plans(QUERY_ENGINE_OPS *ops);
88 +QUERY_ENGINE_OPS *rrd2rrdr_query_ops_prep(RRDR *r, size_t query_metric_id);
89 +void rrd2rrdr_query_ops_release(QUERY_ENGINE_OPS *ops);
90 +time_t rrdset_find_natural_update_every_for_timeframe(QUERY_TARGET *qt, time_t after_wanted, time_t before_wanted, size_t points_wanted, RRDR_OPTIONS options, size_t tier);
91 +void rrd2rrdr_query_ops_freeall(RRDR *r);
92 +
93 +// time aggregation
94 +void time_grouping_add(RRDR *r, NETDATA_DOUBLE value, const RRDR_TIME_GROUPING add_flush);
95 +NETDATA_DOUBLE time_grouping_flush(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr, const RRDR_TIME_GROUPING add_flush);
96 +void rrdr_set_grouping_function(RRDR *r, RRDR_TIME_GROUPING group_method);
97 +
98 +// group by
99 +RRDR *rrd2rrdr_group_by_initialize(ONEWAYALLOC *owa, QUERY_TARGET *qt);
100 +void rrdr2rrdr_group_by_calculate_percentage_of_group(RRDR *r);
101 +void rrdr2rrdr_group_by_partial_trimming(RRDR *r);
102 +void rrd2rrdr_group_by_add_metric(RRDR *r_dst, size_t d_dst, RRDR *r_tmp, size_t d_tmp,
103 + RRDR_GROUP_BY_FUNCTION group_by_aggregate_function,
104 + STORAGE_POINT *query_points, size_t pass);
105 +void rrd2rrdr_convert_values_to_percentage_of_total(RRDR *r);
106 +RRDR *rrd2rrdr_group_by_finalize(RRDR *r_tmp);
107 +RRDR *rrd2rrdr_cardinality_limit(RRDR *r);
108 +
109 +#endif //NETDATA_QUERY_INTERNAL_H
src/web/api/queries/query-plan.c new
+518
@@ -0,0 +1,518 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "query-internal.h"
4 +
5 +static bool query_metric_is_valid_tier(QUERY_METRIC *qm, size_t tier) {
6 + if(!qm->tiers[tier].smh || !qm->tiers[tier].db_first_time_s || !qm->tiers[tier].db_last_time_s || !qm->tiers[tier].db_update_every_s)
7 + return false;
8 +
9 + return true;
10 +}
11 +
12 +static size_t query_metric_first_working_tier(QUERY_METRIC *qm) {
13 + for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
14 +
15 + // find the db time-range for this tier for all metrics
16 + STORAGE_METRIC_HANDLE *smh = qm->tiers[tier].smh;
17 + time_t first_time_s = qm->tiers[tier].db_first_time_s;
18 + time_t last_time_s = qm->tiers[tier].db_last_time_s;
19 + time_t update_every_s = qm->tiers[tier].db_update_every_s;
20 +
21 + if(!smh || !first_time_s || !last_time_s || !update_every_s)
22 + continue;
23 +
24 + return tier;
25 + }
26 +
27 + return 0;
28 +}
29 +
30 +long query_plan_points_coverage_weight(time_t db_first_time_s, time_t db_last_time_s, time_t db_update_every_s, time_t after_wanted, time_t before_wanted, size_t points_wanted, size_t tier __maybe_unused) {
31 + if(db_first_time_s == 0 ||
32 + db_last_time_s == 0 ||
33 + db_update_every_s == 0 ||
34 + db_first_time_s > before_wanted ||
35 + db_last_time_s < after_wanted)
36 + return -LONG_MAX;
37 +
38 + long long common_first_t = MAX(db_first_time_s, after_wanted);
39 + long long common_last_t = MIN(db_last_time_s, before_wanted);
40 +
41 + long long time_coverage = (common_last_t - common_first_t) * 1000000LL / (before_wanted - after_wanted);
42 + long long points_wanted_in_coverage = (long long)points_wanted * time_coverage / 1000000LL;
43 +
44 + long long points_available = (common_last_t - common_first_t) / db_update_every_s;
45 + long long points_delta = (long)(points_available - points_wanted_in_coverage);
46 + long long points_coverage = (points_delta < 0) ? (long)(points_available * time_coverage / points_wanted_in_coverage) : time_coverage;
47 +
48 + // a way to benefit higher tiers
49 + // points_coverage += (long)tier * 10000;
50 +
51 + if(points_available <= 0)
52 + return -LONG_MAX;
53 +
54 + return (long)(points_coverage + (25000LL * tier)); // 2.5% benefit for each higher tier
55 +}
56 +
57 +static size_t query_metric_best_tier_for_timeframe(QUERY_METRIC *qm, time_t after_wanted, time_t before_wanted, size_t points_wanted) {
58 + if(unlikely(nd_profile.storage_tiers < 2))
59 + return 0;
60 +
61 + if(unlikely(after_wanted == before_wanted || points_wanted <= 0))
62 + return query_metric_first_working_tier(qm);
63 +
64 + if(points_wanted < QUERY_PLAN_MIN_POINTS)
65 + // when selecting tiers, aim for a resolution of at least QUERY_PLAN_MIN_POINTS points
66 + points_wanted = (before_wanted - after_wanted) > QUERY_PLAN_MIN_POINTS ? QUERY_PLAN_MIN_POINTS : before_wanted - after_wanted;
67 +
68 + time_t min_first_time_s = 0;
69 + time_t max_last_time_s = 0;
70 +
71 + for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
72 + time_t first_time_s = qm->tiers[tier].db_first_time_s;
73 + time_t last_time_s = qm->tiers[tier].db_last_time_s;
74 +
75 + if(!min_first_time_s || (first_time_s && first_time_s < min_first_time_s))
76 + min_first_time_s = first_time_s;
77 +
78 + if(!max_last_time_s || (last_time_s && last_time_s > max_last_time_s))
79 + max_last_time_s = last_time_s;
80 + }
81 +
82 + for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
83 +
84 + // find the db time-range for this tier for all metrics
85 + STORAGE_METRIC_HANDLE *smh = qm->tiers[tier].smh;
86 + time_t first_time_s = qm->tiers[tier].db_first_time_s;
87 + time_t last_time_s = qm->tiers[tier].db_last_time_s;
88 + time_t update_every_s = qm->tiers[tier].db_update_every_s;
89 +
90 + if( !smh ||
91 + !first_time_s ||
92 + !last_time_s ||
93 + !update_every_s ||
94 + first_time_s > before_wanted ||
95 + last_time_s < after_wanted
96 + ) {
97 + qm->tiers[tier].weight = -LONG_MAX;
98 + continue;
99 + }
100 +
101 + internal_fatal(first_time_s > before_wanted || last_time_s < after_wanted, "QUERY: invalid db durations");
102 +
103 + qm->tiers[tier].weight = query_plan_points_coverage_weight(
104 + min_first_time_s, max_last_time_s, update_every_s,
105 + after_wanted, before_wanted, points_wanted, tier);
106 + }
107 +
108 + size_t best_tier = 0;
109 + for(size_t tier = 1; tier < nd_profile.storage_tiers; tier++) {
110 + if(qm->tiers[tier].weight >= qm->tiers[best_tier].weight)
111 + best_tier = tier;
112 + }
113 +
114 + return best_tier;
115 +}
116 +
117 +static size_t rrddim_find_best_tier_for_timeframe(QUERY_TARGET *qt, time_t after_wanted, time_t before_wanted, size_t points_wanted) {
118 + if(unlikely(nd_profile.storage_tiers < 2))
119 + return 0;
120 +
121 + if(unlikely(after_wanted == before_wanted || points_wanted <= 0)) {
122 + internal_error(true, "QUERY: '%s' has invalid params to tier calculation", qt->id);
123 + return 0;
124 + }
125 +
126 + long weight[nd_profile.storage_tiers];
127 +
128 + for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
129 +
130 + time_t common_first_time_s = 0;
131 + time_t common_last_time_s = 0;
132 + time_t common_update_every_s = 0;
133 +
134 + // find the db time-range for this tier for all metrics
135 + for(size_t i = 0, used = qt->query.used; i < used ; i++) {
136 + QUERY_METRIC *qm = query_metric(qt, i);
137 +
138 + time_t first_time_s = qm->tiers[tier].db_first_time_s;
139 + time_t last_time_s = qm->tiers[tier].db_last_time_s;
140 + time_t update_every_s = qm->tiers[tier].db_update_every_s;
141 +
142 + if(!first_time_s || !last_time_s || !update_every_s)
143 + continue;
144 +
145 + if(!common_first_time_s)
146 + common_first_time_s = first_time_s;
147 + else
148 + common_first_time_s = MIN(first_time_s, common_first_time_s);
149 +
150 + if(!common_last_time_s)
151 + common_last_time_s = last_time_s;
152 + else
153 + common_last_time_s = MAX(last_time_s, common_last_time_s);
154 +
155 + if(!common_update_every_s)
156 + common_update_every_s = update_every_s;
157 + else
158 + common_update_every_s = MIN(update_every_s, common_update_every_s);
159 + }
160 +
161 + weight[tier] = query_plan_points_coverage_weight(common_first_time_s, common_last_time_s, common_update_every_s, after_wanted, before_wanted, points_wanted, tier);
162 + }
163 +
164 + size_t best_tier = 0;
165 + for(size_t tier = 1; tier < nd_profile.storage_tiers; tier++) {
166 + if(weight[tier] >= weight[best_tier])
167 + best_tier = tier;
168 + }
169 +
170 + if(weight[best_tier] == -LONG_MAX)
171 + best_tier = 0;
172 +
173 + return best_tier;
174 +}
175 +
176 +time_t rrdset_find_natural_update_every_for_timeframe(QUERY_TARGET *qt, time_t after_wanted, time_t before_wanted, size_t points_wanted, RRDR_OPTIONS options, size_t tier) {
177 + size_t best_tier;
178 + if((options & RRDR_OPTION_SELECTED_TIER) && tier < nd_profile.storage_tiers)
179 + best_tier = tier;
180 + else
181 + best_tier = rrddim_find_best_tier_for_timeframe(qt, after_wanted, before_wanted, points_wanted);
182 +
183 + // find the db minimum update every for this tier for all metrics
184 + time_t common_update_every_s = nd_profile.update_every;
185 + for(size_t i = 0, used = qt->query.used; i < used ; i++) {
186 + QUERY_METRIC *qm = query_metric(qt, i);
187 +
188 + time_t update_every_s = qm->tiers[best_tier].db_update_every_s;
189 +
190 + if(!i)
191 + common_update_every_s = update_every_s;
192 + else
193 + common_update_every_s = MIN(update_every_s, common_update_every_s);
194 + }
195 +
196 + return common_update_every_s;
197 +}
198 +
199 +static size_t query_planer_expand_duration_in_points(time_t this_update_every, time_t next_update_every) {
200 +
201 + time_t delta = this_update_every - next_update_every;
202 + if(delta < 0) delta = -delta;
203 +
204 + size_t points;
205 + if(delta < this_update_every * POINTS_TO_EXPAND_QUERY)
206 + points = POINTS_TO_EXPAND_QUERY;
207 + else
208 + points = (delta + this_update_every - 1) / this_update_every;
209 +
210 + return points;
211 +}
212 +
213 +static void query_planer_initialize_plans(QUERY_ENGINE_OPS *ops) {
214 + QUERY_METRIC *qm = ops->qm;
215 +
216 + for(size_t p = 0; p < qm->plan.used ; p++) {
217 + size_t tier = qm->plan.array[p].tier;
218 + time_t update_every = qm->tiers[tier].db_update_every_s;
219 +
220 + size_t points_to_add_to_after;
221 + if(p > 0) {
222 + // there is another plan before to this
223 +
224 + size_t tier0 = qm->plan.array[p - 1].tier;
225 + time_t update_every0 = qm->tiers[tier0].db_update_every_s;
226 +
227 + points_to_add_to_after = query_planer_expand_duration_in_points(update_every, update_every0);
228 + }
229 + else
230 + points_to_add_to_after = (tier == 0) ? 0 : POINTS_TO_EXPAND_QUERY;
231 +
232 + size_t points_to_add_to_before;
233 + if(p + 1 < qm->plan.used) {
234 + // there is another plan after to this
235 +
236 + size_t tier1 = qm->plan.array[p+1].tier;
237 + time_t update_every1 = qm->tiers[tier1].db_update_every_s;
238 +
239 + points_to_add_to_before = query_planer_expand_duration_in_points(update_every, update_every1);
240 + }
241 + else
242 + points_to_add_to_before = POINTS_TO_EXPAND_QUERY;
243 +
244 + time_t after = qm->plan.array[p].after - (time_t)(update_every * points_to_add_to_after);
245 + time_t before = qm->plan.array[p].before + (time_t)(update_every * points_to_add_to_before);
246 +
247 + ops->plans[p].expanded_after = after;
248 + ops->plans[p].expanded_before = before;
249 +
250 + ops->r->internal.qt->db.tiers[tier].queries++;
251 +
252 + struct query_metric_tier *tier_ptr = &qm->tiers[tier];
253 + STORAGE_ENGINE *eng = query_metric_storage_engine(ops->r->internal.qt, qm, tier);
254 + storage_engine_query_init(eng->seb, tier_ptr->smh, &ops->plans[p].handle,
255 + after, before, ops->r->internal.qt->request.priority);
256 +
257 + ops->plans[p].initialized = true;
258 + ops->plans[p].finalized = false;
259 + }
260 +}
261 +
262 +static void query_planer_finalize_plan(QUERY_ENGINE_OPS *ops, size_t plan_id) {
263 + // QUERY_METRIC *qm = ops->qm;
264 +
265 + if(ops->plans[plan_id].initialized && !ops->plans[plan_id].finalized) {
266 + storage_engine_query_finalize(&ops->plans[plan_id].handle);
267 + ops->plans[plan_id].initialized = false;
268 + ops->plans[plan_id].finalized = true;
269 + }
270 +}
271 +
272 +void query_planer_finalize_remaining_plans(QUERY_ENGINE_OPS *ops) {
273 + QUERY_METRIC *qm = ops->qm;
274 +
275 + for(size_t p = 0; p < qm->plan.used ; p++)
276 + query_planer_finalize_plan(ops, p);
277 +}
278 +
279 +static void query_planer_activate_plan(QUERY_ENGINE_OPS *ops, size_t plan_id, time_t overwrite_after __maybe_unused) {
280 + QUERY_METRIC *qm = ops->qm;
281 +
282 + internal_fatal(plan_id >= qm->plan.used, "QUERY: invalid plan_id given");
283 + internal_fatal(!ops->plans[plan_id].initialized, "QUERY: plan has not been initialized");
284 + internal_fatal(ops->plans[plan_id].finalized, "QUERY: plan has been finalized");
285 +
286 + internal_fatal(qm->plan.array[plan_id].after > qm->plan.array[plan_id].before, "QUERY: flipped after/before");
287 +
288 + ops->tier = qm->plan.array[plan_id].tier;
289 + ops->tier_ptr = &qm->tiers[ops->tier];
290 + ops->seqh = &ops->plans[plan_id].handle;
291 + ops->current_plan = plan_id;
292 +
293 + if(plan_id + 1 < qm->plan.used && qm->plan.array[plan_id + 1].after < qm->plan.array[plan_id].before)
294 + ops->current_plan_expire_time = qm->plan.array[plan_id + 1].after;
295 + else
296 + ops->current_plan_expire_time = qm->plan.array[plan_id].before;
297 +
298 + ops->plan_expanded_after = ops->plans[plan_id].expanded_after;
299 + ops->plan_expanded_before = ops->plans[plan_id].expanded_before;
300 +}
301 +
302 +bool query_planer_next_plan(QUERY_ENGINE_OPS *ops, time_t now, time_t last_point_end_time) {
303 + QUERY_METRIC *qm = ops->qm;
304 +
305 + size_t old_plan = ops->current_plan;
306 +
307 + time_t next_plan_before_time;
308 + do {
309 + ops->current_plan++;
310 +
311 + if (ops->current_plan >= qm->plan.used) {
312 + ops->current_plan = old_plan;
313 + ops->current_plan_expire_time = ops->r->internal.qt->window.before;
314 + // let the query run with current plan
315 + // we will not switch it
316 + return false;
317 + }
318 +
319 + next_plan_before_time = qm->plan.array[ops->current_plan].before;
320 + } while(now >= next_plan_before_time || last_point_end_time >= next_plan_before_time);
321 +
322 + if(!query_metric_is_valid_tier(qm, qm->plan.array[ops->current_plan].tier)) {
323 + ops->current_plan = old_plan;
324 + ops->current_plan_expire_time = ops->r->internal.qt->window.before;
325 + return false;
326 + }
327 +
328 + query_planer_finalize_plan(ops, old_plan);
329 + query_planer_activate_plan(ops, ops->current_plan, MIN(now, last_point_end_time));
330 + return true;
331 +}
332 +
333 +static int compare_query_plan_entries_on_start_time(const void *a, const void *b) {
334 + QUERY_PLAN_ENTRY *p1 = (QUERY_PLAN_ENTRY *)a;
335 + QUERY_PLAN_ENTRY *p2 = (QUERY_PLAN_ENTRY *)b;
336 + return (p1->after < p2->after)?-1:1;
337 +}
338 +
339 +static bool query_plan(QUERY_ENGINE_OPS *ops, time_t after_wanted, time_t before_wanted, size_t points_wanted) {
340 + QUERY_METRIC *qm = ops->qm;
341 +
342 + // put our selected tier as the first plan
343 + size_t selected_tier;
344 + bool switch_tiers = true;
345 +
346 + if((ops->r->internal.qt->window.options & RRDR_OPTION_SELECTED_TIER)
347 + && ops->r->internal.qt->window.tier < nd_profile.storage_tiers && query_metric_is_valid_tier(qm, ops->r->internal.qt->window.tier)) {
348 + selected_tier = ops->r->internal.qt->window.tier;
349 + switch_tiers = false;
350 + }
351 + else {
352 + selected_tier = query_metric_best_tier_for_timeframe(qm, after_wanted, before_wanted, points_wanted);
353 +
354 + if(!query_metric_is_valid_tier(qm, selected_tier))
355 + return false;
356 + }
357 +
358 + if(qm->tiers[selected_tier].db_first_time_s > before_wanted ||
359 + qm->tiers[selected_tier].db_last_time_s < after_wanted) {
360 + // we don't have any data to satisfy this query
361 + return false;
362 + }
363 +
364 + qm->plan.used = 1;
365 + qm->plan.array[0].tier = selected_tier;
366 + qm->plan.array[0].after = (qm->tiers[selected_tier].db_first_time_s < after_wanted) ? after_wanted : qm->tiers[selected_tier].db_first_time_s;
367 + qm->plan.array[0].before = (qm->tiers[selected_tier].db_last_time_s > before_wanted) ? before_wanted : qm->tiers[selected_tier].db_last_time_s;
368 +
369 + if(switch_tiers) {
370 + // the selected tier
371 + time_t selected_tier_first_time_s = qm->plan.array[0].after;
372 + time_t selected_tier_last_time_s = qm->plan.array[0].before;
373 +
374 + // check if our selected tier can start the query
375 + if (selected_tier_first_time_s > after_wanted) {
376 + // we need some help from other tiers
377 + for (size_t tr = (int)selected_tier + 1; tr < nd_profile.storage_tiers && qm->plan.used < QUERY_PLANS_MAX ; tr++) {
378 + if(!query_metric_is_valid_tier(qm, tr))
379 + continue;
380 +
381 + // find the first time of this tier
382 + time_t tier_first_time_s = qm->tiers[tr].db_first_time_s;
383 + time_t tier_last_time_s = qm->tiers[tr].db_last_time_s;
384 +
385 + // can it help?
386 + if (tier_first_time_s < selected_tier_first_time_s && tier_first_time_s <= before_wanted && tier_last_time_s >= after_wanted) {
387 + // it can help us add detail at the beginning of the query
388 + QUERY_PLAN_ENTRY t = {
389 + .tier = tr,
390 + .after = (tier_first_time_s < after_wanted) ? after_wanted : tier_first_time_s,
391 + .before = selected_tier_first_time_s,
392 + };
393 + ops->plans[qm->plan.used].initialized = false;
394 + ops->plans[qm->plan.used].finalized = false;
395 + qm->plan.array[qm->plan.used++] = t;
396 +
397 + internal_fatal(!t.after || !t.before, "QUERY: invalid plan selected");
398 +
399 + // prepare for the tier
400 + selected_tier_first_time_s = t.after;
401 +
402 + if (t.after <= after_wanted)
403 + break;
404 + }
405 + }
406 + }
407 +
408 + // check if our selected tier can finish the query
409 + if (selected_tier_last_time_s < before_wanted) {
410 + // we need some help from other tiers
411 + for (int tr = (int)selected_tier - 1; tr >= 0 && qm->plan.used < QUERY_PLANS_MAX ; tr--) {
412 + if(!query_metric_is_valid_tier(qm, tr))
413 + continue;
414 +
415 + // find the last time of this tier
416 + time_t tier_first_time_s = qm->tiers[tr].db_first_time_s;
417 + time_t tier_last_time_s = qm->tiers[tr].db_last_time_s;
418 +
419 + //buffer_sprintf(wb, ": EVAL BEFORE tier %d, %ld", tier, last_time_s);
420 +
421 + // can it help?
422 + if (tier_last_time_s > selected_tier_last_time_s && tier_first_time_s <= before_wanted && tier_last_time_s >= after_wanted) {
423 + // it can help us add detail at the end of the query
424 + QUERY_PLAN_ENTRY t = {
425 + .tier = tr,
426 + .after = selected_tier_last_time_s,
427 + .before = (tier_last_time_s > before_wanted) ? before_wanted : tier_last_time_s,
428 + };
429 + ops->plans[qm->plan.used].initialized = false;
430 + ops->plans[qm->plan.used].finalized = false;
431 + qm->plan.array[qm->plan.used++] = t;
432 +
433 + // prepare for the tier
434 + selected_tier_last_time_s = t.before;
435 +
436 + internal_fatal(!t.after || !t.before, "QUERY: invalid plan selected");
437 +
438 + if (t.before >= before_wanted)
439 + break;
440 + }
441 + }
442 + }
443 + }
444 +
445 + // sort the query plan
446 + if(qm->plan.used > 1)
447 + qsort(&qm->plan.array, qm->plan.used, sizeof(QUERY_PLAN_ENTRY), compare_query_plan_entries_on_start_time);
448 +
449 + if(!query_metric_is_valid_tier(qm, qm->plan.array[0].tier))
450 + return false;
451 +
452 +#ifdef NETDATA_INTERNAL_CHECKS
453 + for(size_t p = 0; p < qm->plan.used ;p++) {
454 + internal_fatal(qm->plan.array[p].after > qm->plan.array[p].before, "QUERY: flipped after/before");
455 + internal_fatal(qm->plan.array[p].after < after_wanted, "QUERY: too small plan first time");
456 + internal_fatal(qm->plan.array[p].before > before_wanted, "QUERY: too big plan last time");
457 + }
458 +#endif
459 +
460 + query_planer_initialize_plans(ops);
461 + query_planer_activate_plan(ops, 0, 0);
462 +
463 + return true;
464 +}
465 +
466 +
467 +static __thread QUERY_ENGINE_OPS *released_ops = NULL;
468 +
469 +void rrd2rrdr_query_ops_freeall(RRDR *r __maybe_unused) {
470 + while(released_ops) {
471 + QUERY_ENGINE_OPS *ops = released_ops;
472 + released_ops = ops->next;
473 +
474 + onewayalloc_freez(r->internal.owa, ops);
475 + }
476 +}
477 +
478 +void rrd2rrdr_query_ops_release(QUERY_ENGINE_OPS *ops) {
479 + if(!ops) return;
480 +
481 + ops->next = released_ops;
482 + released_ops = ops;
483 +}
484 +
485 +static QUERY_ENGINE_OPS *rrd2rrdr_query_ops_get(RRDR *r) {
486 + QUERY_ENGINE_OPS *ops;
487 + if(released_ops) {
488 + ops = released_ops;
489 + released_ops = ops->next;
490 + }
491 + else {
492 + ops = onewayalloc_mallocz(r->internal.owa, sizeof(QUERY_ENGINE_OPS));
493 + }
494 +
495 + memset(ops, 0, sizeof(*ops));
496 + return ops;
497 +}
498 +
499 +QUERY_ENGINE_OPS *rrd2rrdr_query_ops_prep(RRDR *r, size_t query_metric_id) {
500 + QUERY_TARGET *qt = r->internal.qt;
501 +
502 + QUERY_ENGINE_OPS *ops = rrd2rrdr_query_ops_get(r);
503 + *ops = (QUERY_ENGINE_OPS) {
504 + .r = r,
505 + .qm = query_metric(qt, query_metric_id),
506 + .tier_query_fetch = r->time_grouping.tier_query_fetch,
507 + .view_update_every = r->view.update_every,
508 + .query_granularity = (time_t)(r->view.update_every / r->view.group),
509 + .group_value_flags = RRDR_VALUE_NOTHING,
510 + };
511 +
512 + if(!query_plan(ops, qt->window.after, qt->window.before, qt->window.points)) {
513 + rrd2rrdr_query_ops_release(ops);
514 + return NULL;
515 + }
516 +
517 + return ops;
518 +}
src/web/api/queries/query.c
+651 -3174
@@ -1,3352 +1,826 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 -#include "query.h"
4 -#include "web/api/formatters/rrd2json.h"
5 -#include "rrdr.h"
6 -
7 -#include "average/average.h"
8 -#include "countif/countif.h"
9 -#include "incremental_sum/incremental_sum.h"
10 -#include "max/max.h"
11 -#include "median/median.h"
12 -#include "min/min.h"
13 -#include "sum/sum.h"
14 -#include "stddev/stddev.h"
15 -#include "ses/ses.h"
16 -#include "des/des.h"
17 -#include "percentile/percentile.h"
18 -#include "trimmed_mean/trimmed_mean.h"
19 -
20 -#define QUERY_PLAN_MIN_POINTS 10
21 -#define POINTS_TO_EXPAND_QUERY 5
3 +#include "query-internal.h"
4
5 // ----------------------------------------------------------------------------
6 +// helpers to find our way in RRDR
7
25 -static struct {
26 - const char *name;
27 - uint32_t hash;
28 - RRDR_TIME_GROUPING value;
29 - RRDR_TIME_GROUPING add_flush;
30 -
31 - // One time initialization for the module.
32 - // This is called once, when netdata starts.
33 - void (*init)(void);
34 -
35 - // Allocate all required structures for a query.
36 - // This is called once for each netdata query.
37 - void (*create)(struct rrdresult *r, const char *options);
38 -
39 - // Cleanup collected values, but don't destroy the structures.
40 - // This is called when the query engine switches dimensions,
41 - // as part of the same query (so same chart, switching metric).
42 - void (*reset)(struct rrdresult *r);
43 -
44 - // Free all resources allocated for the query.
45 - void (*free)(struct rrdresult *r);
46 -
47 - // Add a single value into the calculation.
48 - // The module may decide to cache it, or use it in the fly.
49 - void (*add)(struct rrdresult *r, NETDATA_DOUBLE value);
50 -
51 - // Generate a single result for the values added so far.
52 - // More values and points may be requested later.
53 - // It is up to the module to reset its internal structures
54 - // when flushing it (so for a few modules it may be better to
55 - // continue after a flush as if nothing changed, for others a
56 - // cleanup of the internal structures may be required).
57 - NETDATA_DOUBLE (*flush)(struct rrdresult *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr);
58 -
59 - TIER_QUERY_FETCH tier_query_fetch;
60 -} api_v1_data_groups[] = {
61 - {.name = "average",
62 - .hash = 0,
63 - .value = RRDR_GROUPING_AVERAGE,
64 - .add_flush = RRDR_GROUPING_AVERAGE,
65 - .init = NULL,
66 - .create= tg_average_create,
67 - .reset = tg_average_reset,
68 - .free = tg_average_free,
69 - .add = tg_average_add,
70 - .flush = tg_average_flush,
71 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
72 - },
73 - {.name = "avg", // alias on 'average'
74 - .hash = 0,
75 - .value = RRDR_GROUPING_AVERAGE,
76 - .add_flush = RRDR_GROUPING_AVERAGE,
77 - .init = NULL,
78 - .create= tg_average_create,
79 - .reset = tg_average_reset,
80 - .free = tg_average_free,
81 - .add = tg_average_add,
82 - .flush = tg_average_flush,
83 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
84 - },
85 - {.name = "mean", // alias on 'average'
86 - .hash = 0,
87 - .value = RRDR_GROUPING_AVERAGE,
88 - .add_flush = RRDR_GROUPING_AVERAGE,
89 - .init = NULL,
90 - .create= tg_average_create,
91 - .reset = tg_average_reset,
92 - .free = tg_average_free,
93 - .add = tg_average_add,
94 - .flush = tg_average_flush,
95 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
96 - },
97 - {.name = "trimmed-mean1",
98 - .hash = 0,
99 - .value = RRDR_GROUPING_TRIMMED_MEAN1,
100 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
101 - .init = NULL,
102 - .create= tg_trimmed_mean_create_1,
103 - .reset = tg_trimmed_mean_reset,
104 - .free = tg_trimmed_mean_free,
105 - .add = tg_trimmed_mean_add,
106 - .flush = tg_trimmed_mean_flush,
107 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
108 - },
109 - {.name = "trimmed-mean2",
110 - .hash = 0,
111 - .value = RRDR_GROUPING_TRIMMED_MEAN2,
112 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
113 - .init = NULL,
114 - .create= tg_trimmed_mean_create_2,
115 - .reset = tg_trimmed_mean_reset,
116 - .free = tg_trimmed_mean_free,
117 - .add = tg_trimmed_mean_add,
118 - .flush = tg_trimmed_mean_flush,
119 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
120 - },
121 - {.name = "trimmed-mean3",
122 - .hash = 0,
123 - .value = RRDR_GROUPING_TRIMMED_MEAN3,
124 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
125 - .init = NULL,
126 - .create= tg_trimmed_mean_create_3,
127 - .reset = tg_trimmed_mean_reset,
128 - .free = tg_trimmed_mean_free,
129 - .add = tg_trimmed_mean_add,
130 - .flush = tg_trimmed_mean_flush,
131 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
132 - },
133 - {.name = "trimmed-mean5",
134 - .hash = 0,
135 - .value = RRDR_GROUPING_TRIMMED_MEAN,
136 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
137 - .init = NULL,
138 - .create= tg_trimmed_mean_create_5,
139 - .reset = tg_trimmed_mean_reset,
140 - .free = tg_trimmed_mean_free,
141 - .add = tg_trimmed_mean_add,
142 - .flush = tg_trimmed_mean_flush,
143 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
144 - },
145 - {.name = "trimmed-mean10",
146 - .hash = 0,
147 - .value = RRDR_GROUPING_TRIMMED_MEAN10,
148 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
149 - .init = NULL,
150 - .create= tg_trimmed_mean_create_10,
151 - .reset = tg_trimmed_mean_reset,
152 - .free = tg_trimmed_mean_free,
153 - .add = tg_trimmed_mean_add,
154 - .flush = tg_trimmed_mean_flush,
155 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
156 - },
157 - {.name = "trimmed-mean15",
158 - .hash = 0,
159 - .value = RRDR_GROUPING_TRIMMED_MEAN15,
160 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
161 - .init = NULL,
162 - .create= tg_trimmed_mean_create_15,
163 - .reset = tg_trimmed_mean_reset,
164 - .free = tg_trimmed_mean_free,
165 - .add = tg_trimmed_mean_add,
166 - .flush = tg_trimmed_mean_flush,
167 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
168 - },
169 - {.name = "trimmed-mean20",
170 - .hash = 0,
171 - .value = RRDR_GROUPING_TRIMMED_MEAN20,
172 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
173 - .init = NULL,
174 - .create= tg_trimmed_mean_create_20,
175 - .reset = tg_trimmed_mean_reset,
176 - .free = tg_trimmed_mean_free,
177 - .add = tg_trimmed_mean_add,
178 - .flush = tg_trimmed_mean_flush,
179 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
180 - },
181 - {.name = "trimmed-mean25",
182 - .hash = 0,
183 - .value = RRDR_GROUPING_TRIMMED_MEAN25,
184 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
185 - .init = NULL,
186 - .create= tg_trimmed_mean_create_25,
187 - .reset = tg_trimmed_mean_reset,
188 - .free = tg_trimmed_mean_free,
189 - .add = tg_trimmed_mean_add,
190 - .flush = tg_trimmed_mean_flush,
191 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
192 - },
193 - {.name = "trimmed-mean",
194 - .hash = 0,
195 - .value = RRDR_GROUPING_TRIMMED_MEAN,
196 - .add_flush = RRDR_GROUPING_TRIMMED_MEAN,
197 - .init = NULL,
198 - .create= tg_trimmed_mean_create_5,
199 - .reset = tg_trimmed_mean_reset,
200 - .free = tg_trimmed_mean_free,
201 - .add = tg_trimmed_mean_add,
202 - .flush = tg_trimmed_mean_flush,
203 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
204 - },
205 - {.name = "incremental_sum",
206 - .hash = 0,
207 - .value = RRDR_GROUPING_INCREMENTAL_SUM,
208 - .add_flush = RRDR_GROUPING_INCREMENTAL_SUM,
209 - .init = NULL,
210 - .create= tg_incremental_sum_create,
211 - .reset = tg_incremental_sum_reset,
212 - .free = tg_incremental_sum_free,
213 - .add = tg_incremental_sum_add,
214 - .flush = tg_incremental_sum_flush,
215 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
216 - },
217 - {.name = "incremental-sum",
218 - .hash = 0,
219 - .value = RRDR_GROUPING_INCREMENTAL_SUM,
220 - .add_flush = RRDR_GROUPING_INCREMENTAL_SUM,
221 - .init = NULL,
222 - .create= tg_incremental_sum_create,
223 - .reset = tg_incremental_sum_reset,
224 - .free = tg_incremental_sum_free,
225 - .add = tg_incremental_sum_add,
226 - .flush = tg_incremental_sum_flush,
227 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
228 - },
229 - {.name = "median",
230 - .hash = 0,
231 - .value = RRDR_GROUPING_MEDIAN,
232 - .add_flush = RRDR_GROUPING_MEDIAN,
233 - .init = NULL,
234 - .create= tg_median_create,
235 - .reset = tg_median_reset,
236 - .free = tg_median_free,
237 - .add = tg_median_add,
238 - .flush = tg_median_flush,
239 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
240 - },
241 - {.name = "trimmed-median1",
242 - .hash = 0,
243 - .value = RRDR_GROUPING_TRIMMED_MEDIAN1,
244 - .add_flush = RRDR_GROUPING_MEDIAN,
245 - .init = NULL,
246 - .create= tg_median_create_trimmed_1,
247 - .reset = tg_median_reset,
248 - .free = tg_median_free,
249 - .add = tg_median_add,
250 - .flush = tg_median_flush,
251 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
252 - },
253 - {.name = "trimmed-median2",
254 - .hash = 0,
255 - .value = RRDR_GROUPING_TRIMMED_MEDIAN2,
256 - .add_flush = RRDR_GROUPING_MEDIAN,
257 - .init = NULL,
258 - .create= tg_median_create_trimmed_2,
259 - .reset = tg_median_reset,
260 - .free = tg_median_free,
261 - .add = tg_median_add,
262 - .flush = tg_median_flush,
263 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
264 - },
265 - {.name = "trimmed-median3",
266 - .hash = 0,
267 - .value = RRDR_GROUPING_TRIMMED_MEDIAN3,
268 - .add_flush = RRDR_GROUPING_MEDIAN,
269 - .init = NULL,
270 - .create= tg_median_create_trimmed_3,
271 - .reset = tg_median_reset,
272 - .free = tg_median_free,
273 - .add = tg_median_add,
274 - .flush = tg_median_flush,
275 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
276 - },
277 - {.name = "trimmed-median5",
278 - .hash = 0,
279 - .value = RRDR_GROUPING_TRIMMED_MEDIAN,
280 - .add_flush = RRDR_GROUPING_MEDIAN,
281 - .init = NULL,
282 - .create= tg_median_create_trimmed_5,
283 - .reset = tg_median_reset,
284 - .free = tg_median_free,
285 - .add = tg_median_add,
286 - .flush = tg_median_flush,
287 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
288 - },
289 - {.name = "trimmed-median10",
290 - .hash = 0,
291 - .value = RRDR_GROUPING_TRIMMED_MEDIAN10,
292 - .add_flush = RRDR_GROUPING_MEDIAN,
293 - .init = NULL,
294 - .create= tg_median_create_trimmed_10,
295 - .reset = tg_median_reset,
296 - .free = tg_median_free,
297 - .add = tg_median_add,
298 - .flush = tg_median_flush,
299 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
300 - },
301 - {.name = "trimmed-median15",
302 - .hash = 0,
303 - .value = RRDR_GROUPING_TRIMMED_MEDIAN15,
304 - .add_flush = RRDR_GROUPING_MEDIAN,
305 - .init = NULL,
306 - .create= tg_median_create_trimmed_15,
307 - .reset = tg_median_reset,
308 - .free = tg_median_free,
309 - .add = tg_median_add,
310 - .flush = tg_median_flush,
311 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
312 - },
313 - {.name = "trimmed-median20",
314 - .hash = 0,
315 - .value = RRDR_GROUPING_TRIMMED_MEDIAN20,
316 - .add_flush = RRDR_GROUPING_MEDIAN,
317 - .init = NULL,
318 - .create= tg_median_create_trimmed_20,
319 - .reset = tg_median_reset,
320 - .free = tg_median_free,
321 - .add = tg_median_add,
322 - .flush = tg_median_flush,
323 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
324 - },
325 - {.name = "trimmed-median25",
326 - .hash = 0,
327 - .value = RRDR_GROUPING_TRIMMED_MEDIAN25,
328 - .add_flush = RRDR_GROUPING_MEDIAN,
329 - .init = NULL,
330 - .create= tg_median_create_trimmed_25,
331 - .reset = tg_median_reset,
332 - .free = tg_median_free,
333 - .add = tg_median_add,
334 - .flush = tg_median_flush,
335 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
336 - },
337 - {.name = "trimmed-median",
338 - .hash = 0,
339 - .value = RRDR_GROUPING_TRIMMED_MEDIAN,
340 - .add_flush = RRDR_GROUPING_MEDIAN,
341 - .init = NULL,
342 - .create= tg_median_create_trimmed_5,
343 - .reset = tg_median_reset,
344 - .free = tg_median_free,
345 - .add = tg_median_add,
346 - .flush = tg_median_flush,
347 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
348 - },
349 - {.name = "percentile25",
350 - .hash = 0,
351 - .value = RRDR_GROUPING_PERCENTILE25,
352 - .add_flush = RRDR_GROUPING_PERCENTILE,
353 - .init = NULL,
354 - .create= tg_percentile_create_25,
355 - .reset = tg_percentile_reset,
356 - .free = tg_percentile_free,
357 - .add = tg_percentile_add,
358 - .flush = tg_percentile_flush,
359 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
360 - },
361 - {.name = "percentile50",
362 - .hash = 0,
363 - .value = RRDR_GROUPING_PERCENTILE50,
364 - .add_flush = RRDR_GROUPING_PERCENTILE,
365 - .init = NULL,
366 - .create= tg_percentile_create_50,
367 - .reset = tg_percentile_reset,
368 - .free = tg_percentile_free,
369 - .add = tg_percentile_add,
370 - .flush = tg_percentile_flush,
371 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
372 - },
373 - {.name = "percentile75",
374 - .hash = 0,
375 - .value = RRDR_GROUPING_PERCENTILE75,
376 - .add_flush = RRDR_GROUPING_PERCENTILE,
377 - .init = NULL,
378 - .create= tg_percentile_create_75,
379 - .reset = tg_percentile_reset,
380 - .free = tg_percentile_free,
381 - .add = tg_percentile_add,
382 - .flush = tg_percentile_flush,
383 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
384 - },
385 - {.name = "percentile80",
386 - .hash = 0,
387 - .value = RRDR_GROUPING_PERCENTILE80,
388 - .add_flush = RRDR_GROUPING_PERCENTILE,
389 - .init = NULL,
390 - .create= tg_percentile_create_80,
391 - .reset = tg_percentile_reset,
392 - .free = tg_percentile_free,
393 - .add = tg_percentile_add,
394 - .flush = tg_percentile_flush,
395 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
396 - },
397 - {.name = "percentile90",
398 - .hash = 0,
399 - .value = RRDR_GROUPING_PERCENTILE90,
400 - .add_flush = RRDR_GROUPING_PERCENTILE,
401 - .init = NULL,
402 - .create= tg_percentile_create_90,
403 - .reset = tg_percentile_reset,
404 - .free = tg_percentile_free,
405 - .add = tg_percentile_add,
406 - .flush = tg_percentile_flush,
407 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
408 - },
409 - {.name = "percentile95",
410 - .hash = 0,
411 - .value = RRDR_GROUPING_PERCENTILE,
412 - .add_flush = RRDR_GROUPING_PERCENTILE,
413 - .init = NULL,
414 - .create= tg_percentile_create_95,
415 - .reset = tg_percentile_reset,
416 - .free = tg_percentile_free,
417 - .add = tg_percentile_add,
418 - .flush = tg_percentile_flush,
419 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
420 - },
421 - {.name = "percentile97",
422 - .hash = 0,
423 - .value = RRDR_GROUPING_PERCENTILE97,
424 - .add_flush = RRDR_GROUPING_PERCENTILE,
425 - .init = NULL,
426 - .create= tg_percentile_create_97,
427 - .reset = tg_percentile_reset,
428 - .free = tg_percentile_free,
429 - .add = tg_percentile_add,
430 - .flush = tg_percentile_flush,
431 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
432 - },
433 - {.name = "percentile98",
434 - .hash = 0,
435 - .value = RRDR_GROUPING_PERCENTILE98,
436 - .add_flush = RRDR_GROUPING_PERCENTILE,
437 - .init = NULL,
438 - .create= tg_percentile_create_98,
439 - .reset = tg_percentile_reset,
440 - .free = tg_percentile_free,
441 - .add = tg_percentile_add,
442 - .flush = tg_percentile_flush,
443 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
444 - },
445 - {.name = "percentile99",
446 - .hash = 0,
447 - .value = RRDR_GROUPING_PERCENTILE99,
448 - .add_flush = RRDR_GROUPING_PERCENTILE,
449 - .init = NULL,
450 - .create= tg_percentile_create_99,
451 - .reset = tg_percentile_reset,
452 - .free = tg_percentile_free,
453 - .add = tg_percentile_add,
454 - .flush = tg_percentile_flush,
455 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
456 - },
457 - {.name = "percentile",
458 - .hash = 0,
459 - .value = RRDR_GROUPING_PERCENTILE,
460 - .add_flush = RRDR_GROUPING_PERCENTILE,
461 - .init = NULL,
462 - .create= tg_percentile_create_95,
463 - .reset = tg_percentile_reset,
464 - .free = tg_percentile_free,
465 - .add = tg_percentile_add,
466 - .flush = tg_percentile_flush,
467 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
468 - },
469 - {.name = "min",
470 - .hash = 0,
471 - .value = RRDR_GROUPING_MIN,
472 - .add_flush = RRDR_GROUPING_MIN,
473 - .init = NULL,
474 - .create= tg_min_create,
475 - .reset = tg_min_reset,
476 - .free = tg_min_free,
477 - .add = tg_min_add,
478 - .flush = tg_min_flush,
479 - .tier_query_fetch = TIER_QUERY_FETCH_MIN
480 - },
481 - {.name = "max",
482 - .hash = 0,
483 - .value = RRDR_GROUPING_MAX,
484 - .add_flush = RRDR_GROUPING_MAX,
485 - .init = NULL,
486 - .create= tg_max_create,
487 - .reset = tg_max_reset,
488 - .free = tg_max_free,
489 - .add = tg_max_add,
490 - .flush = tg_max_flush,
491 - .tier_query_fetch = TIER_QUERY_FETCH_MAX
492 - },
493 - {.name = "sum",
494 - .hash = 0,
495 - .value = RRDR_GROUPING_SUM,
496 - .add_flush = RRDR_GROUPING_SUM,
497 - .init = NULL,
498 - .create= tg_sum_create,
499 - .reset = tg_sum_reset,
500 - .free = tg_sum_free,
501 - .add = tg_sum_add,
502 - .flush = tg_sum_flush,
503 - .tier_query_fetch = TIER_QUERY_FETCH_SUM
504 - },
505 -
506 - // standard deviation
507 - {.name = "stddev",
508 - .hash = 0,
509 - .value = RRDR_GROUPING_STDDEV,
510 - .add_flush = RRDR_GROUPING_STDDEV,
511 - .init = NULL,
512 - .create= tg_stddev_create,
513 - .reset = tg_stddev_reset,
514 - .free = tg_stddev_free,
515 - .add = tg_stddev_add,
516 - .flush = tg_stddev_flush,
517 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
518 - },
519 - {.name = "cv", // coefficient variation is calculated by stddev
520 - .hash = 0,
521 - .value = RRDR_GROUPING_CV,
522 - .add_flush = RRDR_GROUPING_CV,
523 - .init = NULL,
524 - .create= tg_stddev_create, // not an error, stddev calculates this too
525 - .reset = tg_stddev_reset, // not an error, stddev calculates this too
526 - .free = tg_stddev_free, // not an error, stddev calculates this too
527 - .add = tg_stddev_add, // not an error, stddev calculates this too
528 - .flush = tg_stddev_coefficient_of_variation_flush,
529 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
530 - },
531 - {.name = "rsd", // alias of 'cv'
532 - .hash = 0,
533 - .value = RRDR_GROUPING_CV,
534 - .add_flush = RRDR_GROUPING_CV,
535 - .init = NULL,
536 - .create= tg_stddev_create, // not an error, stddev calculates this too
537 - .reset = tg_stddev_reset, // not an error, stddev calculates this too
538 - .free = tg_stddev_free, // not an error, stddev calculates this too
539 - .add = tg_stddev_add, // not an error, stddev calculates this too
540 - .flush = tg_stddev_coefficient_of_variation_flush,
541 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
542 - },
543 -
544 - // single exponential smoothing
545 - {.name = "ses",
546 - .hash = 0,
547 - .value = RRDR_GROUPING_SES,
548 - .add_flush = RRDR_GROUPING_SES,
549 - .init = tg_ses_init,
550 - .create= tg_ses_create,
551 - .reset = tg_ses_reset,
552 - .free = tg_ses_free,
553 - .add = tg_ses_add,
554 - .flush = tg_ses_flush,
555 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
556 - },
557 - {.name = "ema", // alias for 'ses'
558 - .hash = 0,
559 - .value = RRDR_GROUPING_SES,
560 - .add_flush = RRDR_GROUPING_SES,
561 - .init = NULL,
562 - .create= tg_ses_create,
563 - .reset = tg_ses_reset,
564 - .free = tg_ses_free,
565 - .add = tg_ses_add,
566 - .flush = tg_ses_flush,
567 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
568 - },
569 - {.name = "ewma", // alias for ses
570 - .hash = 0,
571 - .value = RRDR_GROUPING_SES,
572 - .add_flush = RRDR_GROUPING_SES,
573 - .init = NULL,
574 - .create= tg_ses_create,
575 - .reset = tg_ses_reset,
576 - .free = tg_ses_free,
577 - .add = tg_ses_add,
578 - .flush = tg_ses_flush,
579 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
580 - },
581 -
582 - // double exponential smoothing
583 - {.name = "des",
584 - .hash = 0,
585 - .value = RRDR_GROUPING_DES,
586 - .add_flush = RRDR_GROUPING_DES,
587 - .init = tg_des_init,
588 - .create= tg_des_create,
589 - .reset = tg_des_reset,
590 - .free = tg_des_free,
591 - .add = tg_des_add,
592 - .flush = tg_des_flush,
593 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
594 - },
595 -
596 - {.name = "countif",
597 - .hash = 0,
598 - .value = RRDR_GROUPING_COUNTIF,
599 - .add_flush = RRDR_GROUPING_COUNTIF,
600 - .init = NULL,
601 - .create= tg_countif_create,
602 - .reset = tg_countif_reset,
603 - .free = tg_countif_free,
604 - .add = tg_countif_add,
605 - .flush = tg_countif_flush,
606 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
607 - },
608 -
609 - // terminator
610 - {.name = NULL,
611 - .hash = 0,
612 - .value = RRDR_GROUPING_UNDEFINED,
613 - .add_flush = RRDR_GROUPING_AVERAGE,
614 - .init = NULL,
615 - .create= tg_average_create,
616 - .reset = tg_average_reset,
617 - .free = tg_average_free,
618 - .add = tg_average_add,
619 - .flush = tg_average_flush,
620 - .tier_query_fetch = TIER_QUERY_FETCH_AVERAGE
621 - }
622 -};
623 -
624 -void time_grouping_init(void) {
625 - int i;
626 -
627 - for(i = 0; api_v1_data_groups[i].name ; i++) {
628 - api_v1_data_groups[i].hash = simple_hash(api_v1_data_groups[i].name);
629 -
630 - if(api_v1_data_groups[i].init)
631 - api_v1_data_groups[i].init();
632 - }
8 +ALWAYS_INLINE
9 +static RRDR_VALUE_FLAGS *UNUSED_FUNCTION(rrdr_line_options)(RRDR *r, long rrdr_line) {
10 + return &r->o[ rrdr_line * r->d ];
11 }
12
635 -const char *time_grouping_id2txt(RRDR_TIME_GROUPING group) {
636 - int i;
637 -
638 - for(i = 0; api_v1_data_groups[i].name ; i++) {
639 - if(api_v1_data_groups[i].value == group) {
640 - return api_v1_data_groups[i].name;
641 - }
642 - }
643 -
644 - return "average";
13 +ALWAYS_INLINE
14 +static NETDATA_DOUBLE *UNUSED_FUNCTION(rrdr_line_values)(RRDR *r, long rrdr_line) {
15 + return &r->v[ rrdr_line * r->d ];
16 }
17
647 -RRDR_TIME_GROUPING time_grouping_txt2id(const char *name) {
648 - int i;
649 -
650 - uint32_t hash = simple_hash(name);
651 - for(i = 0; api_v1_data_groups[i].name ; i++)
652 - if(unlikely(hash == api_v1_data_groups[i].hash && !strcmp(name, api_v1_data_groups[i].name)))
653 - return api_v1_data_groups[i].value;
654 -
655 - return RRDR_GROUPING_AVERAGE;
656 -}
18 +ALWAYS_INLINE
19 +static long rrdr_line_init(RRDR *r __maybe_unused, time_t t __maybe_unused, long rrdr_line) {
20 + rrdr_line++;
21
658 -RRDR_TIME_GROUPING time_grouping_parse(const char *name, RRDR_TIME_GROUPING def) {
659 - int i;
22 + internal_fatal(rrdr_line >= (long)r->n,
23 + "QUERY: requested to step above RRDR size for query '%s'",
24 + r->internal.qt->id);
25
661 - uint32_t hash = simple_hash(name);
662 - for(i = 0; api_v1_data_groups[i].name ; i++)
663 - if(unlikely(hash == api_v1_data_groups[i].hash && !strcmp(name, api_v1_data_groups[i].name)))
664 - return api_v1_data_groups[i].value;
26 + internal_fatal(r->t[rrdr_line] != t,
27 + "QUERY: wrong timestamp at RRDR line %ld, expected %ld, got %ld, of query '%s'",
28 + rrdr_line, r->t[rrdr_line], t, r->internal.qt->id);
29
666 - return def;
30 + return rrdr_line;
31 }
32
669 -const char *time_grouping_tostring(RRDR_TIME_GROUPING group) {
670 - int i;
671 -
672 - for(i = 0; api_v1_data_groups[i].name ; i++)
673 - if(unlikely(group == api_v1_data_groups[i].value))
674 - return api_v1_data_groups[i].name;
33 +// ----------------------------------------------------------------------------
34 +// dimension level query engine
35
676 - return "unknown";
677 -}
36 +#define query_interpolate_point(this_point, last_point, now) do { \
37 + if(likely( \
38 + /* the point to interpolate is more than 1s wide */ \
39 + (this_point).sp.end_time_s - (this_point).sp.start_time_s > 1 \
40 + \
41 + /* the two points are exactly next to each other */ \
42 + && (last_point).sp.end_time_s == (this_point).sp.start_time_s \
43 + \
44 + /* both points are valid numbers */ \
45 + && netdata_double_isnumber((this_point).value) \
46 + && netdata_double_isnumber((last_point).value) \
47 + \
48 + )) { \
49 + (this_point).value = (last_point).value + ((this_point).value - (last_point).value) * (1.0 - (NETDATA_DOUBLE)((this_point).sp.end_time_s - (now)) / (NETDATA_DOUBLE)((this_point).sp.end_time_s - (this_point).sp.start_time_s)); \
50 + (this_point).sp.end_time_s = now; \
51 + } \
52 +} while(0)
53
679 -static void rrdr_set_grouping_function(RRDR *r, RRDR_TIME_GROUPING group_method) {
680 - int i, found = 0;
681 - for(i = 0; !found && api_v1_data_groups[i].name ;i++) {
682 - if(api_v1_data_groups[i].value == group_method) {
683 - r->time_grouping.create = api_v1_data_groups[i].create;
684 - r->time_grouping.reset = api_v1_data_groups[i].reset;
685 - r->time_grouping.free = api_v1_data_groups[i].free;
686 - r->time_grouping.add = api_v1_data_groups[i].add;
687 - r->time_grouping.flush = api_v1_data_groups[i].flush;
688 - r->time_grouping.tier_query_fetch = api_v1_data_groups[i].tier_query_fetch;
689 - r->time_grouping.add_flush = api_v1_data_groups[i].add_flush;
690 - found = 1;
691 - }
692 - }
693 - if(!found) {
694 - errno_clear();
695 - internal_error(true, "QUERY: grouping method %u not found. Using 'average'", (unsigned int)group_method);
696 - r->time_grouping.create = tg_average_create;
697 - r->time_grouping.reset = tg_average_reset;
698 - r->time_grouping.free = tg_average_free;
699 - r->time_grouping.add = tg_average_add;
700 - r->time_grouping.flush = tg_average_flush;
701 - r->time_grouping.tier_query_fetch = TIER_QUERY_FETCH_AVERAGE;
702 - r->time_grouping.add_flush = RRDR_GROUPING_AVERAGE;
703 - }
704 -}
54 +#define query_add_point_to_group(r, point, ops, add_flush) do { \
55 + if(likely(netdata_double_isnumber((point).value))) { \
56 + if(likely(fpclassify((point).value) != FP_ZERO)) \
57 + (ops)->group_points_non_zero++; \
58 + \
59 + if(unlikely((point).sp.flags & SN_FLAG_RESET)) \
60 + (ops)->group_value_flags |= RRDR_VALUE_RESET; \
61 + \
62 + time_grouping_add(r, (point).value, add_flush); \
63 + \
64 + storage_point_merge_to((ops)->group_point, (point).sp); \
65 + if(!(point).added) \
66 + storage_point_merge_to((ops)->query_point, (point).sp); \
67 + } \
68 + \
69 + (ops)->group_points_added++; \
70 +} while(0)
71
706 -ALWAYS_INLINE_HOT_FLATTEN
707 -static void time_grouping_add(RRDR *r, NETDATA_DOUBLE value, const RRDR_TIME_GROUPING add_flush) {
708 - switch(add_flush) {
709 - case RRDR_GROUPING_AVERAGE:
710 - tg_average_add(r, value);
711 - break;
72 +NOT_INLINE_HOT static void rrd2rrdr_query_execute(RRDR *r, size_t dim_id_in_rrdr, QUERY_ENGINE_OPS *ops) {
73 + QUERY_TARGET *qt = r->internal.qt;
74 + QUERY_METRIC *qm = ops->qm;
75
713 - case RRDR_GROUPING_MAX:
714 - tg_max_add(r, value);
715 - break;
76 + const RRDR_TIME_GROUPING add_flush = r->time_grouping.add_flush;
77
717 - case RRDR_GROUPING_MIN:
718 - tg_min_add(r, value);
719 - break;
78 + ops->group_point = STORAGE_POINT_UNSET;
79 + ops->query_point = STORAGE_POINT_UNSET;
80
721 - case RRDR_GROUPING_MEDIAN:
722 - tg_median_add(r, value);
723 - break;
81 + RRDR_OPTIONS options = qt->window.options;
82 + size_t points_wanted = qt->window.points;
83 + time_t after_wanted = qt->window.after;
84 + time_t before_wanted = qt->window.before; (void)before_wanted;
85
725 - case RRDR_GROUPING_STDDEV:
726 - case RRDR_GROUPING_CV:
727 - tg_stddev_add(r, value);
728 - break;
86 +// bool debug_this = false;
87 +// if(strcmp("user", string2str(rd->id)) == 0 && strcmp("system.cpu", string2str(rd->rrdset->id)) == 0)
88 +// debug_this = true;
89
730 - case RRDR_GROUPING_SUM:
731 - tg_sum_add(r, value);
732 - break;
90 + size_t points_added = 0;
91
734 - case RRDR_GROUPING_COUNTIF:
735 - tg_countif_add(r, value);
736 - break;
92 + long rrdr_line = -1;
93 + bool use_anomaly_bit_as_value = (r->internal.qt->window.options & RRDR_OPTION_ANOMALY_BIT) ? true : false;
94
738 - case RRDR_GROUPING_TRIMMED_MEAN:
739 - tg_trimmed_mean_add(r, value);
740 - break;
95 + NETDATA_DOUBLE min = r->view.min, max = r->view.max;
96
742 - case RRDR_GROUPING_PERCENTILE:
743 - tg_percentile_add(r, value);
744 - break;
97 + QUERY_POINT last2_point = QUERY_POINT_EMPTY;
98 + QUERY_POINT last1_point = QUERY_POINT_EMPTY;
99 + QUERY_POINT new_point = QUERY_POINT_EMPTY;
100
746 - case RRDR_GROUPING_SES:
747 - tg_ses_add(r, value);
748 - break;
101 + // ONE POINT READ-AHEAD
102 + // when we switch plans, we read-ahead a point from the next plan
103 + // to join them smoothly at the exact time the next plan begins
104 + STORAGE_POINT next1_point = STORAGE_POINT_UNSET;
105
750 - case RRDR_GROUPING_DES:
751 - tg_des_add(r, value);
752 - break;
106 + time_t now_start_time = after_wanted - ops->query_granularity;
107 + time_t now_end_time = after_wanted + ops->view_update_every - ops->query_granularity;
108
754 - case RRDR_GROUPING_INCREMENTAL_SUM:
755 - tg_incremental_sum_add(r, value);
756 - break;
109 + size_t db_points_read_since_plan_switch = 0; (void)db_points_read_since_plan_switch;
110 + size_t query_is_finished_counter = 0;
111
758 - default:
759 - r->time_grouping.add(r, value);
760 - break;
761 - }
762 -}
112 + // The main loop, based on the query granularity we need
113 + for( ; points_added < points_wanted && query_is_finished_counter <= 10 ;
114 + now_start_time = now_end_time, now_end_time += ops->view_update_every) {
115
764 -ALWAYS_INLINE_HOT_FLATTEN
765 -static NETDATA_DOUBLE time_grouping_flush(RRDR *r, RRDR_VALUE_FLAGS *rrdr_value_options_ptr, const RRDR_TIME_GROUPING add_flush) {
766 - switch(add_flush) {
767 - case RRDR_GROUPING_AVERAGE:
768 - return tg_average_flush(r, rrdr_value_options_ptr);
116 + if(unlikely(query_plan_should_switch_plan(ops, now_end_time))) {
117 + query_planer_next_plan(ops, now_end_time, new_point.sp.end_time_s);
118 + db_points_read_since_plan_switch = 0;
119 + }
120
770 - case RRDR_GROUPING_MAX:
771 - return tg_max_flush(r, rrdr_value_options_ptr);
121 + // read all the points of the db, prior to the time we need (now_end_time)
122
773 - case RRDR_GROUPING_MIN:
774 - return tg_min_flush(r, rrdr_value_options_ptr);
123 + size_t count_same_end_time = 0;
124 + while(count_same_end_time < 100) {
125 + if(likely(count_same_end_time == 0)) {
126 + last2_point = last1_point;
127 + last1_point = new_point;
128 + }
129
776 - case RRDR_GROUPING_MEDIAN:
777 - return tg_median_flush(r, rrdr_value_options_ptr);
130 + if(unlikely(storage_engine_query_is_finished(ops->seqh))) {
131 + query_is_finished_counter++;
132
779 - case RRDR_GROUPING_STDDEV:
780 - return tg_stddev_flush(r, rrdr_value_options_ptr);
133 + if(count_same_end_time != 0) {
134 + last2_point = last1_point;
135 + last1_point = new_point;
136 + }
137 + new_point = QUERY_POINT_EMPTY;
138 + new_point.sp.start_time_s = last1_point.sp.end_time_s;
139 + new_point.sp.end_time_s = now_end_time;
140 +//
141 +// if(debug_this) netdata_log_info("QUERY: is finished() returned true");
142 +//
143 + break;
144 + }
145 + else
146 + query_is_finished_counter = 0;
147
782 - case RRDR_GROUPING_CV:
783 - return tg_stddev_coefficient_of_variation_flush(r, rrdr_value_options_ptr);
148 + // fetch the new point
149 + {
150 + STORAGE_POINT sp;
151 + if(likely(storage_point_is_unset(next1_point))) {
152 + db_points_read_since_plan_switch++;
153 + sp = storage_engine_query_next_metric(ops->seqh);
154 + ops->db_points_read_per_tier[ops->tier]++;
155 + ops->db_total_points_read++;
156
785 - case RRDR_GROUPING_SUM:
786 - return tg_sum_flush(r, rrdr_value_options_ptr);
157 + if(unlikely(options & RRDR_OPTION_ABSOLUTE))
158 + storage_point_make_positive(sp);
159 + }
160 + else {
161 + // ONE POINT READ-AHEAD
162 + sp = next1_point;
163 + storage_point_unset(next1_point);
164 + db_points_read_since_plan_switch = 1;
165 + }
166
788 - case RRDR_GROUPING_COUNTIF:
789 - return tg_countif_flush(r, rrdr_value_options_ptr);
167 + // ONE POINT READ-AHEAD
168 + if(unlikely(query_plan_should_switch_plan(ops, sp.end_time_s) &&
169 + query_planer_next_plan(ops, now_end_time, new_point.sp.end_time_s))) {
170
791 - case RRDR_GROUPING_TRIMMED_MEAN:
792 - return tg_trimmed_mean_flush(r, rrdr_value_options_ptr);
171 + // The end time of the current point, crosses our plans (tiers)
172 + // so, we switched plan (tier)
173 + //
174 + // There are 2 cases now:
175 + //
176 + // A. the entire point of the previous plan is to the future of point from the next plan
177 + // B. part of the point of the previous plan overlaps with the point from the next plan
178
794 - case RRDR_GROUPING_PERCENTILE:
795 - return tg_percentile_flush(r, rrdr_value_options_ptr);
179 + STORAGE_POINT sp2 = storage_engine_query_next_metric(ops->seqh);
180 + ops->db_points_read_per_tier[ops->tier]++;
181 + ops->db_total_points_read++;
182
797 - case RRDR_GROUPING_SES:
798 - return tg_ses_flush(r, rrdr_value_options_ptr);
183 + if(unlikely(options & RRDR_OPTION_ABSOLUTE))
184 + storage_point_make_positive(sp);
185
800 - case RRDR_GROUPING_DES:
801 - return tg_des_flush(r, rrdr_value_options_ptr);
186 + if(sp.start_time_s > sp2.start_time_s)
187 + // the point from the previous plan is useless
188 + sp = sp2;
189 + else
190 + // let the query run from the previous plan
191 + // but setting this will also cut off the interpolation
192 + // of the point from the previous plan
193 + next1_point = sp2;
194 + }
195
803 - case RRDR_GROUPING_INCREMENTAL_SUM:
804 - return tg_incremental_sum_flush(r, rrdr_value_options_ptr);
196 + new_point.sp = sp;
197 + new_point.added = false;
198 + query_point_set_id(new_point, ops->db_total_points_read);
199
806 - default:
807 - return r->time_grouping.flush(r, rrdr_value_options_ptr);
808 - }
809 -}
200 +// if(debug_this)
201 +// netdata_log_info("QUERY: got point %zu, from time %ld to %ld // now from %ld to %ld // query from %ld to %ld",
202 +// new_point.id, new_point.start_time, new_point.end_time, now_start_time, now_end_time, after_wanted, before_wanted);
203 +//
204 + // get the right value from the point we got
205 + if(likely(!storage_point_is_unset(sp) && !storage_point_is_gap(sp))) {
206
811 -RRDR_GROUP_BY group_by_parse(char *s) {
812 - RRDR_GROUP_BY group_by = RRDR_GROUP_BY_NONE;
207 + if(unlikely(use_anomaly_bit_as_value))
208 + new_point.value = storage_point_anomaly_rate(new_point.sp);
209
814 - while(s) {
815 - char *key = strsep_skip_consecutive_separators(&s, ",| ");
816 - if (!key || !*key) continue;
210 + else {
211 + switch (ops->tier_query_fetch) {
212 + default:
213 + case TIER_QUERY_FETCH_AVERAGE:
214 + new_point.value = sp.sum / (NETDATA_DOUBLE)sp.count;
215 + break;
216
818 - if (strcmp(key, "selected") == 0)
819 - group_by |= RRDR_GROUP_BY_SELECTED;
217 + case TIER_QUERY_FETCH_MIN:
218 + new_point.value = sp.min;
219 + break;
220
821 - if (strcmp(key, "dimension") == 0)
822 - group_by |= RRDR_GROUP_BY_DIMENSION;
221 + case TIER_QUERY_FETCH_MAX:
222 + new_point.value = sp.max;
223 + break;
224
824 - if (strcmp(key, "instance") == 0)
825 - group_by |= RRDR_GROUP_BY_INSTANCE;
225 + case TIER_QUERY_FETCH_SUM:
226 + new_point.value = sp.sum;
227 + break;
228 + }
229 + }
230 + }
231 + else
232 + new_point.value = NAN;
233 + }
234
827 - if (strcmp(key, "percentage-of-instance") == 0)
828 - group_by |= RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE;
235 + // check if the db is giving us zero duration points
236 + if(unlikely(db_points_read_since_plan_switch > 1 &&
237 + new_point.sp.start_time_s == new_point.sp.end_time_s)) {
238
830 - if (strcmp(key, "label") == 0)
831 - group_by |= RRDR_GROUP_BY_LABEL;
239 + internal_error(true, "QUERY: '%s', dimension '%s' next_metric() returned "
240 + "point %zu from %ld to %ld, that are both equal",
241 + qt->id, query_metric_id(qt, qm),
242 + new_point.id, new_point.sp.start_time_s, new_point.sp.end_time_s);
243
833 - if (strcmp(key, "node") == 0)
834 - group_by |= RRDR_GROUP_BY_NODE;
244 + new_point.sp.start_time_s = new_point.sp.end_time_s - ops->tier_ptr->db_update_every_s;
245 + }
246
836 - if (strcmp(key, "context") == 0)
837 - group_by |= RRDR_GROUP_BY_CONTEXT;
247 + // check if the db is advancing the query
248 + if(unlikely(db_points_read_since_plan_switch > 1 &&
249 + new_point.sp.end_time_s <= last1_point.sp.end_time_s)) {
250
839 - if (strcmp(key, "units") == 0)
840 - group_by |= RRDR_GROUP_BY_UNITS;
841 - }
251 + internal_error(true,
252 + "QUERY: '%s', dimension '%s' next_metric() returned "
253 + "point %zu from %ld to %ld, before the "
254 + "last point %zu from %ld to %ld, "
255 + "now is %ld to %ld",
256 + qt->id, query_metric_id(qt, qm),
257 + new_point.id, new_point.sp.start_time_s, new_point.sp.end_time_s,
258 + last1_point.id, last1_point.sp.start_time_s, last1_point.sp.end_time_s,
259 + now_start_time, now_end_time);
260
843 - if((group_by & RRDR_GROUP_BY_SELECTED) && (group_by & ~RRDR_GROUP_BY_SELECTED)) {
844 - internal_error(true, "group-by given by query has 'selected' together with more groupings");
845 - group_by = RRDR_GROUP_BY_SELECTED; // remove all other groupings
846 - }
261 + count_same_end_time++;
262 + continue;
263 + }
264 + count_same_end_time = 0;
265
848 - if(group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
849 - group_by = RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE; // remove all other groupings
266 + // decide how to use this point
267 + if(likely(new_point.sp.end_time_s < now_end_time)) { // likely to favor tier0
268 + // this db point ends before our now_end_time
269
851 - return group_by;
852 -}
270 + if(likely(new_point.sp.end_time_s >= now_start_time)) { // likely to favor tier0
271 + // this db point ends after our now_start time
272
854 -void buffer_json_group_by_to_array(BUFFER *wb, RRDR_GROUP_BY group_by) {
855 - if(group_by == RRDR_GROUP_BY_NONE)
856 - buffer_json_add_array_item_string(wb, "none");
857 - else {
858 - if (group_by & RRDR_GROUP_BY_DIMENSION)
859 - buffer_json_add_array_item_string(wb, "dimension");
273 + query_add_point_to_group(r, new_point, ops, add_flush);
274 + new_point.added = true;
275 + }
276 + else {
277 + // we don't need this db point
278 + // it is totally outside our current time-frame
279
861 - if (group_by & RRDR_GROUP_BY_INSTANCE)
862 - buffer_json_add_array_item_string(wb, "instance");
280 + // this is desirable for the first point of the query
281 + // because it allows us to interpolate the next point
282 + // at exactly the time we will want
283
864 - if (group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
865 - buffer_json_add_array_item_string(wb, "percentage-of-instance");
866 -
867 - if (group_by & RRDR_GROUP_BY_LABEL)
868 - buffer_json_add_array_item_string(wb, "label");
869 -
870 - if (group_by & RRDR_GROUP_BY_NODE)
871 - buffer_json_add_array_item_string(wb, "node");
872 -
873 - if (group_by & RRDR_GROUP_BY_CONTEXT)
874 - buffer_json_add_array_item_string(wb, "context");
875 -
876 - if (group_by & RRDR_GROUP_BY_UNITS)
877 - buffer_json_add_array_item_string(wb, "units");
878 -
879 - if (group_by & RRDR_GROUP_BY_SELECTED)
880 - buffer_json_add_array_item_string(wb, "selected");
881 - }
882 -}
883 -
884 -RRDR_GROUP_BY_FUNCTION group_by_aggregate_function_parse(const char *s) {
885 - if(strcmp(s, "average") == 0)
886 - return RRDR_GROUP_BY_FUNCTION_AVERAGE;
887 -
888 - if(strcmp(s, "avg") == 0)
889 - return RRDR_GROUP_BY_FUNCTION_AVERAGE;
890 -
891 - if(strcmp(s, "min") == 0)
892 - return RRDR_GROUP_BY_FUNCTION_MIN;
893 -
894 - if(strcmp(s, "max") == 0)
895 - return RRDR_GROUP_BY_FUNCTION_MAX;
896 -
897 - if(strcmp(s, "sum") == 0)
898 - return RRDR_GROUP_BY_FUNCTION_SUM;
899 -
900 - if(strcmp(s, "percentage") == 0)
901 - return RRDR_GROUP_BY_FUNCTION_PERCENTAGE;
902 -
903 - return RRDR_GROUP_BY_FUNCTION_AVERAGE;
904 -}
905 -
906 -const char *group_by_aggregate_function_to_string(RRDR_GROUP_BY_FUNCTION group_by_function) {
907 - switch(group_by_function) {
908 - default:
909 - case RRDR_GROUP_BY_FUNCTION_AVERAGE:
910 - return "average";
911 -
912 - case RRDR_GROUP_BY_FUNCTION_MIN:
913 - return "min";
914 -
915 - case RRDR_GROUP_BY_FUNCTION_MAX:
916 - return "max";
917 -
918 - case RRDR_GROUP_BY_FUNCTION_SUM:
919 - return "sum";
920 -
921 - case RRDR_GROUP_BY_FUNCTION_PERCENTAGE:
922 - return "percentage";
923 - }
924 -}
925 -
926 -// ----------------------------------------------------------------------------
927 -// helpers to find our way in RRDR
928 -
929 -static inline RRDR_VALUE_FLAGS *UNUSED_FUNCTION(rrdr_line_options)(RRDR *r, long rrdr_line) {
930 - return &r->o[ rrdr_line * r->d ];
931 -}
932 -
933 -static inline NETDATA_DOUBLE *UNUSED_FUNCTION(rrdr_line_values)(RRDR *r, long rrdr_line) {
934 - return &r->v[ rrdr_line * r->d ];
935 -}
936 -
937 -static inline long rrdr_line_init(RRDR *r __maybe_unused, time_t t __maybe_unused, long rrdr_line) {
938 - rrdr_line++;
939 -
940 - internal_fatal(rrdr_line >= (long)r->n,
941 - "QUERY: requested to step above RRDR size for query '%s'",
942 - r->internal.qt->id);
943 -
944 - internal_fatal(r->t[rrdr_line] != t,
945 - "QUERY: wrong timestamp at RRDR line %ld, expected %ld, got %ld, of query '%s'",
946 - rrdr_line, r->t[rrdr_line], t, r->internal.qt->id);
947 -
948 - return rrdr_line;
949 -}
950 -
951 -// ----------------------------------------------------------------------------
952 -// tier management
953 -
954 -static bool query_metric_is_valid_tier(QUERY_METRIC *qm, size_t tier) {
955 - if(!qm->tiers[tier].smh || !qm->tiers[tier].db_first_time_s || !qm->tiers[tier].db_last_time_s || !qm->tiers[tier].db_update_every_s)
956 - return false;
957 -
958 - return true;
959 -}
960 -
961 -static size_t query_metric_first_working_tier(QUERY_METRIC *qm) {
962 - for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
963 -
964 - // find the db time-range for this tier for all metrics
965 - STORAGE_METRIC_HANDLE *smh = qm->tiers[tier].smh;
966 - time_t first_time_s = qm->tiers[tier].db_first_time_s;
967 - time_t last_time_s = qm->tiers[tier].db_last_time_s;
968 - time_t update_every_s = qm->tiers[tier].db_update_every_s;
969 -
970 - if(!smh || !first_time_s || !last_time_s || !update_every_s)
971 - continue;
972 -
973 - return tier;
974 - }
975 -
976 - return 0;
977 -}
978 -
979 -static long query_plan_points_coverage_weight(time_t db_first_time_s, time_t db_last_time_s, time_t db_update_every_s, time_t after_wanted, time_t before_wanted, size_t points_wanted, size_t tier __maybe_unused) {
980 - if(db_first_time_s == 0 ||
981 - db_last_time_s == 0 ||
982 - db_update_every_s == 0 ||
983 - db_first_time_s > before_wanted ||
984 - db_last_time_s < after_wanted)
985 - return -LONG_MAX;
986 -
987 - long long common_first_t = MAX(db_first_time_s, after_wanted);
988 - long long common_last_t = MIN(db_last_time_s, before_wanted);
989 -
990 - long long time_coverage = (common_last_t - common_first_t) * 1000000LL / (before_wanted - after_wanted);
991 - long long points_wanted_in_coverage = (long long)points_wanted * time_coverage / 1000000LL;
992 -
993 - long long points_available = (common_last_t - common_first_t) / db_update_every_s;
994 - long long points_delta = (long)(points_available - points_wanted_in_coverage);
995 - long long points_coverage = (points_delta < 0) ? (long)(points_available * time_coverage / points_wanted_in_coverage) : time_coverage;
996 -
997 - // a way to benefit higher tiers
998 - // points_coverage += (long)tier * 10000;
999 -
1000 - if(points_available <= 0)
1001 - return -LONG_MAX;
1002 -
1003 - return (long)(points_coverage + (25000LL * tier)); // 2.5% benefit for each higher tier
1004 -}
1005 -
1006 -static size_t query_metric_best_tier_for_timeframe(QUERY_METRIC *qm, time_t after_wanted, time_t before_wanted, size_t points_wanted) {
1007 - if(unlikely(nd_profile.storage_tiers < 2))
1008 - return 0;
1009 -
1010 - if(unlikely(after_wanted == before_wanted || points_wanted <= 0))
1011 - return query_metric_first_working_tier(qm);
1012 -
1013 - if(points_wanted < QUERY_PLAN_MIN_POINTS)
1014 - // when selecting tiers, aim for a resolution of at least QUERY_PLAN_MIN_POINTS points
1015 - points_wanted = (before_wanted - after_wanted) > QUERY_PLAN_MIN_POINTS ? QUERY_PLAN_MIN_POINTS : before_wanted - after_wanted;
1016 -
1017 - time_t min_first_time_s = 0;
1018 - time_t max_last_time_s = 0;
1019 -
1020 - for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
1021 - time_t first_time_s = qm->tiers[tier].db_first_time_s;
1022 - time_t last_time_s = qm->tiers[tier].db_last_time_s;
1023 -
1024 - if(!min_first_time_s || (first_time_s && first_time_s < min_first_time_s))
1025 - min_first_time_s = first_time_s;
1026 -
1027 - if(!max_last_time_s || (last_time_s && last_time_s > max_last_time_s))
1028 - max_last_time_s = last_time_s;
1029 - }
1030 -
1031 - for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
1032 -
1033 - // find the db time-range for this tier for all metrics
1034 - STORAGE_METRIC_HANDLE *smh = qm->tiers[tier].smh;
1035 - time_t first_time_s = qm->tiers[tier].db_first_time_s;
1036 - time_t last_time_s = qm->tiers[tier].db_last_time_s;
1037 - time_t update_every_s = qm->tiers[tier].db_update_every_s;
1038 -
1039 - if( !smh ||
1040 - !first_time_s ||
1041 - !last_time_s ||
1042 - !update_every_s ||
1043 - first_time_s > before_wanted ||
1044 - last_time_s < after_wanted
1045 - ) {
1046 - qm->tiers[tier].weight = -LONG_MAX;
1047 - continue;
1048 - }
1049 -
1050 - internal_fatal(first_time_s > before_wanted || last_time_s < after_wanted, "QUERY: invalid db durations");
1051 -
1052 - qm->tiers[tier].weight = query_plan_points_coverage_weight(
1053 - min_first_time_s, max_last_time_s, update_every_s,
1054 - after_wanted, before_wanted, points_wanted, tier);
1055 - }
1056 -
1057 - size_t best_tier = 0;
1058 - for(size_t tier = 1; tier < nd_profile.storage_tiers; tier++) {
1059 - if(qm->tiers[tier].weight >= qm->tiers[best_tier].weight)
1060 - best_tier = tier;
1061 - }
1062 -
1063 - return best_tier;
1064 -}
1065 -
1066 -static size_t rrddim_find_best_tier_for_timeframe(QUERY_TARGET *qt, time_t after_wanted, time_t before_wanted, size_t points_wanted) {
1067 - if(unlikely(nd_profile.storage_tiers < 2))
1068 - return 0;
1069 -
1070 - if(unlikely(after_wanted == before_wanted || points_wanted <= 0)) {
1071 - internal_error(true, "QUERY: '%s' has invalid params to tier calculation", qt->id);
1072 - return 0;
1073 - }
1074 -
1075 - long weight[nd_profile.storage_tiers];
1076 -
1077 - for(size_t tier = 0; tier < nd_profile.storage_tiers; tier++) {
1078 -
1079 - time_t common_first_time_s = 0;
1080 - time_t common_last_time_s = 0;
1081 - time_t common_update_every_s = 0;
1082 -
1083 - // find the db time-range for this tier for all metrics
1084 - for(size_t i = 0, used = qt->query.used; i < used ; i++) {
1085 - QUERY_METRIC *qm = query_metric(qt, i);
1086 -
1087 - time_t first_time_s = qm->tiers[tier].db_first_time_s;
1088 - time_t last_time_s = qm->tiers[tier].db_last_time_s;
1089 - time_t update_every_s = qm->tiers[tier].db_update_every_s;
1090 -
1091 - if(!first_time_s || !last_time_s || !update_every_s)
1092 - continue;
1093 -
1094 - if(!common_first_time_s)
1095 - common_first_time_s = first_time_s;
1096 - else
1097 - common_first_time_s = MIN(first_time_s, common_first_time_s);
1098 -
1099 - if(!common_last_time_s)
1100 - common_last_time_s = last_time_s;
1101 - else
1102 - common_last_time_s = MAX(last_time_s, common_last_time_s);
1103 -
1104 - if(!common_update_every_s)
1105 - common_update_every_s = update_every_s;
1106 - else
1107 - common_update_every_s = MIN(update_every_s, common_update_every_s);
1108 - }
1109 -
1110 - weight[tier] = query_plan_points_coverage_weight(common_first_time_s, common_last_time_s, common_update_every_s, after_wanted, before_wanted, points_wanted, tier);
1111 - }
1112 -
1113 - size_t best_tier = 0;
1114 - for(size_t tier = 1; tier < nd_profile.storage_tiers; tier++) {
1115 - if(weight[tier] >= weight[best_tier])
1116 - best_tier = tier;
1117 - }
1118 -
1119 - if(weight[best_tier] == -LONG_MAX)
1120 - best_tier = 0;
1121 -
1122 - return best_tier;
1123 -}
1124 -
1125 -static time_t rrdset_find_natural_update_every_for_timeframe(QUERY_TARGET *qt, time_t after_wanted, time_t before_wanted, size_t points_wanted, RRDR_OPTIONS options, size_t tier) {
1126 - size_t best_tier;
1127 - if((options & RRDR_OPTION_SELECTED_TIER) && tier < nd_profile.storage_tiers)
1128 - best_tier = tier;
1129 - else
1130 - best_tier = rrddim_find_best_tier_for_timeframe(qt, after_wanted, before_wanted, points_wanted);
1131 -
1132 - // find the db minimum update every for this tier for all metrics
1133 - time_t common_update_every_s = nd_profile.update_every;
1134 - for(size_t i = 0, used = qt->query.used; i < used ; i++) {
1135 - QUERY_METRIC *qm = query_metric(qt, i);
1136 -
1137 - time_t update_every_s = qm->tiers[best_tier].db_update_every_s;
1138 -
1139 - if(!i)
1140 - common_update_every_s = update_every_s;
1141 - else
1142 - common_update_every_s = MIN(update_every_s, common_update_every_s);
1143 - }
1144 -
1145 - return common_update_every_s;
1146 -}
1147 -
1148 -// ----------------------------------------------------------------------------
1149 -// query ops
1150 -
1151 -typedef struct query_point {
1152 - STORAGE_POINT sp;
1153 - NETDATA_DOUBLE value;
1154 - bool added;
1155 -#ifdef NETDATA_INTERNAL_CHECKS
1156 - size_t id;
1157 -#endif
1158 -} QUERY_POINT;
1159 -
1160 -QUERY_POINT QUERY_POINT_EMPTY = {
1161 - .sp = STORAGE_POINT_UNSET,
1162 - .value = NAN,
1163 - .added = false,
1164 -#ifdef NETDATA_INTERNAL_CHECKS
1165 - .id = 0,
1166 -#endif
1167 -};
1168 -
1169 -#ifdef NETDATA_INTERNAL_CHECKS
1170 -#define query_point_set_id(point, point_id) (point).id = point_id
1171 -#else
1172 -#define query_point_set_id(point, point_id) debug_dummy()
1173 -#endif
1174 -
1175 -typedef struct query_engine_ops {
1176 - // configuration
1177 - RRDR *r;
1178 - QUERY_METRIC *qm;
1179 - time_t view_update_every;
1180 - time_t query_granularity;
1181 - TIER_QUERY_FETCH tier_query_fetch;
1182 -
1183 - // query planer
1184 - size_t current_plan;
1185 - time_t current_plan_expire_time;
1186 - time_t plan_expanded_after;
1187 - time_t plan_expanded_before;
1188 -
1189 - // storage queries
1190 - size_t tier;
1191 - struct query_metric_tier *tier_ptr;
1192 - struct storage_engine_query_handle *seqh;
1193 -
1194 - // aggregating points over time
1195 - size_t group_points_non_zero;
1196 - size_t group_points_added;
1197 - STORAGE_POINT group_point; // aggregates min, max, sum, count, anomaly count for each group point
1198 - STORAGE_POINT query_point; // aggregates min, max, sum, count, anomaly count across the whole query
1199 - RRDR_VALUE_FLAGS group_value_flags;
1200 -
1201 - // statistics
1202 - size_t db_total_points_read;
1203 - size_t db_points_read_per_tier[RRD_STORAGE_TIERS];
1204 -
1205 - struct {
1206 - time_t expanded_after;
1207 - time_t expanded_before;
1208 - struct storage_engine_query_handle handle;
1209 - bool initialized;
1210 - bool finalized;
1211 - } plans[QUERY_PLANS_MAX];
1212 -
1213 - struct query_engine_ops *next;
1214 -} QUERY_ENGINE_OPS;
1215 -
1216 -
1217 -// ----------------------------------------------------------------------------
1218 -// query planer
1219 -
1220 -#define query_plan_should_switch_plan(ops, now) ((now) >= (ops)->current_plan_expire_time)
1221 -
1222 -static size_t query_planer_expand_duration_in_points(time_t this_update_every, time_t next_update_every) {
1223 -
1224 - time_t delta = this_update_every - next_update_every;
1225 - if(delta < 0) delta = -delta;
1226 -
1227 - size_t points;
1228 - if(delta < this_update_every * POINTS_TO_EXPAND_QUERY)
1229 - points = POINTS_TO_EXPAND_QUERY;
1230 - else
1231 - points = (delta + this_update_every - 1) / this_update_every;
1232 -
1233 - return points;
1234 -}
1235 -
1236 -static void query_planer_initialize_plans(QUERY_ENGINE_OPS *ops) {
1237 - QUERY_METRIC *qm = ops->qm;
1238 -
1239 - for(size_t p = 0; p < qm->plan.used ; p++) {
1240 - size_t tier = qm->plan.array[p].tier;
1241 - time_t update_every = qm->tiers[tier].db_update_every_s;
1242 -
1243 - size_t points_to_add_to_after;
1244 - if(p > 0) {
1245 - // there is another plan before to this
1246 -
1247 - size_t tier0 = qm->plan.array[p - 1].tier;
1248 - time_t update_every0 = qm->tiers[tier0].db_update_every_s;
1249 -
1250 - points_to_add_to_after = query_planer_expand_duration_in_points(update_every, update_every0);
1251 - }
1252 - else
1253 - points_to_add_to_after = (tier == 0) ? 0 : POINTS_TO_EXPAND_QUERY;
1254 -
1255 - size_t points_to_add_to_before;
1256 - if(p + 1 < qm->plan.used) {
1257 - // there is another plan after to this
1258 -
1259 - size_t tier1 = qm->plan.array[p+1].tier;
1260 - time_t update_every1 = qm->tiers[tier1].db_update_every_s;
1261 -
1262 - points_to_add_to_before = query_planer_expand_duration_in_points(update_every, update_every1);
1263 - }
1264 - else
1265 - points_to_add_to_before = POINTS_TO_EXPAND_QUERY;
1266 -
1267 - time_t after = qm->plan.array[p].after - (time_t)(update_every * points_to_add_to_after);
1268 - time_t before = qm->plan.array[p].before + (time_t)(update_every * points_to_add_to_before);
1269 -
1270 - ops->plans[p].expanded_after = after;
1271 - ops->plans[p].expanded_before = before;
1272 -
1273 - ops->r->internal.qt->db.tiers[tier].queries++;
1274 -
1275 - struct query_metric_tier *tier_ptr = &qm->tiers[tier];
1276 - STORAGE_ENGINE *eng = query_metric_storage_engine(ops->r->internal.qt, qm, tier);
1277 - storage_engine_query_init(eng->seb, tier_ptr->smh, &ops->plans[p].handle,
1278 - after, before, ops->r->internal.qt->request.priority);
1279 -
1280 - ops->plans[p].initialized = true;
1281 - ops->plans[p].finalized = false;
1282 - }
1283 -}
1284 -
1285 -static void query_planer_finalize_plan(QUERY_ENGINE_OPS *ops, size_t plan_id) {
1286 - // QUERY_METRIC *qm = ops->qm;
1287 -
1288 - if(ops->plans[plan_id].initialized && !ops->plans[plan_id].finalized) {
1289 - storage_engine_query_finalize(&ops->plans[plan_id].handle);
1290 - ops->plans[plan_id].initialized = false;
1291 - ops->plans[plan_id].finalized = true;
1292 - }
1293 -}
1294 -
1295 -static void query_planer_finalize_remaining_plans(QUERY_ENGINE_OPS *ops) {
1296 - QUERY_METRIC *qm = ops->qm;
1297 -
1298 - for(size_t p = 0; p < qm->plan.used ; p++)
1299 - query_planer_finalize_plan(ops, p);
1300 -}
1301 -
1302 -static void query_planer_activate_plan(QUERY_ENGINE_OPS *ops, size_t plan_id, time_t overwrite_after __maybe_unused) {
1303 - QUERY_METRIC *qm = ops->qm;
1304 -
1305 - internal_fatal(plan_id >= qm->plan.used, "QUERY: invalid plan_id given");
1306 - internal_fatal(!ops->plans[plan_id].initialized, "QUERY: plan has not been initialized");
1307 - internal_fatal(ops->plans[plan_id].finalized, "QUERY: plan has been finalized");
1308 -
1309 - internal_fatal(qm->plan.array[plan_id].after > qm->plan.array[plan_id].before, "QUERY: flipped after/before");
1310 -
1311 - ops->tier = qm->plan.array[plan_id].tier;
1312 - ops->tier_ptr = &qm->tiers[ops->tier];
1313 - ops->seqh = &ops->plans[plan_id].handle;
1314 - ops->current_plan = plan_id;
1315 -
1316 - if(plan_id + 1 < qm->plan.used && qm->plan.array[plan_id + 1].after < qm->plan.array[plan_id].before)
1317 - ops->current_plan_expire_time = qm->plan.array[plan_id + 1].after;
1318 - else
1319 - ops->current_plan_expire_time = qm->plan.array[plan_id].before;
1320 -
1321 - ops->plan_expanded_after = ops->plans[plan_id].expanded_after;
1322 - ops->plan_expanded_before = ops->plans[plan_id].expanded_before;
1323 -}
1324 -
1325 -static bool query_planer_next_plan(QUERY_ENGINE_OPS *ops, time_t now, time_t last_point_end_time) {
1326 - QUERY_METRIC *qm = ops->qm;
1327 -
1328 - size_t old_plan = ops->current_plan;
1329 -
1330 - time_t next_plan_before_time;
1331 - do {
1332 - ops->current_plan++;
1333 -
1334 - if (ops->current_plan >= qm->plan.used) {
1335 - ops->current_plan = old_plan;
1336 - ops->current_plan_expire_time = ops->r->internal.qt->window.before;
1337 - // let the query run with current plan
1338 - // we will not switch it
1339 - return false;
1340 - }
1341 -
1342 - next_plan_before_time = qm->plan.array[ops->current_plan].before;
1343 - } while(now >= next_plan_before_time || last_point_end_time >= next_plan_before_time);
1344 -
1345 - if(!query_metric_is_valid_tier(qm, qm->plan.array[ops->current_plan].tier)) {
1346 - ops->current_plan = old_plan;
1347 - ops->current_plan_expire_time = ops->r->internal.qt->window.before;
1348 - return false;
1349 - }
1350 -
1351 - query_planer_finalize_plan(ops, old_plan);
1352 - query_planer_activate_plan(ops, ops->current_plan, MIN(now, last_point_end_time));
1353 - return true;
1354 -}
1355 -
1356 -static int compare_query_plan_entries_on_start_time(const void *a, const void *b) {
1357 - QUERY_PLAN_ENTRY *p1 = (QUERY_PLAN_ENTRY *)a;
1358 - QUERY_PLAN_ENTRY *p2 = (QUERY_PLAN_ENTRY *)b;
1359 - return (p1->after < p2->after)?-1:1;
1360 -}
1361 -
1362 -static bool query_plan(QUERY_ENGINE_OPS *ops, time_t after_wanted, time_t before_wanted, size_t points_wanted) {
1363 - QUERY_METRIC *qm = ops->qm;
1364 -
1365 - // put our selected tier as the first plan
1366 - size_t selected_tier;
1367 - bool switch_tiers = true;
1368 -
1369 - if((ops->r->internal.qt->window.options & RRDR_OPTION_SELECTED_TIER)
1370 - && ops->r->internal.qt->window.tier < nd_profile.storage_tiers && query_metric_is_valid_tier(qm, ops->r->internal.qt->window.tier)) {
1371 - selected_tier = ops->r->internal.qt->window.tier;
1372 - switch_tiers = false;
1373 - }
1374 - else {
1375 - selected_tier = query_metric_best_tier_for_timeframe(qm, after_wanted, before_wanted, points_wanted);
1376 -
1377 - if(!query_metric_is_valid_tier(qm, selected_tier))
1378 - return false;
1379 - }
1380 -
1381 - if(qm->tiers[selected_tier].db_first_time_s > before_wanted ||
1382 - qm->tiers[selected_tier].db_last_time_s < after_wanted) {
1383 - // we don't have any data to satisfy this query
1384 - return false;
1385 - }
1386 -
1387 - qm->plan.used = 1;
1388 - qm->plan.array[0].tier = selected_tier;
1389 - qm->plan.array[0].after = (qm->tiers[selected_tier].db_first_time_s < after_wanted) ? after_wanted : qm->tiers[selected_tier].db_first_time_s;
1390 - qm->plan.array[0].before = (qm->tiers[selected_tier].db_last_time_s > before_wanted) ? before_wanted : qm->tiers[selected_tier].db_last_time_s;
1391 -
1392 - if(switch_tiers) {
1393 - // the selected tier
1394 - time_t selected_tier_first_time_s = qm->plan.array[0].after;
1395 - time_t selected_tier_last_time_s = qm->plan.array[0].before;
1396 -
1397 - // check if our selected tier can start the query
1398 - if (selected_tier_first_time_s > after_wanted) {
1399 - // we need some help from other tiers
1400 - for (size_t tr = (int)selected_tier + 1; tr < nd_profile.storage_tiers && qm->plan.used < QUERY_PLANS_MAX ; tr++) {
1401 - if(!query_metric_is_valid_tier(qm, tr))
1402 - continue;
1403 -
1404 - // find the first time of this tier
1405 - time_t tier_first_time_s = qm->tiers[tr].db_first_time_s;
1406 - time_t tier_last_time_s = qm->tiers[tr].db_last_time_s;
1407 -
1408 - // can it help?
1409 - if (tier_first_time_s < selected_tier_first_time_s && tier_first_time_s <= before_wanted && tier_last_time_s >= after_wanted) {
1410 - // it can help us add detail at the beginning of the query
1411 - QUERY_PLAN_ENTRY t = {
1412 - .tier = tr,
1413 - .after = (tier_first_time_s < after_wanted) ? after_wanted : tier_first_time_s,
1414 - .before = selected_tier_first_time_s,
1415 - };
1416 - ops->plans[qm->plan.used].initialized = false;
1417 - ops->plans[qm->plan.used].finalized = false;
1418 - qm->plan.array[qm->plan.used++] = t;
1419 -
1420 - internal_fatal(!t.after || !t.before, "QUERY: invalid plan selected");
1421 -
1422 - // prepare for the tier
1423 - selected_tier_first_time_s = t.after;
1424 -
1425 - if (t.after <= after_wanted)
1426 - break;
1427 - }
1428 - }
1429 - }
1430 -
1431 - // check if our selected tier can finish the query
1432 - if (selected_tier_last_time_s < before_wanted) {
1433 - // we need some help from other tiers
1434 - for (int tr = (int)selected_tier - 1; tr >= 0 && qm->plan.used < QUERY_PLANS_MAX ; tr--) {
1435 - if(!query_metric_is_valid_tier(qm, tr))
1436 - continue;
1437 -
1438 - // find the last time of this tier
1439 - time_t tier_first_time_s = qm->tiers[tr].db_first_time_s;
1440 - time_t tier_last_time_s = qm->tiers[tr].db_last_time_s;
1441 -
1442 - //buffer_sprintf(wb, ": EVAL BEFORE tier %d, %ld", tier, last_time_s);
1443 -
1444 - // can it help?
1445 - if (tier_last_time_s > selected_tier_last_time_s && tier_first_time_s <= before_wanted && tier_last_time_s >= after_wanted) {
1446 - // it can help us add detail at the end of the query
1447 - QUERY_PLAN_ENTRY t = {
1448 - .tier = tr,
1449 - .after = selected_tier_last_time_s,
1450 - .before = (tier_last_time_s > before_wanted) ? before_wanted : tier_last_time_s,
1451 - };
1452 - ops->plans[qm->plan.used].initialized = false;
1453 - ops->plans[qm->plan.used].finalized = false;
1454 - qm->plan.array[qm->plan.used++] = t;
1455 -
1456 - // prepare for the tier
1457 - selected_tier_last_time_s = t.before;
1458 -
1459 - internal_fatal(!t.after || !t.before, "QUERY: invalid plan selected");
1460 -
1461 - if (t.before >= before_wanted)
1462 - break;
1463 - }
1464 - }
1465 - }
1466 - }
1467 -
1468 - // sort the query plan
1469 - if(qm->plan.used > 1)
1470 - qsort(&qm->plan.array, qm->plan.used, sizeof(QUERY_PLAN_ENTRY), compare_query_plan_entries_on_start_time);
1471 -
1472 - if(!query_metric_is_valid_tier(qm, qm->plan.array[0].tier))
1473 - return false;
1474 -
1475 -#ifdef NETDATA_INTERNAL_CHECKS
1476 - for(size_t p = 0; p < qm->plan.used ;p++) {
1477 - internal_fatal(qm->plan.array[p].after > qm->plan.array[p].before, "QUERY: flipped after/before");
1478 - internal_fatal(qm->plan.array[p].after < after_wanted, "QUERY: too small plan first time");
1479 - internal_fatal(qm->plan.array[p].before > before_wanted, "QUERY: too big plan last time");
1480 - }
1481 -#endif
1482 -
1483 - query_planer_initialize_plans(ops);
1484 - query_planer_activate_plan(ops, 0, 0);
1485 -
1486 - return true;
1487 -}
1488 -
1489 -
1490 -// ----------------------------------------------------------------------------
1491 -// dimension level query engine
1492 -
1493 -#define query_interpolate_point(this_point, last_point, now) do { \
1494 - if(likely( \
1495 - /* the point to interpolate is more than 1s wide */ \
1496 - (this_point).sp.end_time_s - (this_point).sp.start_time_s > 1 \
1497 - \
1498 - /* the two points are exactly next to each other */ \
1499 - && (last_point).sp.end_time_s == (this_point).sp.start_time_s \
1500 - \
1501 - /* both points are valid numbers */ \
1502 - && netdata_double_isnumber((this_point).value) \
1503 - && netdata_double_isnumber((last_point).value) \
1504 - \
1505 - )) { \
1506 - (this_point).value = (last_point).value + ((this_point).value - (last_point).value) * (1.0 - (NETDATA_DOUBLE)((this_point).sp.end_time_s - (now)) / (NETDATA_DOUBLE)((this_point).sp.end_time_s - (this_point).sp.start_time_s)); \
1507 - (this_point).sp.end_time_s = now; \
1508 - } \
1509 -} while(0)
1510 -
1511 -#define query_add_point_to_group(r, point, ops, add_flush) do { \
1512 - if(likely(netdata_double_isnumber((point).value))) { \
1513 - if(likely(fpclassify((point).value) != FP_ZERO)) \
1514 - (ops)->group_points_non_zero++; \
1515 - \
1516 - if(unlikely((point).sp.flags & SN_FLAG_RESET)) \
1517 - (ops)->group_value_flags |= RRDR_VALUE_RESET; \
1518 - \
1519 - time_grouping_add(r, (point).value, add_flush); \
1520 - \
1521 - storage_point_merge_to((ops)->group_point, (point).sp); \
1522 - if(!(point).added) \
1523 - storage_point_merge_to((ops)->query_point, (point).sp); \
1524 - } \
1525 - \
1526 - (ops)->group_points_added++; \
1527 -} while(0)
1528 -
1529 -static __thread QUERY_ENGINE_OPS *released_ops = NULL;
1530 -
1531 -static void rrd2rrdr_query_ops_freeall(RRDR *r __maybe_unused) {
1532 - while(released_ops) {
1533 - QUERY_ENGINE_OPS *ops = released_ops;
1534 - released_ops = ops->next;
1535 -
1536 - onewayalloc_freez(r->internal.owa, ops);
1537 - }
1538 -}
1539 -
1540 -static void rrd2rrdr_query_ops_release(QUERY_ENGINE_OPS *ops) {
1541 - if(!ops) return;
1542 -
1543 - ops->next = released_ops;
1544 - released_ops = ops;
1545 -}
1546 -
1547 -static QUERY_ENGINE_OPS *rrd2rrdr_query_ops_get(RRDR *r) {
1548 - QUERY_ENGINE_OPS *ops;
1549 - if(released_ops) {
1550 - ops = released_ops;
1551 - released_ops = ops->next;
1552 - }
1553 - else {
1554 - ops = onewayalloc_mallocz(r->internal.owa, sizeof(QUERY_ENGINE_OPS));
1555 - }
1556 -
1557 - memset(ops, 0, sizeof(*ops));
1558 - return ops;
1559 -}
1560 -
1561 -static QUERY_ENGINE_OPS *rrd2rrdr_query_ops_prep(RRDR *r, size_t query_metric_id) {
1562 - QUERY_TARGET *qt = r->internal.qt;
1563 -
1564 - QUERY_ENGINE_OPS *ops = rrd2rrdr_query_ops_get(r);
1565 - *ops = (QUERY_ENGINE_OPS) {
1566 - .r = r,
1567 - .qm = query_metric(qt, query_metric_id),
1568 - .tier_query_fetch = r->time_grouping.tier_query_fetch,
1569 - .view_update_every = r->view.update_every,
1570 - .query_granularity = (time_t)(r->view.update_every / r->view.group),
1571 - .group_value_flags = RRDR_VALUE_NOTHING,
1572 - };
1573 -
1574 - if(!query_plan(ops, qt->window.after, qt->window.before, qt->window.points)) {
1575 - rrd2rrdr_query_ops_release(ops);
1576 - return NULL;
1577 - }
1578 -
1579 - return ops;
1580 -}
1581 -
1582 -NOT_INLINE_HOT static void rrd2rrdr_query_execute(RRDR *r, size_t dim_id_in_rrdr, QUERY_ENGINE_OPS *ops) {
1583 - QUERY_TARGET *qt = r->internal.qt;
1584 - QUERY_METRIC *qm = ops->qm;
1585 -
1586 - const RRDR_TIME_GROUPING add_flush = r->time_grouping.add_flush;
1587 -
1588 - ops->group_point = STORAGE_POINT_UNSET;
1589 - ops->query_point = STORAGE_POINT_UNSET;
1590 -
1591 - RRDR_OPTIONS options = qt->window.options;
1592 - size_t points_wanted = qt->window.points;
1593 - time_t after_wanted = qt->window.after;
1594 - time_t before_wanted = qt->window.before; (void)before_wanted;
1595 -
1596 -// bool debug_this = false;
1597 -// if(strcmp("user", string2str(rd->id)) == 0 && strcmp("system.cpu", string2str(rd->rrdset->id)) == 0)
1598 -// debug_this = true;
1599 -
1600 - size_t points_added = 0;
1601 -
1602 - long rrdr_line = -1;
1603 - bool use_anomaly_bit_as_value = (r->internal.qt->window.options & RRDR_OPTION_ANOMALY_BIT) ? true : false;
1604 -
1605 - NETDATA_DOUBLE min = r->view.min, max = r->view.max;
1606 -
1607 - QUERY_POINT last2_point = QUERY_POINT_EMPTY;
1608 - QUERY_POINT last1_point = QUERY_POINT_EMPTY;
1609 - QUERY_POINT new_point = QUERY_POINT_EMPTY;
1610 -
1611 - // ONE POINT READ-AHEAD
1612 - // when we switch plans, we read-ahead a point from the next plan
1613 - // to join them smoothly at the exact time the next plan begins
1614 - STORAGE_POINT next1_point = STORAGE_POINT_UNSET;
1615 -
1616 - time_t now_start_time = after_wanted - ops->query_granularity;
1617 - time_t now_end_time = after_wanted + ops->view_update_every - ops->query_granularity;
1618 -
1619 - size_t db_points_read_since_plan_switch = 0; (void)db_points_read_since_plan_switch;
1620 - size_t query_is_finished_counter = 0;
1621 -
1622 - // The main loop, based on the query granularity we need
1623 - for( ; points_added < points_wanted && query_is_finished_counter <= 10 ;
1624 - now_start_time = now_end_time, now_end_time += ops->view_update_every) {
1625 -
1626 - if(unlikely(query_plan_should_switch_plan(ops, now_end_time))) {
1627 - query_planer_next_plan(ops, now_end_time, new_point.sp.end_time_s);
1628 - db_points_read_since_plan_switch = 0;
1629 - }
1630 -
1631 - // read all the points of the db, prior to the time we need (now_end_time)
1632 -
1633 - size_t count_same_end_time = 0;
1634 - while(count_same_end_time < 100) {
1635 - if(likely(count_same_end_time == 0)) {
1636 - last2_point = last1_point;
1637 - last1_point = new_point;
1638 - }
1639 -
1640 - if(unlikely(storage_engine_query_is_finished(ops->seqh))) {
1641 - query_is_finished_counter++;
1642 -
1643 - if(count_same_end_time != 0) {
1644 - last2_point = last1_point;
1645 - last1_point = new_point;
1646 - }
1647 - new_point = QUERY_POINT_EMPTY;
1648 - new_point.sp.start_time_s = last1_point.sp.end_time_s;
1649 - new_point.sp.end_time_s = now_end_time;
1650 -//
1651 -// if(debug_this) netdata_log_info("QUERY: is finished() returned true");
1652 -//
1653 - break;
1654 - }
1655 - else
1656 - query_is_finished_counter = 0;
1657 -
1658 - // fetch the new point
1659 - {
1660 - STORAGE_POINT sp;
1661 - if(likely(storage_point_is_unset(next1_point))) {
1662 - db_points_read_since_plan_switch++;
1663 - sp = storage_engine_query_next_metric(ops->seqh);
1664 - ops->db_points_read_per_tier[ops->tier]++;
1665 - ops->db_total_points_read++;
1666 -
1667 - if(unlikely(options & RRDR_OPTION_ABSOLUTE))
1668 - storage_point_make_positive(sp);
1669 - }
1670 - else {
1671 - // ONE POINT READ-AHEAD
1672 - sp = next1_point;
1673 - storage_point_unset(next1_point);
1674 - db_points_read_since_plan_switch = 1;
1675 - }
1676 -
1677 - // ONE POINT READ-AHEAD
1678 - if(unlikely(query_plan_should_switch_plan(ops, sp.end_time_s) &&
1679 - query_planer_next_plan(ops, now_end_time, new_point.sp.end_time_s))) {
1680 -
1681 - // The end time of the current point, crosses our plans (tiers)
1682 - // so, we switched plan (tier)
1683 - //
1684 - // There are 2 cases now:
1685 - //
1686 - // A. the entire point of the previous plan is to the future of point from the next plan
1687 - // B. part of the point of the previous plan overlaps with the point from the next plan
1688 -
1689 - STORAGE_POINT sp2 = storage_engine_query_next_metric(ops->seqh);
1690 - ops->db_points_read_per_tier[ops->tier]++;
1691 - ops->db_total_points_read++;
1692 -
1693 - if(unlikely(options & RRDR_OPTION_ABSOLUTE))
1694 - storage_point_make_positive(sp);
1695 -
1696 - if(sp.start_time_s > sp2.start_time_s)
1697 - // the point from the previous plan is useless
1698 - sp = sp2;
1699 - else
1700 - // let the query run from the previous plan
1701 - // but setting this will also cut off the interpolation
1702 - // of the point from the previous plan
1703 - next1_point = sp2;
1704 - }
1705 -
1706 - new_point.sp = sp;
1707 - new_point.added = false;
1708 - query_point_set_id(new_point, ops->db_total_points_read);
1709 -
1710 -// if(debug_this)
1711 -// netdata_log_info("QUERY: got point %zu, from time %ld to %ld // now from %ld to %ld // query from %ld to %ld",
1712 -// new_point.id, new_point.start_time, new_point.end_time, now_start_time, now_end_time, after_wanted, before_wanted);
1713 -//
1714 - // get the right value from the point we got
1715 - if(likely(!storage_point_is_unset(sp) && !storage_point_is_gap(sp))) {
1716 -
1717 - if(unlikely(use_anomaly_bit_as_value))
1718 - new_point.value = storage_point_anomaly_rate(new_point.sp);
1719 -
1720 - else {
1721 - switch (ops->tier_query_fetch) {
1722 - default:
1723 - case TIER_QUERY_FETCH_AVERAGE:
1724 - new_point.value = sp.sum / (NETDATA_DOUBLE)sp.count;
1725 - break;
1726 -
1727 - case TIER_QUERY_FETCH_MIN:
1728 - new_point.value = sp.min;
1729 - break;
1730 -
1731 - case TIER_QUERY_FETCH_MAX:
1732 - new_point.value = sp.max;
1733 - break;
1734 -
1735 - case TIER_QUERY_FETCH_SUM:
1736 - new_point.value = sp.sum;
1737 - break;
1738 - }
1739 - }
1740 - }
1741 - else
1742 - new_point.value = NAN;
1743 - }
1744 -
1745 - // check if the db is giving us zero duration points
1746 - if(unlikely(db_points_read_since_plan_switch > 1 &&
1747 - new_point.sp.start_time_s == new_point.sp.end_time_s)) {
1748 -
1749 - internal_error(true, "QUERY: '%s', dimension '%s' next_metric() returned "
1750 - "point %zu from %ld to %ld, that are both equal",
1751 - qt->id, query_metric_id(qt, qm),
1752 - new_point.id, new_point.sp.start_time_s, new_point.sp.end_time_s);
1753 -
1754 - new_point.sp.start_time_s = new_point.sp.end_time_s - ops->tier_ptr->db_update_every_s;
1755 - }
1756 -
1757 - // check if the db is advancing the query
1758 - if(unlikely(db_points_read_since_plan_switch > 1 &&
1759 - new_point.sp.end_time_s <= last1_point.sp.end_time_s)) {
1760 -
1761 - internal_error(true,
1762 - "QUERY: '%s', dimension '%s' next_metric() returned "
1763 - "point %zu from %ld to %ld, before the "
1764 - "last point %zu from %ld to %ld, "
1765 - "now is %ld to %ld",
1766 - qt->id, query_metric_id(qt, qm),
1767 - new_point.id, new_point.sp.start_time_s, new_point.sp.end_time_s,
1768 - last1_point.id, last1_point.sp.start_time_s, last1_point.sp.end_time_s,
1769 - now_start_time, now_end_time);
1770 -
1771 - count_same_end_time++;
1772 - continue;
1773 - }
1774 - count_same_end_time = 0;
1775 -
1776 - // decide how to use this point
1777 - if(likely(new_point.sp.end_time_s < now_end_time)) { // likely to favor tier0
1778 - // this db point ends before our now_end_time
1779 -
1780 - if(likely(new_point.sp.end_time_s >= now_start_time)) { // likely to favor tier0
1781 - // this db point ends after our now_start time
1782 -
1783 - query_add_point_to_group(r, new_point, ops, add_flush);
1784 - new_point.added = true;
1785 - }
1786 - else {
1787 - // we don't need this db point
1788 - // it is totally outside our current time-frame
1789 -
1790 - // this is desirable for the first point of the query
1791 - // because it allows us to interpolate the next point
1792 - // at exactly the time we will want
1793 -
1794 - // we only log if this is not point 1
1795 - internal_error(new_point.sp.end_time_s < ops->plan_expanded_after &&
1796 - db_points_read_since_plan_switch > 1,
1797 - "QUERY: '%s', dimension '%s' next_metric() "
1798 - "returned point %zu from %ld time %ld, "
1799 - "which is entirely before our current timeframe %ld to %ld "
1800 - "(and before the entire query, after %ld, before %ld)",
1801 - qt->id, query_metric_id(qt, qm),
1802 - new_point.id, new_point.sp.start_time_s, new_point.sp.end_time_s,
1803 - now_start_time, now_end_time,
1804 - ops->plan_expanded_after, ops->plan_expanded_before);
1805 - }
1806 -
1807 - }
1808 - else {
1809 - // the point ends in the future
1810 - // so, we will interpolate it below, at the inner loop
1811 - break;
1812 - }
1813 - }
1814 -
1815 - if(unlikely(count_same_end_time)) {
1816 - internal_error(true,
1817 - "QUERY: '%s', dimension '%s', the database does not advance the query,"
1818 - " it returned an end time less or equal to the end time of the last "
1819 - "point we got %ld, %zu times",
1820 - qt->id, query_metric_id(qt, qm),
1821 - last1_point.sp.end_time_s, count_same_end_time);
1822 -
1823 - if(unlikely(new_point.sp.end_time_s <= last1_point.sp.end_time_s))
1824 - new_point.sp.end_time_s = now_end_time;
1825 - }
1826 -
1827 - time_t stop_time = new_point.sp.end_time_s;
1828 - if(unlikely(!storage_point_is_unset(next1_point) && next1_point.start_time_s >= now_end_time)) {
1829 - // ONE POINT READ-AHEAD
1830 - // the point crosses the start time of the
1831 - // read ahead storage point we have read
1832 - stop_time = next1_point.start_time_s;
1833 - }
1834 -
1835 - // the inner loop
1836 - // we have 3 points in memory: last2, last1, new
1837 - // we select the one to use based on their timestamps
1838 -
1839 - internal_fatal(now_end_time > stop_time || points_added >= points_wanted,
1840 - "QUERY: first part of query provides invalid point to interpolate (now_end_time %ld, stop_time %ld",
1841 - now_end_time, stop_time);
1842 -
1843 - do {
1844 - // now_start_time is wrong in this loop
1845 - // but, we don't need it
1846 -
1847 - QUERY_POINT current_point;
1848 -
1849 - if(likely(now_end_time > new_point.sp.start_time_s)) {
1850 - // it is time for our NEW point to be used
1851 - current_point = new_point;
1852 - new_point.added = true; // first copy, then set it, so that new_point will not be added again
1853 - query_interpolate_point(current_point, last1_point, now_end_time);
1854 -
1855 -// internal_error(current_point.id > 0
1856 -// && last1_point.id == 0
1857 -// && current_point.end_time > after_wanted
1858 -// && current_point.end_time > now_end_time,
1859 -// "QUERY: '%s', dimension '%s', after %ld, before %ld, view update every %ld,"
1860 -// " query granularity %ld, interpolating point %zu (from %ld to %ld) at %ld,"
1861 -// " but we could really favor by having last_point1 in this query.",
1862 -// qt->id, string2str(qm->dimension.id),
1863 -// after_wanted, before_wanted,
1864 -// ops.view_update_every, ops.query_granularity,
1865 -// current_point.id, current_point.start_time, current_point.end_time,
1866 -// now_end_time);
1867 - }
1868 - else if(likely(now_end_time <= last1_point.sp.end_time_s)) {
1869 - // our LAST point is still valid
1870 - current_point = last1_point;
1871 - last1_point.added = true; // first copy, then set it, so that last1_point will not be added again
1872 - query_interpolate_point(current_point, last2_point, now_end_time);
1873 -
1874 -// internal_error(current_point.id > 0
1875 -// && last2_point.id == 0
1876 -// && current_point.end_time > after_wanted
1877 -// && current_point.end_time > now_end_time,
1878 -// "QUERY: '%s', dimension '%s', after %ld, before %ld, view update every %ld,"
1879 -// " query granularity %ld, interpolating point %zu (from %ld to %ld) at %ld,"
1880 -// " but we could really favor by having last_point2 in this query.",
1881 -// qt->id, string2str(qm->dimension.id),
1882 -// after_wanted, before_wanted, ops.view_update_every, ops.query_granularity,
1883 -// current_point.id, current_point.start_time, current_point.end_time,
1884 -// now_end_time);
1885 - }
1886 - else {
1887 - // a GAP, we don't have a value this time
1888 - current_point = QUERY_POINT_EMPTY;
1889 - }
1890 -
1891 - query_add_point_to_group(r, current_point, ops, add_flush);
1892 -
1893 - rrdr_line = rrdr_line_init(r, now_end_time, rrdr_line);
1894 - size_t rrdr_o_v_index = rrdr_line * r->d + dim_id_in_rrdr;
1895 -
1896 - // find the place to store our values
1897 - RRDR_VALUE_FLAGS *rrdr_value_options_ptr = &r->o[rrdr_o_v_index];
1898 -
1899 - // update the dimension options
1900 - if(likely(ops->group_points_non_zero))
1901 - r->od[dim_id_in_rrdr] |= RRDR_DIMENSION_NONZERO;
1902 -
1903 - // store the specific point options
1904 - *rrdr_value_options_ptr = ops->group_value_flags;
1905 -
1906 - // store the group value
1907 - NETDATA_DOUBLE group_value = time_grouping_flush(r, rrdr_value_options_ptr, add_flush);
1908 - r->v[rrdr_o_v_index] = group_value;
1909 -
1910 - r->ar[rrdr_o_v_index] = storage_point_anomaly_rate(ops->group_point);
1911 -
1912 - if(likely(points_added || r->internal.queries_count)) {
1913 - // find the min/max across all dimensions
1914 -
1915 - if(unlikely(group_value < min)) min = group_value;
1916 - if(unlikely(group_value > max)) max = group_value;
1917 -
1918 - }
1919 - else {
1920 - // runs only when r->internal.queries_count == 0 && points_added == 0
1921 - // so, on the first point added for the query.
1922 - min = max = group_value;
1923 - }
1924 -
1925 - points_added++;
1926 - ops->group_points_added = 0;
1927 - ops->group_value_flags = RRDR_VALUE_NOTHING;
1928 - ops->group_points_non_zero = 0;
1929 - ops->group_point = STORAGE_POINT_UNSET;
1930 -
1931 - now_end_time += ops->view_update_every;
1932 - } while(now_end_time <= stop_time && points_added < points_wanted);
1933 -
1934 - // the loop above increased "now" by ops->view_update_every,
1935 - // but the main loop will increase it too,
1936 - // so, let's undo the last iteration of this loop
1937 - now_end_time -= ops->view_update_every;
1938 - }
1939 - query_planer_finalize_remaining_plans(ops);
1940 -
1941 - qm->query_points = ops->query_point;
1942 -
1943 - // fill the rest of the points with empty values
1944 - while (points_added < points_wanted) {
1945 - rrdr_line++;
1946 - size_t rrdr_o_v_index = rrdr_line * r->d + dim_id_in_rrdr;
1947 - r->o[rrdr_o_v_index] = RRDR_VALUE_EMPTY;
1948 - r->v[rrdr_o_v_index] = 0.0;
1949 - r->ar[rrdr_o_v_index] = 0.0;
1950 - points_added++;
1951 - }
1952 -
1953 - r->internal.queries_count++;
1954 - r->view.min = min;
1955 - r->view.max = max;
1956 -
1957 - r->stats.result_points_generated += points_added;
1958 - r->stats.db_points_read += ops->db_total_points_read;
1959 - for(size_t tr = 0; tr < nd_profile.storage_tiers; tr++)
1960 - qt->db.tiers[tr].points += ops->db_points_read_per_tier[tr];
1961 -}
1962 -
1963 -// ----------------------------------------------------------------------------
1964 -// fill RRDR for the whole chart
1965 -
1966 -#ifdef NETDATA_INTERNAL_CHECKS
1967 -static void rrd2rrdr_log_request_response_metadata(RRDR *r
1968 - , RRDR_OPTIONS options __maybe_unused
1969 - , RRDR_TIME_GROUPING group_method
1970 - , bool aligned
1971 - , size_t group
1972 - , time_t resampling_time
1973 - , size_t resampling_group
1974 - , time_t after_wanted
1975 - , time_t after_requested
1976 - , time_t before_wanted
1977 - , time_t before_requested
1978 - , size_t points_requested
1979 - , size_t points_wanted
1980 - //, size_t after_slot
1981 - //, size_t before_slot
1982 - , const char *msg
1983 - ) {
1984 -
1985 - QUERY_TARGET *qt = r->internal.qt;
1986 - time_t first_entry_s = qt->db.first_time_s;
1987 - time_t last_entry_s = qt->db.last_time_s;
1988 -
1989 - internal_error(
1990 - true,
1991 - "rrd2rrdr() on %s update every %ld with %s grouping %s (group: %zu, resampling_time: %ld, resampling_group: %zu), "
1992 - "after (got: %ld, want: %ld, req: %ld, db: %ld), "
1993 - "before (got: %ld, want: %ld, req: %ld, db: %ld), "
1994 - "duration (got: %ld, want: %ld, req: %ld, db: %ld), "
1995 - "points (got: %zu, want: %zu, req: %zu), "
1996 - "%s"
1997 - , qt->id
1998 - , qt->window.query_granularity
1999 -
2000 - // grouping
2001 - , (aligned) ? "aligned" : "unaligned"
2002 - , time_grouping_id2txt(group_method)
2003 - , group
2004 - , resampling_time
2005 - , resampling_group
2006 -
2007 - // after
2008 - , r->view.after
2009 - , after_wanted
2010 - , after_requested
2011 - , first_entry_s
2012 -
2013 - // before
2014 - , r->view.before
2015 - , before_wanted
2016 - , before_requested
2017 - , last_entry_s
2018 -
2019 - // duration
2020 - , (long)(r->view.before - r->view.after + qt->window.query_granularity)
2021 - , (long)(before_wanted - after_wanted + qt->window.query_granularity)
2022 - , (long)before_requested - after_requested
2023 - , (long)((last_entry_s - first_entry_s) + qt->window.query_granularity)
2024 -
2025 - // points
2026 - , r->rows
2027 - , points_wanted
2028 - , points_requested
2029 -
2030 - // message
2031 - , msg
2032 - );
2033 -}
2034 -#endif // NETDATA_INTERNAL_CHECKS
2035 -
2036 -// #define DEBUG_QUERY_LOGIC 1
2037 -
2038 -#ifdef DEBUG_QUERY_LOGIC
2039 -#define query_debug_log_init() BUFFER *debug_log = buffer_create(1000)
2040 -#define query_debug_log(args...) buffer_sprintf(debug_log, ##args)
2041 -#define query_debug_log_fin() { \
2042 - netdata_log_info("QUERY: '%s', after:%ld, before:%ld, duration:%ld, points:%zu, res:%ld - wanted => after:%ld, before:%ld, points:%zu, group:%zu, granularity:%ld, resgroup:%ld, resdiv:" NETDATA_DOUBLE_FORMAT_AUTO " %s", qt->id, after_requested, before_requested, before_requested - after_requested, points_requested, resampling_time_requested, after_wanted, before_wanted, points_wanted, group, query_granularity, resampling_group, resampling_divisor, buffer_tostring(debug_log)); \
2043 - buffer_free(debug_log); \
2044 - debug_log = NULL; \
2045 - }
2046 -#define query_debug_log_free() do { buffer_free(debug_log); } while(0)
2047 -#else
2048 -#define query_debug_log_init() debug_dummy()
2049 -#define query_debug_log(args...) debug_dummy()
2050 -#define query_debug_log_fin() debug_dummy()
2051 -#define query_debug_log_free() debug_dummy()
2052 -#endif
2053 -
2054 -bool query_target_calculate_window(QUERY_TARGET *qt) {
2055 - if (unlikely(!qt)) return false;
2056 -
2057 - size_t points_requested = (long)qt->request.points;
2058 - time_t after_requested = qt->request.after;
2059 - time_t before_requested = qt->request.before;
2060 - RRDR_TIME_GROUPING group_method = qt->request.time_group_method;
2061 - time_t resampling_time_requested = qt->request.resampling_time;
2062 - RRDR_OPTIONS options = qt->window.options;
2063 - size_t tier = qt->request.tier;
2064 - time_t update_every = qt->db.minimum_latest_update_every_s ? qt->db.minimum_latest_update_every_s : 1;
2065 -
2066 - // RULES
2067 - // points_requested = 0
2068 - // the user wants all the natural points the database has
2069 - //
2070 - // after_requested = 0
2071 - // the user wants to start the query from the oldest point in our database
2072 - //
2073 - // before_requested = 0
2074 - // the user wants the query to end to the latest point in our database
2075 - //
2076 - // when natural points are wanted, the query has to be aligned to the update_every
2077 - // of the database
2078 -
2079 - size_t points_wanted = points_requested;
2080 - time_t after_wanted = after_requested;
2081 - time_t before_wanted = before_requested;
2082 -
2083 - bool aligned = !(options & RRDR_OPTION_NOT_ALIGNED);
2084 - bool automatic_natural_points = (points_wanted == 0);
2085 - bool relative_period_requested = false;
2086 - bool natural_points = (options & RRDR_OPTION_NATURAL_POINTS) || automatic_natural_points;
2087 - bool before_is_aligned_to_db_end = false;
2088 -
2089 - query_debug_log_init();
2090 -
2091 - if (ABS(before_requested) <= API_RELATIVE_TIME_MAX || ABS(after_requested) <= API_RELATIVE_TIME_MAX) {
2092 - relative_period_requested = true;
2093 - natural_points = true;
2094 - options |= RRDR_OPTION_NATURAL_POINTS;
2095 - query_debug_log(":relative+natural");
2096 - }
2097 -
2098 - // if the user wants virtual points, make sure we do it
2099 - if (options & RRDR_OPTION_VIRTUAL_POINTS)
2100 - natural_points = false;
2101 -
2102 - // set the right flag about natural and virtual points
2103 - if (natural_points) {
2104 - options |= RRDR_OPTION_NATURAL_POINTS;
2105 -
2106 - if (options & RRDR_OPTION_VIRTUAL_POINTS)
2107 - options &= ~RRDR_OPTION_VIRTUAL_POINTS;
2108 - }
2109 - else {
2110 - options |= RRDR_OPTION_VIRTUAL_POINTS;
2111 -
2112 - if (options & RRDR_OPTION_NATURAL_POINTS)
2113 - options &= ~RRDR_OPTION_NATURAL_POINTS;
2114 - }
2115 -
2116 - if (after_wanted == 0 || before_wanted == 0) {
2117 - relative_period_requested = true;
2118 -
2119 - time_t first_entry_s = qt->db.first_time_s;
2120 - time_t last_entry_s = qt->db.last_time_s;
2121 -
2122 - if (first_entry_s == 0 || last_entry_s == 0) {
2123 - internal_error(true, "QUERY: no data detected on query '%s' (db first_entry_t = %ld, last_entry_t = %ld)", qt->id, first_entry_s, last_entry_s);
2124 - after_wanted = qt->window.after;
2125 - before_wanted = qt->window.before;
2126 -
2127 - if(after_wanted == before_wanted)
2128 - after_wanted = before_wanted - update_every;
2129 -
2130 - if (points_wanted == 0) {
2131 - points_wanted = (before_wanted - after_wanted) / update_every;
2132 - query_debug_log(":zero points_wanted %zu", points_wanted);
2133 - }
2134 - }
2135 - else {
2136 - query_debug_log(":first_entry_t %ld, last_entry_t %ld", first_entry_s, last_entry_s);
2137 -
2138 - if (after_wanted == 0) {
2139 - after_wanted = first_entry_s;
2140 - query_debug_log(":zero after_wanted %ld", after_wanted);
2141 - }
2142 -
2143 - if (before_wanted == 0) {
2144 - before_wanted = last_entry_s;
2145 - before_is_aligned_to_db_end = true;
2146 - query_debug_log(":zero before_wanted %ld", before_wanted);
2147 - }
2148 -
2149 - if (points_wanted == 0) {
2150 - points_wanted = (last_entry_s - first_entry_s) / update_every;
2151 - query_debug_log(":zero points_wanted %zu", points_wanted);
2152 - }
2153 - }
2154 - }
2155 -
2156 - if (points_wanted == 0) {
2157 - points_wanted = 600;
2158 - query_debug_log(":zero600 points_wanted %zu", points_wanted);
2159 - }
2160 -
2161 - // convert our before_wanted and after_wanted to absolute
2162 - rrdr_relative_window_to_absolute_query(&after_wanted, &before_wanted, NULL, unittest_running);
2163 - query_debug_log(":relative2absolute after %ld, before %ld", after_wanted, before_wanted);
2164 -
2165 - if (natural_points && (options & RRDR_OPTION_SELECTED_TIER) && tier > 0 && nd_profile.storage_tiers > 1) {
2166 - update_every = rrdset_find_natural_update_every_for_timeframe(
2167 - qt, after_wanted, before_wanted, points_wanted, options, tier);
2168 -
2169 - if (update_every <= 0) update_every = qt->db.minimum_latest_update_every_s;
2170 - query_debug_log(":natural update every %ld", update_every);
2171 - }
2172 -
2173 - // this is the update_every of the query
2174 - // it may be different to the update_every of the database
2175 - time_t query_granularity = (natural_points) ? update_every : 1;
2176 - if (query_granularity <= 0) query_granularity = 1;
2177 - query_debug_log(":query_granularity %ld", query_granularity);
2178 -
2179 - // align before_wanted and after_wanted to query_granularity
2180 - if (before_wanted % query_granularity) {
2181 - before_wanted -= before_wanted % query_granularity;
2182 - query_debug_log(":granularity align before_wanted %ld", before_wanted);
2183 - }
2184 -
2185 - if (after_wanted % query_granularity) {
2186 - after_wanted -= after_wanted % query_granularity;
2187 - query_debug_log(":granularity align after_wanted %ld", after_wanted);
2188 - }
2189 -
2190 - // automatic_natural_points is set when the user wants all the points available in the database
2191 - if (automatic_natural_points) {
2192 - points_wanted = (before_wanted - after_wanted + 1) / query_granularity;
2193 - if (unlikely(points_wanted <= 0)) points_wanted = 1;
2194 - query_debug_log(":auto natural points_wanted %zu", points_wanted);
2195 - }
2196 -
2197 - time_t duration = before_wanted - after_wanted;
2198 -
2199 - // if the resampling time is too big, extend the duration to the past
2200 - if (unlikely(resampling_time_requested > duration)) {
2201 - after_wanted = before_wanted - resampling_time_requested;
2202 - duration = before_wanted - after_wanted;
2203 - query_debug_log(":resampling after_wanted %ld", after_wanted);
2204 - }
2205 -
2206 - // if the duration is not aligned to resampling time
2207 - // extend the duration to the past, to avoid a gap at the chart
2208 - // only when the missing duration is above 1/10th of a point
2209 - if (resampling_time_requested > query_granularity && duration % resampling_time_requested) {
2210 - time_t delta = duration % resampling_time_requested;
2211 - if (delta > resampling_time_requested / 10) {
2212 - after_wanted -= resampling_time_requested - delta;
2213 - duration = before_wanted - after_wanted;
2214 - query_debug_log(":resampling2 after_wanted %ld", after_wanted);
2215 - }
2216 - }
2217 -
2218 - // the available points of the query
2219 - size_t points_available = (duration + 1) / query_granularity;
2220 - if (unlikely(points_available <= 0)) points_available = 1;
2221 - query_debug_log(":points_available %zu", points_available);
2222 -
2223 - if (points_wanted > points_available) {
2224 - points_wanted = points_available;
2225 - query_debug_log(":max points_wanted %zu", points_wanted);
2226 - }
2227 -
2228 - if(points_wanted > 86400 && !unittest_running) {
2229 - points_wanted = 86400;
2230 - query_debug_log(":absolute max points_wanted %zu", points_wanted);
2231 - }
2232 -
2233 - // calculate the desired grouping of source data points
2234 - size_t group = points_available / points_wanted;
2235 - if (group == 0) group = 1;
2236 -
2237 - // round "group" to the closest integer
2238 - if (points_available % points_wanted > points_wanted / 2)
2239 - group++;
2240 -
2241 - query_debug_log(":group %zu", group);
2242 -
2243 - if (points_wanted * group * query_granularity < (size_t)duration) {
2244 - // the grouping we are going to do, is not enough
2245 - // to cover the entire duration requested, so
2246 - // we have to change the number of points, to make sure we will
2247 - // respect the timeframe as closely as possibly
2248 -
2249 - // let's see how many points are the optimal
2250 - points_wanted = points_available / group;
2251 -
2252 - if (points_wanted * group < points_available)
2253 - points_wanted++;
2254 -
2255 - if (unlikely(points_wanted == 0))
2256 - points_wanted = 1;
2257 -
2258 - query_debug_log(":optimal points %zu", points_wanted);
2259 - }
2260 -
2261 - // resampling_time_requested enforces a certain grouping multiple
2262 - NETDATA_DOUBLE resampling_divisor = 1.0;
2263 - size_t resampling_group = 1;
2264 - if (unlikely(resampling_time_requested > query_granularity)) {
2265 - // the points we should group to satisfy gtime
2266 - resampling_group = resampling_time_requested / query_granularity;
2267 - if (unlikely(resampling_time_requested % query_granularity))
2268 - resampling_group++;
2269 -
2270 - query_debug_log(":resampling group %zu", resampling_group);
2271 -
2272 - // adapt group according to resampling_group
2273 - if (unlikely(group < resampling_group)) {
2274 - group = resampling_group; // do not allow grouping below the desired one
2275 - query_debug_log(":group less res %zu", group);
2276 - }
2277 - if (unlikely(group % resampling_group)) {
2278 - group += resampling_group - (group % resampling_group); // make sure group is multiple of resampling_group
2279 - query_debug_log(":group mod res %zu", group);
2280 - }
2281 -
2282 - // resampling_divisor = group / resampling_group;
2283 - resampling_divisor = (NETDATA_DOUBLE) (group * query_granularity) / (NETDATA_DOUBLE) resampling_time_requested;
2284 - query_debug_log(":resampling divisor " NETDATA_DOUBLE_FORMAT, resampling_divisor);
2285 - }
2286 -
2287 - // now that we have group, align the requested timeframe to fit it.
2288 - if (aligned && before_wanted % (group * query_granularity)) {
2289 - if (before_is_aligned_to_db_end)
2290 - before_wanted -= before_wanted % (time_t)(group * query_granularity);
2291 - else
2292 - before_wanted += (time_t)(group * query_granularity) - before_wanted % (time_t)(group * query_granularity);
2293 - query_debug_log(":align before_wanted %ld", before_wanted);
2294 - }
2295 -
2296 - after_wanted = before_wanted - (time_t)(points_wanted * group * query_granularity) + query_granularity;
2297 - query_debug_log(":final after_wanted %ld", after_wanted);
2298 -
2299 - duration = before_wanted - after_wanted;
2300 - query_debug_log(":final duration %ld", duration + 1);
2301 -
2302 - query_debug_log_fin();
2303 -
2304 - internal_error(points_wanted != duration / (query_granularity * group) + 1,
2305 - "QUERY: points_wanted %zu is not points %zu",
2306 - points_wanted, (size_t)(duration / (query_granularity * group) + 1));
2307 -
2308 - internal_error(group < resampling_group,
2309 - "QUERY: group %zu is less than the desired group points %zu",
2310 - group, resampling_group);
2311 -
2312 - internal_error(group > resampling_group && group % resampling_group,
2313 - "QUERY: group %zu is not a multiple of the desired group points %zu",
2314 - group, resampling_group);
2315 -
2316 - // -------------------------------------------------------------------------
2317 - // update QUERY_TARGET with our calculations
2318 -
2319 - qt->window.after = after_wanted;
2320 - qt->window.before = before_wanted;
2321 - qt->window.relative = relative_period_requested;
2322 - qt->window.points = points_wanted;
2323 - qt->window.group = group;
2324 - qt->window.time_group_method = group_method;
2325 - qt->window.time_group_options = qt->request.time_group_options;
2326 - qt->window.query_granularity = query_granularity;
2327 - qt->window.resampling_group = resampling_group;
2328 - qt->window.resampling_divisor = resampling_divisor;
2329 - qt->window.options = options;
2330 - qt->window.tier = tier;
2331 - qt->window.aligned = aligned;
2332 -
2333 - return true;
2334 -}
2335 -
2336 -// ----------------------------------------------------------------------------
2337 -// group by
2338 -
2339 -struct group_by_label_key {
2340 - DICTIONARY *values;
2341 -};
2342 -
2343 -static void group_by_label_key_insert_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
2344 - // add the key to our r->label_keys global keys dictionary
2345 - DICTIONARY *label_keys = data;
2346 - dictionary_set(label_keys, dictionary_acquired_item_name(item), NULL, 0);
2347 -
2348 - // create a dictionary for the values of this key
2349 - struct group_by_label_key *k = value;
2350 - k->values = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
2351 -}
2352 -
2353 -static void group_by_label_key_delete_cb(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
2354 - struct group_by_label_key *k = value;
2355 - dictionary_destroy(k->values);
2356 -}
2357 -
2358 -static int rrdlabels_traversal_cb_to_group_by_label_key(const char *name, const char *value, RRDLABEL_SRC ls __maybe_unused, void *data) {
2359 - DICTIONARY *dl = data;
2360 - struct group_by_label_key *k = dictionary_set(dl, name, NULL, sizeof(struct group_by_label_key));
2361 - dictionary_set(k->values, value, NULL, 0);
2362 - return 1;
2363 -}
2364 -
2365 -void rrdr_json_group_by_labels(BUFFER *wb, const char *key, RRDR *r, RRDR_OPTIONS options) {
2366 - if(!r->label_keys || !r->dl)
2367 - return;
2368 -
2369 - buffer_json_member_add_object(wb, key);
2370 -
2371 - void *t;
2372 - dfe_start_read(r->label_keys, t) {
2373 - buffer_json_member_add_array(wb, t_dfe.name);
2374 -
2375 - for(size_t d = 0; d < r->d ;d++) {
2376 - if(!rrdr_dimension_should_be_exposed(r->od[d], options))
2377 - continue;
2378 -
2379 - struct group_by_label_key *k = dictionary_get(r->dl[d], t_dfe.name);
2380 - if(k) {
2381 - buffer_json_add_array_item_array(wb);
2382 - void *tt;
2383 - dfe_start_read(k->values, tt) {
2384 - buffer_json_add_array_item_string(wb, tt_dfe.name);
2385 - }
2386 - dfe_done(tt);
2387 - buffer_json_array_close(wb);
2388 - }
2389 - else
2390 - buffer_json_add_array_item_string(wb, NULL);
284 + // we only log if this is not point 1
285 + internal_error(new_point.sp.end_time_s < ops->plan_expanded_after &&
286 + db_points_read_since_plan_switch > 1,
287 + "QUERY: '%s', dimension '%s' next_metric() "
288 + "returned point %zu from %ld time %ld, "
289 + "which is entirely before our current timeframe %ld to %ld "
290 + "(and before the entire query, after %ld, before %ld)",
291 + qt->id, query_metric_id(qt, qm),
292 + new_point.id, new_point.sp.start_time_s, new_point.sp.end_time_s,
293 + now_start_time, now_end_time,
294 + ops->plan_expanded_after, ops->plan_expanded_before);
295 }
296
2393 - buffer_json_array_close(wb);
2394 - }
2395 - dfe_done(t);
2396 -
2397 - buffer_json_object_close(wb); // key
2398 -}
2399 -
2400 -static void rrd2rrdr_set_timestamps(RRDR *r) {
2401 - QUERY_TARGET *qt = r->internal.qt;
2402 -
2403 - internal_fatal(qt->window.points != r->n, "QUERY: mismatch to the number of points in qt and r");
2404 -
2405 - r->view.group = qt->window.group;
2406 - r->view.update_every = (int) query_view_update_every(qt);
2407 - r->view.before = qt->window.before;
2408 - r->view.after = qt->window.after;
2409 -
2410 - r->time_grouping.points_wanted = qt->window.points;
2411 - r->time_grouping.resampling_group = qt->window.resampling_group;
2412 - r->time_grouping.resampling_divisor = qt->window.resampling_divisor;
2413 -
2414 - r->rows = qt->window.points;
2415 -
2416 - size_t points_wanted = qt->window.points;
2417 - time_t after_wanted = qt->window.after;
2418 - time_t before_wanted = qt->window.before; (void)before_wanted;
2419 -
2420 - time_t view_update_every = r->view.update_every;
2421 - time_t query_granularity = (time_t)(r->view.update_every / r->view.group);
2422 -
2423 - size_t rrdr_line = 0;
2424 - time_t first_point_end_time = after_wanted + view_update_every - query_granularity;
2425 - time_t now_end_time = first_point_end_time;
2426 -
2427 - while (rrdr_line < points_wanted) {
2428 - r->t[rrdr_line++] = now_end_time;
2429 - now_end_time += view_update_every;
2430 - }
2431 -
2432 - internal_fatal(r->t[0] != first_point_end_time, "QUERY: wrong first timestamp in the query");
2433 - internal_error(r->t[points_wanted - 1] != before_wanted,
2434 - "QUERY: wrong last timestamp in the query, expected %ld, found %ld",
2435 - before_wanted, r->t[points_wanted - 1]);
2436 -}
2437 -
2438 -static void query_group_by_make_dimension_key(BUFFER *key, RRDR_GROUP_BY group_by, size_t group_by_id, QUERY_TARGET *qt, QUERY_NODE *qn, QUERY_CONTEXT *qc, QUERY_INSTANCE *qi, QUERY_DIMENSION *qd __maybe_unused, QUERY_METRIC *qm, bool query_has_percentage_of_group) {
2439 - buffer_flush(key);
2440 - if(unlikely(!query_has_percentage_of_group && qm->status & RRDR_DIMENSION_HIDDEN)) {
2441 - buffer_strcat(key, "__hidden_dimensions__");
2442 - }
2443 - else if(unlikely(group_by & RRDR_GROUP_BY_SELECTED)) {
2444 - buffer_strcat(key, "selected");
2445 - }
2446 - else {
2447 - if (group_by & RRDR_GROUP_BY_DIMENSION) {
2448 - buffer_fast_strcat(key, "|", 1);
2449 - buffer_strcat(key, query_metric_name(qt, qm));
2450 - }
2451 -
2452 - if (group_by & (RRDR_GROUP_BY_INSTANCE|RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)) {
2453 - buffer_fast_strcat(key, "|", 1);
2454 - buffer_strcat(key, string2str(query_instance_id_fqdn(qi, qt->request.version)));
2455 - }
2456 -
2457 - if (group_by & RRDR_GROUP_BY_LABEL) {
2458 - RRDLABELS *labels = rrdinstance_acquired_labels(qi->ria);
2459 - for (size_t l = 0; l < qt->group_by[group_by_id].used; l++) {
2460 - buffer_fast_strcat(key, "|", 1);
2461 - rrdlabels_get_value_to_buffer_or_unset(labels, key, qt->group_by[group_by_id].label_keys[l], "[unset]");
2462 - }
2463 - }
2464 -
2465 - if (group_by & RRDR_GROUP_BY_NODE) {
2466 - buffer_fast_strcat(key, "|", 1);
2467 - buffer_strcat(key, qn->rrdhost->machine_guid);
2468 - }
2469 -
2470 - if (group_by & RRDR_GROUP_BY_CONTEXT) {
2471 - buffer_fast_strcat(key, "|", 1);
2472 - buffer_strcat(key, rrdcontext_acquired_id(qc->rca));
2473 - }
2474 -
2475 - if (group_by & RRDR_GROUP_BY_UNITS) {
2476 - buffer_fast_strcat(key, "|", 1);
2477 - buffer_strcat(key, query_target_has_percentage_units(qt) ? "%" : rrdinstance_acquired_units(qi->ria));
2478 - }
2479 - }
2480 -}
2481 -
2482 -static void query_group_by_make_dimension_id(BUFFER *key, RRDR_GROUP_BY group_by, size_t group_by_id, QUERY_TARGET *qt, QUERY_NODE *qn, QUERY_CONTEXT *qc, QUERY_INSTANCE *qi, QUERY_DIMENSION *qd __maybe_unused, QUERY_METRIC *qm, bool query_has_percentage_of_group) {
2483 - buffer_flush(key);
2484 - if(unlikely(!query_has_percentage_of_group && qm->status & RRDR_DIMENSION_HIDDEN)) {
2485 - buffer_strcat(key, "__hidden_dimensions__");
2486 - }
2487 - else if(unlikely(group_by & RRDR_GROUP_BY_SELECTED)) {
2488 - buffer_strcat(key, "selected");
2489 - }
2490 - else {
2491 - if (group_by & RRDR_GROUP_BY_DIMENSION) {
2492 - buffer_strcat(key, query_metric_name(qt, qm));
2493 - }
2494 -
2495 - if (group_by & (RRDR_GROUP_BY_INSTANCE|RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)) {
2496 - if (buffer_strlen(key) != 0)
2497 - buffer_fast_strcat(key, ",", 1);
2498 -
2499 - if (group_by & RRDR_GROUP_BY_NODE)
2500 - buffer_strcat(key, rrdinstance_acquired_id(qi->ria));
2501 - else
2502 - buffer_strcat(key, string2str(query_instance_id_fqdn(qi, qt->request.version)));
2503 - }
2504 -
2505 - if (group_by & RRDR_GROUP_BY_LABEL) {
2506 - RRDLABELS *labels = rrdinstance_acquired_labels(qi->ria);
2507 - for (size_t l = 0; l < qt->group_by[group_by_id].used; l++) {
2508 - if (buffer_strlen(key) != 0)
2509 - buffer_fast_strcat(key, ",", 1);
2510 - rrdlabels_get_value_to_buffer_or_unset(labels, key, qt->group_by[group_by_id].label_keys[l], "[unset]");
2511 - }
2512 - }
2513 -
2514 - if (group_by & RRDR_GROUP_BY_NODE) {
2515 - if (buffer_strlen(key) != 0)
2516 - buffer_fast_strcat(key, ",", 1);
2517 -
2518 - buffer_strcat(key, qn->rrdhost->machine_guid);
2519 - }
2520 -
2521 - if (group_by & RRDR_GROUP_BY_CONTEXT) {
2522 - if (buffer_strlen(key) != 0)
2523 - buffer_fast_strcat(key, ",", 1);
2524 -
2525 - buffer_strcat(key, rrdcontext_acquired_id(qc->rca));
2526 - }
2527 -
2528 - if (group_by & RRDR_GROUP_BY_UNITS) {
2529 - if (buffer_strlen(key) != 0)
2530 - buffer_fast_strcat(key, ",", 1);
2531 -
2532 - buffer_strcat(key, query_target_has_percentage_units(qt) ? "%" : rrdinstance_acquired_units(qi->ria));
2533 - }
2534 - }
2535 -}
2536 -
2537 -static void query_group_by_make_dimension_name(BUFFER *key, RRDR_GROUP_BY group_by, size_t group_by_id, QUERY_TARGET *qt, QUERY_NODE *qn, QUERY_CONTEXT *qc, QUERY_INSTANCE *qi, QUERY_DIMENSION *qd __maybe_unused, QUERY_METRIC *qm, bool query_has_percentage_of_group) {
2538 - buffer_flush(key);
2539 - if(unlikely(!query_has_percentage_of_group && qm->status & RRDR_DIMENSION_HIDDEN)) {
2540 - buffer_strcat(key, "__hidden_dimensions__");
2541 - }
2542 - else if(unlikely(group_by & RRDR_GROUP_BY_SELECTED)) {
2543 - buffer_strcat(key, "selected");
2544 - }
2545 - else {
2546 - if (group_by & RRDR_GROUP_BY_DIMENSION) {
2547 - buffer_strcat(key, query_metric_name(qt, qm));
2548 - }
2549 -
2550 - if (group_by & (RRDR_GROUP_BY_INSTANCE|RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)) {
2551 - if (buffer_strlen(key) != 0)
2552 - buffer_fast_strcat(key, ",", 1);
2553 -
2554 - if (group_by & RRDR_GROUP_BY_NODE)
2555 - buffer_strcat(key, rrdinstance_acquired_name(qi->ria));
2556 - else
2557 - buffer_strcat(key, string2str(query_instance_name_fqdn(qi, qt->request.version)));
2558 - }
2559 -
2560 - if (group_by & RRDR_GROUP_BY_LABEL) {
2561 - RRDLABELS *labels = rrdinstance_acquired_labels(qi->ria);
2562 - for (size_t l = 0; l < qt->group_by[group_by_id].used; l++) {
2563 - if (buffer_strlen(key) != 0)
2564 - buffer_fast_strcat(key, ",", 1);
2565 - rrdlabels_get_value_to_buffer_or_unset(labels, key, qt->group_by[group_by_id].label_keys[l], "[unset]");
2566 - }
2567 - }
2568 -
2569 - if (group_by & RRDR_GROUP_BY_NODE) {
2570 - if (buffer_strlen(key) != 0)
2571 - buffer_fast_strcat(key, ",", 1);
2572 -
2573 - buffer_strcat(key, rrdhost_hostname(qn->rrdhost));
2574 - }
2575 -
2576 - if (group_by & RRDR_GROUP_BY_CONTEXT) {
2577 - if (buffer_strlen(key) != 0)
2578 - buffer_fast_strcat(key, ",", 1);
2579 -
2580 - buffer_strcat(key, rrdcontext_acquired_id(qc->rca));
2581 - }
2582 -
2583 - if (group_by & RRDR_GROUP_BY_UNITS) {
2584 - if (buffer_strlen(key) != 0)
2585 - buffer_fast_strcat(key, ",", 1);
2586 -
2587 - buffer_strcat(key, query_target_has_percentage_units(qt) ? "%" : rrdinstance_acquired_units(qi->ria));
2588 - }
2589 - }
2590 -}
2591 -
2592 -struct rrdr_group_by_entry {
2593 - size_t priority;
2594 - size_t count;
2595 - STRING *id;
2596 - STRING *name;
2597 - STRING *units;
2598 - RRDR_DIMENSION_FLAGS od;
2599 - DICTIONARY *dl;
2600 -};
2601 -
2602 -static RRDR *rrd2rrdr_group_by_initialize(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
2603 - RRDR *r_tmp = NULL;
2604 - RRDR_OPTIONS options = qt->window.options;
2605 -
2606 - if(qt->request.version < 2) {
2607 - // v1 query
2608 - RRDR *r = rrdr_create(owa, qt, qt->query.used, qt->window.points);
2609 - if(unlikely(!r)) {
2610 - internal_error(true, "QUERY: cannot create RRDR for %s, after=%ld, before=%ld, dimensions=%u, points=%zu",
2611 - qt->id, qt->window.after, qt->window.before, qt->query.used, qt->window.points);
2612 - return NULL;
2613 - }
2614 - r->group_by.r = NULL;
2615 -
2616 - for(size_t d = 0; d < qt->query.used ; d++) {
2617 - QUERY_METRIC *qm = query_metric(qt, d);
2618 - QUERY_DIMENSION *qd = query_dimension(qt, qm->link.query_dimension_id);
2619 - r->di[d] = rrdmetric_acquired_id_dup(qd->rma);
2620 - r->dn[d] = rrdmetric_acquired_name_dup(qd->rma);
2621 - }
2622 -
2623 - rrd2rrdr_set_timestamps(r);
2624 - return r;
2625 - }
2626 - // v2 query
2627 -
2628 - // parse all the group-by label keys
2629 - for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
2630 - if (qt->request.group_by[g].group_by & RRDR_GROUP_BY_LABEL &&
2631 - qt->request.group_by[g].group_by_label && *qt->request.group_by[g].group_by_label)
2632 - qt->group_by[g].used = quoted_strings_splitter_query_group_by_label(
2633 - qt->request.group_by[g].group_by_label, qt->group_by[g].label_keys,
2634 - GROUP_BY_MAX_LABEL_KEYS);
2635 -
2636 - if (!qt->group_by[g].used)
2637 - qt->request.group_by[g].group_by &= ~RRDR_GROUP_BY_LABEL;
2638 - }
2639 -
2640 - // make sure there are valid group-by methods
2641 - for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
2642 - if(!(qt->request.group_by[g].group_by & SUPPORTED_GROUP_BY_METHODS))
2643 - qt->request.group_by[g].group_by = (g == 0) ? RRDR_GROUP_BY_DIMENSION : RRDR_GROUP_BY_NONE;
2644 - }
2645 -
2646 - bool query_has_percentage_of_group = query_target_has_percentage_of_group(qt);
2647 -
2648 - // merge all group-by options to upper levels,
2649 - // so that the top level has all the groupings of the inner levels,
2650 - // and each subsequent level has all the groupings of its inner levels.
2651 - for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES - 1 ;g++) {
2652 - if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_NONE)
2653 - continue;
2654 -
2655 - if(qt->request.group_by[g].group_by == RRDR_GROUP_BY_SELECTED) {
2656 - for (size_t r = g + 1; r < MAX_QUERY_GROUP_BY_PASSES; r++)
2657 - qt->request.group_by[r].group_by = RRDR_GROUP_BY_NONE;
2658 - }
2659 - else {
2660 - for (size_t r = g + 1; r < MAX_QUERY_GROUP_BY_PASSES; r++) {
2661 - if (qt->request.group_by[r].group_by == RRDR_GROUP_BY_NONE)
2662 - continue;
2663 -
2664 - if (qt->request.group_by[r].group_by != RRDR_GROUP_BY_SELECTED) {
2665 - if(qt->request.group_by[r].group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE)
2666 - qt->request.group_by[g].group_by |= RRDR_GROUP_BY_INSTANCE;
2667 - else
2668 - qt->request.group_by[g].group_by |= qt->request.group_by[r].group_by;
2669 -
2670 - if(qt->request.group_by[r].group_by & RRDR_GROUP_BY_LABEL) {
2671 - for (size_t lr = 0; lr < qt->group_by[r].used; lr++) {
2672 - bool found = false;
2673 - for (size_t lg = 0; lg < qt->group_by[g].used; lg++) {
2674 - if (strcmp(qt->group_by[g].label_keys[lg], qt->group_by[r].label_keys[lr]) == 0) {
2675 - found = true;
2676 - break;
2677 - }
2678 - }
2679 -
2680 - if (!found && qt->group_by[g].used < GROUP_BY_MAX_LABEL_KEYS * MAX_QUERY_GROUP_BY_PASSES)
2681 - qt->group_by[g].label_keys[qt->group_by[g].used++] = qt->group_by[r].label_keys[lr];
2682 - }
2683 - }
2684 - }
297 + }
298 + else {
299 + // the point ends in the future
300 + // so, we will interpolate it below, at the inner loop
301 + break;
302 }
303 }
2687 - }
304
2689 - int added = 0;
2690 - RRDR *first_r = NULL, *last_r = NULL;
2691 - BUFFER *key = buffer_create(0, NULL);
2692 - struct rrdr_group_by_entry *entries = onewayalloc_mallocz(owa, qt->query.used * sizeof(struct rrdr_group_by_entry));
2693 - DICTIONARY *groups = dictionary_create(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE);
2694 - DICTIONARY *label_keys = NULL;
305 + if(unlikely(count_same_end_time)) {
306 + internal_error(true,
307 + "QUERY: '%s', dimension '%s', the database does not advance the query,"
308 + " it returned an end time less or equal to the end time of the last "
309 + "point we got %ld, %zu times",
310 + qt->id, query_metric_id(qt, qm),
311 + last1_point.sp.end_time_s, count_same_end_time);
312
2696 - for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++) {
2697 - RRDR_GROUP_BY group_by = qt->request.group_by[g].group_by;
2698 - RRDR_GROUP_BY_FUNCTION aggregation_method = qt->request.group_by[g].aggregation;
313 + if(unlikely(new_point.sp.end_time_s <= last1_point.sp.end_time_s))
314 + new_point.sp.end_time_s = now_end_time;
315 + }
316
2700 - if(group_by == RRDR_GROUP_BY_NONE)
2701 - break;
317 + time_t stop_time = new_point.sp.end_time_s;
318 + if(unlikely(!storage_point_is_unset(next1_point) && next1_point.start_time_s >= now_end_time)) {
319 + // ONE POINT READ-AHEAD
320 + // the point crosses the start time of the
321 + // read ahead storage point we have read
322 + stop_time = next1_point.start_time_s;
323 + }
324
2703 - memset(entries, 0, qt->query.used * sizeof(struct rrdr_group_by_entry));
2704 - dictionary_flush(groups);
2705 - added = 0;
325 + // the inner loop
326 + // we have 3 points in memory: last2, last1, new
327 + // we select the one to use based on their timestamps
328
2707 - size_t hidden_dimensions = 0;
2708 - bool final_grouping = (g == MAX_QUERY_GROUP_BY_PASSES - 1 || qt->request.group_by[g + 1].group_by == RRDR_GROUP_BY_NONE) ? true : false;
329 + internal_fatal(now_end_time > stop_time || points_added >= points_wanted,
330 + "QUERY: first part of query provides invalid point to interpolate (now_end_time %ld, stop_time %ld",
331 + now_end_time, stop_time);
332
2710 - if (final_grouping && (options & RRDR_OPTION_GROUP_BY_LABELS))
2711 - label_keys = dictionary_create_advanced(DICT_OPTION_SINGLE_THREADED | DICT_OPTION_DONT_OVERWRITE_VALUE, NULL, 0);
333 + do {
334 + // now_start_time is wrong in this loop
335 + // but, we don't need it
336
2713 - QUERY_INSTANCE *last_qi = NULL;
2714 - size_t priority = 0;
2715 - time_t update_every_max = 0;
2716 - for (size_t d = 0; d < qt->query.used; d++) {
2717 - QUERY_METRIC *qm = query_metric(qt, d);
2718 - QUERY_DIMENSION *qd = query_dimension(qt, qm->link.query_dimension_id);
2719 - QUERY_INSTANCE *qi = query_instance(qt, qm->link.query_instance_id);
2720 - QUERY_CONTEXT *qc = query_context(qt, qm->link.query_context_id);
2721 - QUERY_NODE *qn = query_node(qt, qm->link.query_node_id);
337 + QUERY_POINT current_point;
338
2723 - if (qi != last_qi) {
2724 - last_qi = qi;
339 + if(likely(now_end_time > new_point.sp.start_time_s)) {
340 + // it is time for our NEW point to be used
341 + current_point = new_point;
342 + new_point.added = true; // first copy, then set it, so that new_point will not be added again
343 + query_interpolate_point(current_point, last1_point, now_end_time);
344
2726 - time_t update_every = rrdinstance_acquired_update_every(qi->ria);
2727 - if (update_every > update_every_max)
2728 - update_every_max = update_every;
345 +// internal_error(current_point.id > 0
346 +// && last1_point.id == 0
347 +// && current_point.end_time > after_wanted
348 +// && current_point.end_time > now_end_time,
349 +// "QUERY: '%s', dimension '%s', after %ld, before %ld, view update every %ld,"
350 +// " query granularity %ld, interpolating point %zu (from %ld to %ld) at %ld,"
351 +// " but we could really favor by having last_point1 in this query.",
352 +// qt->id, string2str(qm->dimension.id),
353 +// after_wanted, before_wanted,
354 +// ops.view_update_every, ops.query_granularity,
355 +// current_point.id, current_point.start_time, current_point.end_time,
356 +// now_end_time);
357 }
358 + else if(likely(now_end_time <= last1_point.sp.end_time_s)) {
359 + // our LAST point is still valid
360 + current_point = last1_point;
361 + last1_point.added = true; // first copy, then set it, so that last1_point will not be added again
362 + query_interpolate_point(current_point, last2_point, now_end_time);
363
2731 - priority = qd->priority;
2732 -
2733 - if(qm->status & RRDR_DIMENSION_HIDDEN)
2734 - hidden_dimensions++;
2735 -
2736 - // --------------------------------------------------------------------
2737 - // generate the group by key
2738 -
2739 - query_group_by_make_dimension_key(key, group_by, g, qt, qn, qc, qi, qd, qm, query_has_percentage_of_group);
2740 -
2741 - // lookup the key in the dictionary
2742 -
2743 - int pos = -1;
2744 - int *set = dictionary_set(groups, buffer_tostring(key), &pos, sizeof(pos));
2745 - if (*set == -1) {
2746 - // the key just added to the dictionary
2747 -
2748 - *set = pos = added++;
2749 -
2750 - // ----------------------------------------------------------------
2751 - // generate the dimension id
2752 -
2753 - query_group_by_make_dimension_id(key, group_by, g, qt, qn, qc, qi, qd, qm, query_has_percentage_of_group);
2754 - entries[pos].id = string_strdupz(buffer_tostring(key));
2755 -
2756 - // ----------------------------------------------------------------
2757 - // generate the dimension name
2758 -
2759 - query_group_by_make_dimension_name(key, group_by, g, qt, qn, qc, qi, qd, qm, query_has_percentage_of_group);
2760 - entries[pos].name = string_strdupz(buffer_tostring(key));
2761 -
2762 - // add the rest of the info
2763 - entries[pos].units = rrdinstance_acquired_units_dup(qi->ria);
2764 - entries[pos].priority = priority;
2765 -
2766 - if (label_keys) {
2767 - entries[pos].dl = dictionary_create_advanced(
2768 - DICT_OPTION_SINGLE_THREADED | DICT_OPTION_FIXED_SIZE | DICT_OPTION_DONT_OVERWRITE_VALUE,
2769 - NULL, sizeof(struct group_by_label_key));
2770 - dictionary_register_insert_callback(entries[pos].dl, group_by_label_key_insert_cb, label_keys);
2771 - dictionary_register_delete_callback(entries[pos].dl, group_by_label_key_delete_cb, label_keys);
2772 - }
2773 - } else {
2774 - // the key found in the dictionary
2775 - pos = *set;
364 +// internal_error(current_point.id > 0
365 +// && last2_point.id == 0
366 +// && current_point.end_time > after_wanted
367 +// && current_point.end_time > now_end_time,
368 +// "QUERY: '%s', dimension '%s', after %ld, before %ld, view update every %ld,"
369 +// " query granularity %ld, interpolating point %zu (from %ld to %ld) at %ld,"
370 +// " but we could really favor by having last_point2 in this query.",
371 +// qt->id, string2str(qm->dimension.id),
372 +// after_wanted, before_wanted, ops.view_update_every, ops.query_granularity,
373 +// current_point.id, current_point.start_time, current_point.end_time,
374 +// now_end_time);
375 + }
376 + else {
377 + // a GAP, we don't have a value this time
378 + current_point = QUERY_POINT_EMPTY;
379 }
380
2778 - entries[pos].count++;
2779 -
2780 - if (unlikely(priority < entries[pos].priority))
2781 - entries[pos].priority = priority;
381 + query_add_point_to_group(r, current_point, ops, add_flush);
382
2783 - if(g > 0)
2784 - last_r->dgbs[qm->grouped_as.slot] = pos;
2785 - else
2786 - qm->grouped_as.first_slot = pos;
2787 -
2788 - qm->grouped_as.slot = pos;
2789 - qm->grouped_as.id = entries[pos].id;
2790 - qm->grouped_as.name = entries[pos].name;
2791 - qm->grouped_as.units = entries[pos].units;
2792 -
2793 - // copy the dimension flags decided by the query target
2794 - // we need this, because if a dimension is explicitly selected
2795 - // the query target adds to it the non-zero flag
2796 - qm->status |= RRDR_DIMENSION_GROUPED;
2797 -
2798 - if(query_has_percentage_of_group)
2799 - // when the query has percentage of group
2800 - // there will be no hidden dimensions in the final query,
2801 - // so we have to remove the hidden flag from all dimensions
2802 - entries[pos].od |= qm->status & ~RRDR_DIMENSION_HIDDEN;
2803 - else
2804 - entries[pos].od |= qm->status;
383 + rrdr_line = rrdr_line_init(r, now_end_time, rrdr_line);
384 + size_t rrdr_o_v_index = rrdr_line * r->d + dim_id_in_rrdr;
385
2806 - if (entries[pos].dl)
2807 - rrdlabels_walkthrough_read(rrdinstance_acquired_labels(qi->ria),
2808 - rrdlabels_traversal_cb_to_group_by_label_key, entries[pos].dl);
2809 - }
386 + // find the place to store our values
387 + RRDR_VALUE_FLAGS *rrdr_value_options_ptr = &r->o[rrdr_o_v_index];
388
2811 - RRDR *r = rrdr_create(owa, qt, added, qt->window.points);
2812 - if (!r) {
2813 - internal_error(true,
2814 - "QUERY: cannot create group by RRDR for %s, after=%ld, before=%ld, dimensions=%d, points=%zu",
2815 - qt->id, qt->window.after, qt->window.before, added, qt->window.points);
2816 - goto cleanup;
2817 - }
2818 - // prevent double free at cleanup in case of error
2819 - added = 0;
389 + // update the dimension options
390 + if(likely(ops->group_points_non_zero))
391 + r->od[dim_id_in_rrdr] |= RRDR_DIMENSION_NONZERO;
392
2821 - // link this RRDR
2822 - if(!last_r)
2823 - first_r = last_r = r;
2824 - else
2825 - last_r->group_by.r = r;
393 + // store the specific point options
394 + *rrdr_value_options_ptr = ops->group_value_flags;
395
2827 - last_r = r;
396 + // store the group value
397 + NETDATA_DOUBLE group_value = time_grouping_flush(r, rrdr_value_options_ptr, add_flush);
398 + r->v[rrdr_o_v_index] = group_value;
399
2829 - rrd2rrdr_set_timestamps(r);
400 + r->ar[rrdr_o_v_index] = storage_point_anomaly_rate(ops->group_point);
401
2831 - if(r->d) {
2832 - r->dp = onewayalloc_callocz(owa, r->d, sizeof(*r->dp));
2833 - r->dview = onewayalloc_callocz(owa, r->d, sizeof(*r->dview));
2834 - r->dgbc = onewayalloc_callocz(owa, r->d, sizeof(*r->dgbc));
2835 - r->dqp = onewayalloc_callocz(owa, r->d, sizeof(STORAGE_POINT));
402 + if(likely(points_added || r->internal.queries_count)) {
403 + // find the min/max across all dimensions
404
2837 - if(!final_grouping)
2838 - // this is where we are going to store the slot in the next RRDR
2839 - // that we are going to group by the dimension of this RRDR
2840 - r->dgbs = onewayalloc_callocz(owa, r->d, sizeof(*r->dgbs));
405 + if(unlikely(group_value < min)) min = group_value;
406 + if(unlikely(group_value > max)) max = group_value;
407
2842 - if (label_keys) {
2843 - r->dl = onewayalloc_callocz(owa, r->d, sizeof(DICTIONARY *));
2844 - r->label_keys = label_keys;
2845 - label_keys = NULL;
408 }
2847 -
2848 - if(r->n) {
2849 - r->gbc = onewayalloc_callocz(owa, r->n * r->d, sizeof(*r->gbc));
2850 -
2851 - if(hidden_dimensions && ((group_by & RRDR_GROUP_BY_PERCENTAGE_OF_INSTANCE) || (aggregation_method == RRDR_GROUP_BY_FUNCTION_PERCENTAGE)))
2852 - // this is where we are going to group the hidden dimensions
2853 - r->vh = onewayalloc_mallocz(owa, r->n * r->d * sizeof(*r->vh));
409 + else {
410 + // runs only when r->internal.queries_count == 0 && points_added == 0
411 + // so, on the first point added for the query.
412 + min = max = group_value;
413 }
2855 - }
2856 -
2857 - // zero r (dimension options, names, and ids)
2858 - // this is required, because group-by may lead to empty dimensions
2859 - for (size_t d = 0; d < r->d; d++) {
2860 - r->di[d] = entries[d].id;
2861 - r->dn[d] = entries[d].name;
2862 -
2863 - r->od[d] = entries[d].od;
2864 - r->du[d] = entries[d].units;
2865 - r->dp[d] = entries[d].priority;
2866 - r->dgbc[d] = entries[d].count;
2867 -
2868 - if (r->dl)
2869 - r->dl[d] = entries[d].dl;
2870 - }
414
2872 - // initialize partial trimming
2873 - r->partial_data_trimming.max_update_every = update_every_max * 2;
2874 - r->partial_data_trimming.expected_after =
2875 - (!query_target_aggregatable(qt) &&
2876 - qt->window.before >= qt->window.now - r->partial_data_trimming.max_update_every) ?
2877 - qt->window.before - r->partial_data_trimming.max_update_every :
2878 - qt->window.before;
2879 - r->partial_data_trimming.trimmed_after = qt->window.before;
2880 -
2881 - // make all values empty
2882 - if(r->n && r->d) {
2883 - for (size_t i = 0; i != r->n; i++) {
2884 - NETDATA_DOUBLE *cn = &r->v[i * r->d];
2885 - RRDR_VALUE_FLAGS *co = &r->o[i * r->d];
2886 - NETDATA_DOUBLE *ar = &r->ar[i * r->d];
2887 - NETDATA_DOUBLE *vh = r->vh ? &r->vh[i * r->d] : NULL;
2888 -
2889 - for (size_t d = 0; d < r->d; d++) {
2890 - cn[d] = NAN;
2891 - ar[d] = 0.0;
2892 - co[d] = RRDR_VALUE_EMPTY;
2893 -
2894 - if (vh)
2895 - vh[d] = NAN;
2896 - }
2897 - }
2898 - }
2899 - }
415 + points_added++;
416 + ops->group_points_added = 0;
417 + ops->group_value_flags = RRDR_VALUE_NOTHING;
418 + ops->group_points_non_zero = 0;
419 + ops->group_point = STORAGE_POINT_UNSET;
420
2901 - if(!first_r || !last_r)
2902 - goto cleanup;
421 + now_end_time += ops->view_update_every;
422 + } while(now_end_time <= stop_time && points_added < points_wanted);
423
2904 - r_tmp = rrdr_create(owa, qt, 1, qt->window.points);
2905 - if (!r_tmp) {
2906 - internal_error(true,
2907 - "QUERY: cannot create group by temporary RRDR for %s, after=%ld, before=%ld, dimensions=%d, points=%zu",
2908 - qt->id, qt->window.after, qt->window.before, 1, qt->window.points);
2909 - goto cleanup;
424 + // the loop above increased "now" by ops->view_update_every,
425 + // but the main loop will increase it too,
426 + // so, let's undo the last iteration of this loop
427 + now_end_time -= ops->view_update_every;
428 }
2911 - rrd2rrdr_set_timestamps(r_tmp);
2912 - r_tmp->group_by.r = first_r;
2913 -
2914 -cleanup:
2915 - if(!first_r || !last_r || !r_tmp) {
2916 - if(r_tmp) {
2917 - r_tmp->group_by.r = NULL;
2918 - rrdr_free(owa, r_tmp);
2919 - }
2920 -
2921 - if(first_r) {
2922 - RRDR *r = first_r;
2923 - while (r) {
2924 - r_tmp = r->group_by.r;
2925 - r->group_by.r = NULL;
2926 - rrdr_free(owa, r);
2927 - r = r_tmp;
2928 - }
2929 - }
429 + query_planer_finalize_remaining_plans(ops);
430
2931 - if(entries && added) {
2932 - for (int d = 0; d < added; d++) {
2933 - string_freez(entries[d].id);
2934 - string_freez(entries[d].name);
2935 - string_freez(entries[d].units);
2936 - dictionary_destroy(entries[d].dl);
2937 - }
2938 - }
2939 - dictionary_destroy(label_keys);
431 + qm->query_points = ops->query_point;
432
2941 - first_r = last_r = r_tmp = NULL;
433 + // fill the rest of the points with empty values
434 + while (points_added < points_wanted) {
435 + rrdr_line++;
436 + size_t rrdr_o_v_index = rrdr_line * r->d + dim_id_in_rrdr;
437 + r->o[rrdr_o_v_index] = RRDR_VALUE_EMPTY;
438 + r->v[rrdr_o_v_index] = 0.0;
439 + r->ar[rrdr_o_v_index] = 0.0;
440 + points_added++;
441 }
442
2944 - buffer_free(key);
2945 - onewayalloc_freez(owa, entries);
2946 - dictionary_destroy(groups);
443 + r->internal.queries_count++;
444 + r->view.min = min;
445 + r->view.max = max;
446
2948 - return r_tmp;
447 + r->stats.result_points_generated += points_added;
448 + r->stats.db_points_read += ops->db_total_points_read;
449 + for(size_t tr = 0; tr < nd_profile.storage_tiers; tr++)
450 + qt->db.tiers[tr].points += ops->db_points_read_per_tier[tr];
451 }
452
2951 -static void rrd2rrdr_group_by_add_metric(RRDR *r_dst, size_t d_dst, RRDR *r_tmp, size_t d_tmp,
2952 - RRDR_GROUP_BY_FUNCTION group_by_aggregate_function,
2953 - STORAGE_POINT *query_points, size_t pass __maybe_unused) {
2954 - if(!r_tmp || r_dst == r_tmp || !(r_tmp->od[d_tmp] & RRDR_DIMENSION_QUERIED))
2955 - return;
2956 -
2957 - internal_fatal(r_dst->n != r_tmp->n, "QUERY: group-by source and destination do not have the same number of rows");
2958 - internal_fatal(d_dst >= r_dst->d, "QUERY: group-by destination dimension number exceeds destination RRDR size");
2959 - internal_fatal(d_tmp >= r_tmp->d, "QUERY: group-by source dimension number exceeds source RRDR size");
2960 - internal_fatal(!r_dst->dqp, "QUERY: group-by destination is not properly prepared (missing dqp array)");
2961 - internal_fatal(!r_dst->gbc, "QUERY: group-by destination is not properly prepared (missing gbc array)");
453 +// ----------------------------------------------------------------------------
454 +// fill RRDR for the whole chart
455
2963 - bool hidden_dimension_on_percentage_of_group = (r_tmp->od[d_tmp] & RRDR_DIMENSION_HIDDEN) && r_dst->vh;
456 +#ifdef NETDATA_INTERNAL_CHECKS
457 +static void rrd2rrdr_log_request_response_metadata(RRDR *r
458 + , RRDR_OPTIONS options __maybe_unused
459 + , RRDR_TIME_GROUPING group_method
460 + , bool aligned
461 + , size_t group
462 + , time_t resampling_time
463 + , size_t resampling_group
464 + , time_t after_wanted
465 + , time_t after_requested
466 + , time_t before_wanted
467 + , time_t before_requested
468 + , size_t points_requested
469 + , size_t points_wanted
470 + //, size_t after_slot
471 + //, size_t before_slot
472 + , const char *msg
473 + ) {
474
2965 - if(!hidden_dimension_on_percentage_of_group) {
2966 - r_dst->od[d_dst] |= r_tmp->od[d_tmp];
2967 - storage_point_merge_to(r_dst->dqp[d_dst], *query_points);
2968 - }
475 + QUERY_TARGET *qt = r->internal.qt;
476 + time_t first_entry_s = qt->db.first_time_s;
477 + time_t last_entry_s = qt->db.last_time_s;
478
2970 - // do the group_by
2971 - for(size_t i = 0; i != rrdr_rows(r_tmp) ; i++) {
479 + internal_error(
480 + true,
481 + "rrd2rrdr() on %s update every %ld with %s grouping %s (group: %zu, resampling_time: %ld, resampling_group: %zu), "
482 + "after (got: %ld, want: %ld, req: %ld, db: %ld), "
483 + "before (got: %ld, want: %ld, req: %ld, db: %ld), "
484 + "duration (got: %ld, want: %ld, req: %ld, db: %ld), "
485 + "points (got: %zu, want: %zu, req: %zu), "
486 + "%s"
487 + , qt->id
488 + , qt->window.query_granularity
489
2973 - size_t idx_tmp = i * r_tmp->d + d_tmp;
2974 - NETDATA_DOUBLE n_tmp = r_tmp->v[ idx_tmp ];
2975 - RRDR_VALUE_FLAGS o_tmp = r_tmp->o[ idx_tmp ];
2976 - NETDATA_DOUBLE ar_tmp = r_tmp->ar[ idx_tmp ];
490 + // grouping
491 + , (aligned) ? "aligned" : "unaligned"
492 + , time_grouping_id2txt(group_method)
493 + , group
494 + , resampling_time
495 + , resampling_group
496
2978 - if(o_tmp & RRDR_VALUE_EMPTY)
2979 - continue;
497 + // after
498 + , r->view.after
499 + , after_wanted
500 + , after_requested
501 + , first_entry_s
502
2981 - size_t idx_dst = i * r_dst->d + d_dst;
2982 - NETDATA_DOUBLE *cn = (hidden_dimension_on_percentage_of_group) ? &r_dst->vh[ idx_dst ] : &r_dst->v[ idx_dst ];
2983 - RRDR_VALUE_FLAGS *co = &r_dst->o[ idx_dst ];
2984 - NETDATA_DOUBLE *ar = &r_dst->ar[ idx_dst ];
2985 - uint32_t *gbc = &r_dst->gbc[ idx_dst ];
2986 -
2987 - switch(group_by_aggregate_function) {
2988 - default:
2989 - case RRDR_GROUP_BY_FUNCTION_AVERAGE:
2990 - case RRDR_GROUP_BY_FUNCTION_SUM:
2991 - case RRDR_GROUP_BY_FUNCTION_PERCENTAGE:
2992 - if(isnan(*cn))
2993 - *cn = n_tmp;
2994 - else
2995 - *cn += n_tmp;
2996 - break;
503 + // before
504 + , r->view.before
505 + , before_wanted
506 + , before_requested
507 + , last_entry_s
508
2998 - case RRDR_GROUP_BY_FUNCTION_MIN:
2999 - if(isnan(*cn) || n_tmp < *cn)
3000 - *cn = n_tmp;
3001 - break;
509 + // duration
510 + , (long)(r->view.before - r->view.after + qt->window.query_granularity)
511 + , (long)(before_wanted - after_wanted + qt->window.query_granularity)
512 + , (long)before_requested - after_requested
513 + , (long)((last_entry_s - first_entry_s) + qt->window.query_granularity)
514
3003 - case RRDR_GROUP_BY_FUNCTION_MAX:
3004 - if(isnan(*cn) || n_tmp > *cn)
3005 - *cn = n_tmp;
3006 - break;
3007 - }
515 + // points
516 + , r->rows
517 + , points_wanted
518 + , points_requested
519
3009 - if(!hidden_dimension_on_percentage_of_group) {
3010 - *co &= ~RRDR_VALUE_EMPTY;
3011 - *co |= (o_tmp & (RRDR_VALUE_RESET | RRDR_VALUE_PARTIAL));
3012 - *ar += ar_tmp;
3013 - (*gbc)++;
3014 - }
3015 - }
520 + // message
521 + , msg
522 + );
523 }
524 +#endif // NETDATA_INTERNAL_CHECKS
525
3018 -static void rrdr2rrdr_group_by_partial_trimming(RRDR *r) {
3019 - time_t trimmable_after = r->partial_data_trimming.expected_after;
526 +// #define DEBUG_QUERY_LOGIC 1
527
3021 - // find the point just before the trimmable ones
3022 - ssize_t i = (ssize_t)r->n - 1;
3023 - for( ; i >= 0 ;i--) {
3024 - if (r->t[i] < trimmable_after)
3025 - break;
528 +#ifdef DEBUG_QUERY_LOGIC
529 +#define query_debug_log_init() BUFFER *debug_log = buffer_create(1000)
530 +#define query_debug_log(args...) buffer_sprintf(debug_log, ##args)
531 +#define query_debug_log_fin() { \
532 + netdata_log_info("QUERY: '%s', after:%ld, before:%ld, duration:%ld, points:%zu, res:%ld - wanted => after:%ld, before:%ld, points:%zu, group:%zu, granularity:%ld, resgroup:%ld, resdiv:" NETDATA_DOUBLE_FORMAT_AUTO " %s", qt->id, after_requested, before_requested, before_requested - after_requested, points_requested, resampling_time_requested, after_wanted, before_wanted, points_wanted, group, query_granularity, resampling_group, resampling_divisor, buffer_tostring(debug_log)); \
533 + buffer_free(debug_log); \
534 + debug_log = NULL; \
535 }
536 +#define query_debug_log_free() do { buffer_free(debug_log); } while(0)
537 +#else
538 +#define query_debug_log_init() debug_dummy()
539 +#define query_debug_log(args...) debug_dummy()
540 +#define query_debug_log_fin() debug_dummy()
541 +#define query_debug_log_free() debug_dummy()
542 +#endif
543 +
544 +bool query_target_calculate_window(QUERY_TARGET *qt) {
545 + if (unlikely(!qt)) return false;
546
3028 - if(unlikely(i < 0))
3029 - return;
547 + size_t points_requested = (long)qt->request.points;
548 + time_t after_requested = qt->request.after;
549 + time_t before_requested = qt->request.before;
550 + RRDR_TIME_GROUPING group_method = qt->request.time_group_method;
551 + time_t resampling_time_requested = qt->request.resampling_time;
552 + RRDR_OPTIONS options = qt->window.options;
553 + size_t tier = qt->request.tier;
554 + time_t update_every = qt->db.minimum_latest_update_every_s ? qt->db.minimum_latest_update_every_s : 1;
555
3031 - // internal_error(true, "Found trimmable index %zd (from 0 to %zu)", i, r->n - 1);
556 + // RULES
557 + // points_requested = 0
558 + // the user wants all the natural points the database has
559 + //
560 + // after_requested = 0
561 + // the user wants to start the query from the oldest point in our database
562 + //
563 + // before_requested = 0
564 + // the user wants the query to end to the latest point in our database
565 + //
566 + // when natural points are wanted, the query has to be aligned to the update_every
567 + // of the database
568
3033 - size_t last_row_gbc = 0;
3034 - for (; i < (ssize_t)r->n; i++) {
3035 - size_t row_gbc = 0;
3036 - for (size_t d = 0; d < r->d; d++) {
3037 - if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
3038 - continue;
569 + size_t points_wanted = points_requested;
570 + time_t after_wanted = after_requested;
571 + time_t before_wanted = before_requested;
572
3040 - row_gbc += r->gbc[ i * r->d + d ];
3041 - }
573 + bool aligned = !(options & RRDR_OPTION_NOT_ALIGNED);
574 + bool automatic_natural_points = (points_wanted == 0);
575 + bool relative_period_requested = false;
576 + bool natural_points = (options & RRDR_OPTION_NATURAL_POINTS) || automatic_natural_points;
577 + bool before_is_aligned_to_db_end = false;
578
3043 - // internal_error(true, "GBC of index %zd is %zu", i, row_gbc);
579 + query_debug_log_init();
580
3045 - if (unlikely(r->t[i] >= trimmable_after && (row_gbc < last_row_gbc || !row_gbc))) {
3046 - // discard the rest of the points
3047 - // internal_error(true, "Discarding points %zd to %zu", i, r->n - 1);
3048 - r->partial_data_trimming.trimmed_after = r->t[i];
3049 - r->rows = i;
3050 - break;
3051 - }
3052 - else
3053 - last_row_gbc = row_gbc;
581 + if (ABS(before_requested) <= API_RELATIVE_TIME_MAX || ABS(after_requested) <= API_RELATIVE_TIME_MAX) {
582 + relative_period_requested = true;
583 + natural_points = true;
584 + options |= RRDR_OPTION_NATURAL_POINTS;
585 + query_debug_log(":relative+natural");
586 }
3055 -}
3056 -
3057 -static void rrdr2rrdr_group_by_calculate_percentage_of_group(RRDR *r) {
3058 - if(!r->vh)
3059 - return;
3060 -
3061 - if(query_target_aggregatable(r->internal.qt) && query_has_group_by_aggregation_percentage(r->internal.qt))
3062 - return;
3063 -
3064 - for(size_t i = 0; i < r->n ;i++) {
3065 - NETDATA_DOUBLE *cn = &r->v[ i * r->d ];
3066 - NETDATA_DOUBLE *ch = &r->vh[ i * r->d ];
587
3068 - for(size_t d = 0; d < r->d ;d++) {
3069 - NETDATA_DOUBLE n = cn[d];
3070 - NETDATA_DOUBLE h = ch[d];
3071 -
3072 - if(isnan(n))
3073 - cn[d] = 0.0;
588 + // if the user wants virtual points, make sure we do it
589 + if (options & RRDR_OPTION_VIRTUAL_POINTS)
590 + natural_points = false;
591
3075 - else if(isnan(h))
3076 - cn[d] = 100.0;
592 + // set the right flag about natural and virtual points
593 + if (natural_points) {
594 + options |= RRDR_OPTION_NATURAL_POINTS;
595
3078 - else
3079 - cn[d] = n * 100.0 / (n + h);
3080 - }
596 + if (options & RRDR_OPTION_VIRTUAL_POINTS)
597 + options &= ~RRDR_OPTION_VIRTUAL_POINTS;
598 }
3082 -}
599 + else {
600 + options |= RRDR_OPTION_VIRTUAL_POINTS;
601
3084 -static void rrd2rrdr_convert_values_to_percentage_of_total(RRDR *r) {
3085 - if(!(r->internal.qt->window.options & RRDR_OPTION_PERCENTAGE) || query_target_aggregatable(r->internal.qt))
3086 - return;
602 + if (options & RRDR_OPTION_NATURAL_POINTS)
603 + options &= ~RRDR_OPTION_NATURAL_POINTS;
604 + }
605
3088 - size_t global_min_max_values = 0;
3089 - NETDATA_DOUBLE global_min = NAN, global_max = NAN;
606 + if (after_wanted == 0 || before_wanted == 0) {
607 + relative_period_requested = true;
608
3091 - for(size_t i = 0; i != r->n ;i++) {
3092 - NETDATA_DOUBLE *cn = &r->v[ i * r->d ];
3093 - RRDR_VALUE_FLAGS *co = &r->o[ i * r->d ];
609 + time_t first_entry_s = qt->db.first_time_s;
610 + time_t last_entry_s = qt->db.last_time_s;
611
3095 - NETDATA_DOUBLE total = 0;
3096 - for (size_t d = 0; d < r->d; d++) {
3097 - if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
3098 - continue;
612 + if (first_entry_s == 0 || last_entry_s == 0) {
613 + internal_error(true, "QUERY: no data detected on query '%s' (db first_entry_t = %ld, last_entry_t = %ld)", qt->id, first_entry_s, last_entry_s);
614 + after_wanted = qt->window.after;
615 + before_wanted = qt->window.before;
616
3100 - if(co[d] & RRDR_VALUE_EMPTY)
3101 - continue;
617 + if(after_wanted == before_wanted)
618 + after_wanted = before_wanted - update_every;
619
3103 - total += cn[d];
620 + if (points_wanted == 0) {
621 + points_wanted = (before_wanted - after_wanted) / update_every;
622 + query_debug_log(":zero points_wanted %zu", points_wanted);
623 + }
624 }
625 + else {
626 + query_debug_log(":first_entry_t %ld, last_entry_t %ld", first_entry_s, last_entry_s);
627
3106 - if(total == 0.0)
3107 - total = 1.0;
3108 -
3109 - for (size_t d = 0; d < r->d; d++) {
3110 - if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
3111 - continue;
3112 -
3113 - if(co[d] & RRDR_VALUE_EMPTY)
3114 - continue;
628 + if (after_wanted == 0) {
629 + after_wanted = first_entry_s;
630 + query_debug_log(":zero after_wanted %ld", after_wanted);
631 + }
632
3116 - NETDATA_DOUBLE n = cn[d];
3117 - n = cn[d] = n * 100.0 / total;
633 + if (before_wanted == 0) {
634 + before_wanted = last_entry_s;
635 + before_is_aligned_to_db_end = true;
636 + query_debug_log(":zero before_wanted %ld", before_wanted);
637 + }
638
3119 - if(unlikely(!global_min_max_values++))
3120 - global_min = global_max = n;
3121 - else {
3122 - if(n < global_min)
3123 - global_min = n;
3124 - if(n > global_max)
3125 - global_max = n;
639 + if (points_wanted == 0) {
640 + points_wanted = (last_entry_s - first_entry_s) / update_every;
641 + query_debug_log(":zero points_wanted %zu", points_wanted);
642 }
643 }
644 }
645
3130 - r->view.min = global_min;
3131 - r->view.max = global_max;
3132 -
3133 - if(!r->dview)
3134 - // v1 query
3135 - return;
3136 -
3137 - // v2 query
3138 -
3139 - for (size_t d = 0; d < r->d; d++) {
3140 - if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
3141 - continue;
3142 -
3143 - size_t count = 0;
3144 - NETDATA_DOUBLE min = 0.0, max = 0.0, sum = 0.0, ars = 0.0;
3145 - for(size_t i = 0; i != r->rows ;i++) { // we use r->rows to respect trimming
3146 - size_t idx = i * r->d + d;
3147 -
3148 - RRDR_VALUE_FLAGS o = r->o[ idx ];
646 + if (points_wanted == 0) {
647 + points_wanted = 600;
648 + query_debug_log(":zero600 points_wanted %zu", points_wanted);
649 + }
650
3150 - if (o & RRDR_VALUE_EMPTY)
3151 - continue;
651 + // convert our before_wanted and after_wanted to absolute
652 + rrdr_relative_window_to_absolute_query(&after_wanted, &before_wanted, NULL, unittest_running);
653 + query_debug_log(":relative2absolute after %ld, before %ld", after_wanted, before_wanted);
654
3153 - NETDATA_DOUBLE ar = r->ar[ idx ];
3154 - ars += ar;
655 + if (natural_points && (options & RRDR_OPTION_SELECTED_TIER) && tier > 0 && nd_profile.storage_tiers > 1) {
656 + update_every = rrdset_find_natural_update_every_for_timeframe(
657 + qt, after_wanted, before_wanted, points_wanted, options, tier);
658
3156 - NETDATA_DOUBLE n = r->v[ idx ];
3157 - sum += n;
659 + if (update_every <= 0) update_every = qt->db.minimum_latest_update_every_s;
660 + query_debug_log(":natural update every %ld", update_every);
661 + }
662
3159 - if(!count++)
3160 - min = max = n;
3161 - else {
3162 - if(n < min)
3163 - min = n;
3164 - if(n > max)
3165 - max = n;
3166 - }
3167 - }
663 + // this is the update_every of the query
664 + // it may be different to the update_every of the database
665 + time_t query_granularity = (natural_points) ? update_every : 1;
666 + if (query_granularity <= 0) query_granularity = 1;
667 + query_debug_log(":query_granularity %ld", query_granularity);
668
3169 - r->dview[d] = (STORAGE_POINT) {
3170 - .sum = sum,
3171 - .count = count,
3172 - .min = min,
3173 - .max = max,
3174 - .anomaly_count = (size_t)(ars * (NETDATA_DOUBLE)count),
3175 - };
669 + // align before_wanted and after_wanted to query_granularity
670 + if (before_wanted % query_granularity) {
671 + before_wanted -= before_wanted % query_granularity;
672 + query_debug_log(":granularity align before_wanted %ld", before_wanted);
673 }
3177 -}
3178 -
3179 -static RRDR *rrd2rrdr_group_by_finalize(RRDR *r_tmp) {
3180 - QUERY_TARGET *qt = r_tmp->internal.qt;
674
3182 - if(!r_tmp->group_by.r) {
3183 - // v1 query
3184 - rrd2rrdr_convert_values_to_percentage_of_total(r_tmp);
3185 - return r_tmp;
675 + if (after_wanted % query_granularity) {
676 + after_wanted -= after_wanted % query_granularity;
677 + query_debug_log(":granularity align after_wanted %ld", after_wanted);
678 }
3187 - // v2 query
3188 -
3189 - // do the additional passes on RRDRs
3190 - RRDR *last_r = r_tmp->group_by.r;
3191 - rrdr2rrdr_group_by_calculate_percentage_of_group(last_r);
3192 -
3193 - RRDR *r = last_r->group_by.r;
3194 - size_t pass = 0;
3195 - while(r) {
3196 - pass++;
3197 - for(size_t d = 0; d < last_r->d ;d++) {
3198 - rrd2rrdr_group_by_add_metric(r, last_r->dgbs[d], last_r, d,
3199 - qt->request.group_by[pass].aggregation,
3200 - &last_r->dqp[d], pass);
3201 - }
3202 - rrdr2rrdr_group_by_calculate_percentage_of_group(r);
679
3204 - last_r = r;
3205 - r = last_r->group_by.r;
680 + // automatic_natural_points is set when the user wants all the points available in the database
681 + if (automatic_natural_points) {
682 + points_wanted = (before_wanted - after_wanted + 1) / query_granularity;
683 + if (unlikely(points_wanted <= 0)) points_wanted = 1;
684 + query_debug_log(":auto natural points_wanted %zu", points_wanted);
685 }
686
3208 - // free all RRDRs except the last one
3209 - r = r_tmp;
3210 - while(r != last_r) {
3211 - r_tmp = r->group_by.r;
3212 - r->group_by.r = NULL;
3213 - rrdr_free(r->internal.owa, r);
3214 - r = r_tmp;
687 + time_t duration = before_wanted - after_wanted;
688 +
689 + // if the resampling time is too big, extend the duration to the past
690 + if (unlikely(resampling_time_requested > duration)) {
691 + after_wanted = before_wanted - resampling_time_requested;
692 + duration = before_wanted - after_wanted;
693 + query_debug_log(":resampling after_wanted %ld", after_wanted);
694 }
3216 - r = last_r;
3217 -
3218 - // find the final aggregation
3219 - RRDR_GROUP_BY_FUNCTION aggregation = qt->request.group_by[0].aggregation;
3220 - for(size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES ;g++)
3221 - if(qt->request.group_by[g].group_by != RRDR_GROUP_BY_NONE)
3222 - aggregation = qt->request.group_by[g].aggregation;
3223 -
3224 - if(!query_target_aggregatable(qt) && r->partial_data_trimming.expected_after < qt->window.before)
3225 - rrdr2rrdr_group_by_partial_trimming(r);
3226 -
3227 - // apply averaging, remove RRDR_VALUE_EMPTY, find the non-zero dimensions, min and max
3228 - size_t global_min_max_values = 0;
3229 - size_t dimensions_nonzero = 0;
3230 - NETDATA_DOUBLE global_min = NAN, global_max = NAN;
3231 - for (size_t d = 0; d < r->d; d++) {
3232 - if (unlikely(!(r->od[d] & RRDR_DIMENSION_QUERIED)))
3233 - continue;
695
3235 - size_t points_nonzero = 0;
3236 - NETDATA_DOUBLE min = 0, max = 0, sum = 0, ars = 0;
3237 - size_t count = 0;
696 + // if the duration is not aligned to resampling time
697 + // extend the duration to the past, to avoid a gap at the chart
698 + // only when the missing duration is above 1/10th of a point
699 + if (resampling_time_requested > query_granularity && duration % resampling_time_requested) {
700 + time_t delta = duration % resampling_time_requested;
701 + if (delta > resampling_time_requested / 10) {
702 + after_wanted -= resampling_time_requested - delta;
703 + duration = before_wanted - after_wanted;
704 + query_debug_log(":resampling2 after_wanted %ld", after_wanted);
705 + }
706 + }
707
3239 - for(size_t i = 0; i != r->n ;i++) {
3240 - size_t idx = i * r->d + d;
708 + // the available points of the query
709 + size_t points_available = (duration + 1) / query_granularity;
710 + if (unlikely(points_available <= 0)) points_available = 1;
711 + query_debug_log(":points_available %zu", points_available);
712
3242 - NETDATA_DOUBLE *cn = &r->v[ idx ];
3243 - RRDR_VALUE_FLAGS *co = &r->o[ idx ];
3244 - NETDATA_DOUBLE *ar = &r->ar[ idx ];
3245 - uint32_t gbc = r->gbc[ idx ];
713 + if (points_wanted > points_available) {
714 + points_wanted = points_available;
715 + query_debug_log(":max points_wanted %zu", points_wanted);
716 + }
717
3247 - if(likely(gbc)) {
3248 - *co &= ~RRDR_VALUE_EMPTY;
718 + if(points_wanted > 86400 && !unittest_running) {
719 + points_wanted = 86400;
720 + query_debug_log(":absolute max points_wanted %zu", points_wanted);
721 + }
722
3250 - if(gbc != r->dgbc[d])
3251 - *co |= RRDR_VALUE_PARTIAL;
723 + // calculate the desired grouping of source data points
724 + size_t group = points_available / points_wanted;
725 + if (group == 0) group = 1;
726
3253 - NETDATA_DOUBLE n;
727 + // round "group" to the closest integer
728 + if (points_available % points_wanted > points_wanted / 2)
729 + group++;
730
3255 - sum += *cn;
3256 - ars += *ar;
731 + query_debug_log(":group %zu", group);
732
3258 - if(aggregation == RRDR_GROUP_BY_FUNCTION_AVERAGE && !query_target_aggregatable(qt))
3259 - n = (*cn /= gbc);
3260 - else
3261 - n = *cn;
733 + if (points_wanted * group * query_granularity < (size_t)duration) {
734 + // the grouping we are going to do, is not enough
735 + // to cover the entire duration requested, so
736 + // we have to change the number of points, to make sure we will
737 + // respect the timeframe as closely as possibly
738
3263 - if(!query_target_aggregatable(qt))
3264 - *ar /= gbc;
739 + // let's see how many points are the optimal
740 + points_wanted = points_available / group;
741
3266 - if(islessgreater(n, 0.0))
3267 - points_nonzero++;
742 + if (points_wanted * group < points_available)
743 + points_wanted++;
744
3269 - if(unlikely(!count))
3270 - min = max = n;
3271 - else {
3272 - if(n < min)
3273 - min = n;
745 + if (unlikely(points_wanted == 0))
746 + points_wanted = 1;
747
3275 - if(n > max)
3276 - max = n;
3277 - }
748 + query_debug_log(":optimal points %zu", points_wanted);
749 + }
750
3279 - if(unlikely(!global_min_max_values++))
3280 - global_min = global_max = n;
3281 - else {
3282 - if(n < global_min)
3283 - global_min = n;
751 + // resampling_time_requested enforces a certain grouping multiple
752 + NETDATA_DOUBLE resampling_divisor = 1.0;
753 + size_t resampling_group = 1;
754 + if (unlikely(resampling_time_requested > query_granularity)) {
755 + // the points we should group to satisfy gtime
756 + resampling_group = resampling_time_requested / query_granularity;
757 + if (unlikely(resampling_time_requested % query_granularity))
758 + resampling_group++;
759
3285 - if(n > global_max)
3286 - global_max = n;
3287 - }
760 + query_debug_log(":resampling group %zu", resampling_group);
761
3289 - count += gbc;
3290 - }
762 + // adapt group according to resampling_group
763 + if (unlikely(group < resampling_group)) {
764 + group = resampling_group; // do not allow grouping below the desired one
765 + query_debug_log(":group less res %zu", group);
766 }
3292 -
3293 - if(points_nonzero) {
3294 - r->od[d] |= RRDR_DIMENSION_NONZERO;
3295 - dimensions_nonzero++;
767 + if (unlikely(group % resampling_group)) {
768 + group += resampling_group - (group % resampling_group); // make sure group is multiple of resampling_group
769 + query_debug_log(":group mod res %zu", group);
770 }
771
3298 - r->dview[d] = (STORAGE_POINT) {
3299 - .sum = sum,
3300 - .count = count,
3301 - .min = min,
3302 - .max = max,
3303 - .anomaly_count = (size_t)(ars * RRDR_DVIEW_ANOMALY_COUNT_MULTIPLIER / 100.0),
3304 - };
772 + // resampling_divisor = group / resampling_group;
773 + resampling_divisor = (NETDATA_DOUBLE) (group * query_granularity) / (NETDATA_DOUBLE) resampling_time_requested;
774 + query_debug_log(":resampling divisor " NETDATA_DOUBLE_FORMAT, resampling_divisor);
775 }
776
3307 - r->view.min = global_min;
3308 - r->view.max = global_max;
3309 -
3310 - if(!dimensions_nonzero && (qt->window.options & RRDR_OPTION_NONZERO)) {
3311 - // all dimensions are zero
3312 - // remove the nonzero option
3313 - qt->window.options &= ~RRDR_OPTION_NONZERO;
777 + // now that we have group, align the requested timeframe to fit it.
778 + if (aligned && before_wanted % (group * query_granularity)) {
779 + if (before_is_aligned_to_db_end)
780 + before_wanted -= before_wanted % (time_t)(group * query_granularity);
781 + else
782 + before_wanted += (time_t)(group * query_granularity) - before_wanted % (time_t)(group * query_granularity);
783 + query_debug_log(":align before_wanted %ld", before_wanted);
784 }
785
3316 - rrd2rrdr_convert_values_to_percentage_of_total(r);
786 + after_wanted = before_wanted - (time_t)(points_wanted * group * query_granularity) + query_granularity;
787 + query_debug_log(":final after_wanted %ld", after_wanted);
788 +
789 + duration = before_wanted - after_wanted;
790 + query_debug_log(":final duration %ld", duration + 1);
791
3318 - // update query instance counts in query host and query context
3319 - {
3320 - size_t h = 0, c = 0, i = 0;
3321 - for(; h < qt->nodes.used ; h++) {
3322 - QUERY_NODE *qn = &qt->nodes.array[h];
792 + query_debug_log_fin();
793
3324 - for(; c < qt->contexts.used ;c++) {
3325 - QUERY_CONTEXT *qc = &qt->contexts.array[c];
794 + internal_error(points_wanted != duration / (query_granularity * group) + 1,
795 + "QUERY: points_wanted %zu is not points %zu",
796 + points_wanted, (size_t)(duration / (query_granularity * group) + 1));
797
3327 - if(!rrdcontext_acquired_belongs_to_host(qc->rca, qn->rrdhost))
3328 - break;
798 + internal_error(group < resampling_group,
799 + "QUERY: group %zu is less than the desired group points %zu",
800 + group, resampling_group);
801
3330 - for(; i < qt->instances.used ;i++) {
3331 - QUERY_INSTANCE *qi = &qt->instances.array[i];
802 + internal_error(group > resampling_group && group % resampling_group,
803 + "QUERY: group %zu is not a multiple of the desired group points %zu",
804 + group, resampling_group);
805
3333 - if(!rrdinstance_acquired_belongs_to_context(qi->ria, qc->rca))
3334 - break;
806 + // -------------------------------------------------------------------------
807 + // update QUERY_TARGET with our calculations
808
3336 - if(qi->metrics.queried) {
3337 - qc->instances.queried++;
3338 - qn->instances.queried++;
3339 - }
3340 - else if(qi->metrics.failed) {
3341 - qc->instances.failed++;
3342 - qn->instances.failed++;
3343 - }
3344 - }
3345 - }
3346 - }
3347 - }
809 + qt->window.after = after_wanted;
810 + qt->window.before = before_wanted;
811 + qt->window.relative = relative_period_requested;
812 + qt->window.points = points_wanted;
813 + qt->window.group = group;
814 + qt->window.time_group_method = group_method;
815 + qt->window.time_group_options = qt->request.time_group_options;
816 + qt->window.query_granularity = query_granularity;
817 + qt->window.resampling_group = resampling_group;
818 + qt->window.resampling_divisor = resampling_divisor;
819 + qt->window.options = options;
820 + qt->window.tier = tier;
821 + qt->window.aligned = aligned;
822
3349 - return r;
823 + return true;
824 }
825
826 // ----------------------------------------------------------------------------
@@ -3428,7 +902,7 @@ RRDR *rrd2rrdr(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
902 size_t last_db_points_read = 0;
903 size_t last_result_points_generated = 0;
904
3431 - internal_fatal(released_ops, "QUERY: released_ops should be NULL when the query starts");
905 + // internal_fatal(released_ops, "QUERY: released_ops should be NULL when the query starts");
906
907 query_progress_set_finish_line(qt->request.transaction, qt->query.used);
908
@@ -3612,6 +1086,9 @@ RRDR *rrd2rrdr(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
1086
1087 // get the final RRDR to send to the caller
1088 r = rrd2rrdr_group_by_finalize(r_tmp);
1089 +
1090 + // apply cardinality limit if requested
1091 + r = rrd2rrdr_cardinality_limit(r);
1092
1093 #ifdef NETDATA_INTERNAL_CHECKS
1094 if (dimensions_used && !(r->view.flags & RRDR_RESULT_FLAG_CANCEL)) {
@@ -3665,7 +1142,7 @@ RRDR *rrd2rrdr(ONEWAYALLOC *owa, QUERY_TARGET *qt) {
1142 ops[d] = NULL;
1143 }
1144 rrd2rrdr_query_ops_freeall(r);
3668 - internal_fatal(released_ops, "QUERY: released_ops should be NULL when the query ends");
1145 + // internal_fatal(released_ops, "QUERY: released_ops should be NULL when the query ends");
1146
1147 onewayalloc_freez(owa, ops);
1148
src/web/api/queries/query.h
+3 -1
@@ -47,6 +47,7 @@ typedef enum rrdr_time_grouping {
47 RRDR_GROUPING_SES,
48 RRDR_GROUPING_DES,
49 RRDR_GROUPING_COUNTIF,
50 + RRDR_GROUPING_EXTREMES,
51 } RRDR_TIME_GROUPING;
52
53 const char *time_grouping_id2txt(RRDR_TIME_GROUPING group);
@@ -81,7 +82,7 @@ typedef enum rrdr_group_by {
82
83 struct web_buffer;
84
84 -RRDR_GROUP_BY group_by_parse(char *s);
85 +RRDR_GROUP_BY group_by_parse(const char *group_by_txt);
86 void buffer_json_group_by_to_array(struct web_buffer *wb, RRDR_GROUP_BY group_by);
87
88 typedef enum rrdr_group_by_function {
@@ -90,6 +91,7 @@ typedef enum rrdr_group_by_function {
91 RRDR_GROUP_BY_FUNCTION_MAX,
92 RRDR_GROUP_BY_FUNCTION_SUM,
93 RRDR_GROUP_BY_FUNCTION_PERCENTAGE,
94 + RRDR_GROUP_BY_FUNCTION_EXTREMES,
95 } RRDR_GROUP_BY_FUNCTION;
96
97 RRDR_GROUP_BY_FUNCTION group_by_aggregate_function_parse(const char *s);
src/web/api/queries/weights.c
+261 -18
@@ -125,14 +125,14 @@ static void results_header_to_json(DICTIONARY *results __maybe_unused, BUFFER *w
125 size_t examined_dimensions __maybe_unused, usec_t duration,
126 WEIGHTS_STATS *stats) {
127
128 - buffer_json_member_add_time_t(wb, "after", after);
129 - buffer_json_member_add_time_t(wb, "before", before);
128 + buffer_json_member_add_time_t_formatted(wb, "after", after, options & RRDR_OPTION_RFC3339);
129 + buffer_json_member_add_time_t_formatted(wb, "before", before, options & RRDR_OPTION_RFC3339);
130 buffer_json_member_add_time_t(wb, "duration", before - after);
131 buffer_json_member_add_uint64(wb, "points", points);
132
133 if(method == WEIGHTS_METHOD_MC_KS2 || method == WEIGHTS_METHOD_MC_VOLUME) {
134 - buffer_json_member_add_time_t(wb, "baseline_after", baseline_after);
135 - buffer_json_member_add_time_t(wb, "baseline_before", baseline_before);
134 + buffer_json_member_add_time_t_formatted(wb, "baseline_after", baseline_after, options & RRDR_OPTION_RFC3339);
135 + buffer_json_member_add_time_t_formatted(wb, "baseline_before", baseline_before, options & RRDR_OPTION_RFC3339);
136 buffer_json_member_add_time_t(wb, "baseline_duration", baseline_before - baseline_after);
137 buffer_json_member_add_uint64(wb, "baseline_points", points << shifts);
138 }
@@ -306,12 +306,18 @@ struct query_weights_data {
306
307 SIMPLE_PATTERN *scope_nodes_sp;
308 SIMPLE_PATTERN *scope_contexts_sp;
309 + SIMPLE_PATTERN *scope_instances_sp;
310 + SIMPLE_PATTERN *scope_labels_sp;
311 + SIMPLE_PATTERN *scope_dimensions_sp;
312 SIMPLE_PATTERN *nodes_sp;
313 SIMPLE_PATTERN *contexts_sp;
314 SIMPLE_PATTERN *instances_sp;
315 SIMPLE_PATTERN *dimensions_sp;
316 SIMPLE_PATTERN *labels_sp;
317 SIMPLE_PATTERN *alerts_sp;
318 +
319 + struct pattern_array *scope_labels_pa;
320 + struct pattern_array *labels_pa;
321
322 usec_t timeout_us;
323 bool timed_out;
@@ -375,6 +381,8 @@ static void results_header_to_json_v2(DICTIONARY *results __maybe_unused, BUFFER
381 buffer_json_member_add_object(wb, "scope");
382 buffer_json_member_add_string(wb, "scope_nodes", qwd->qwr->scope_nodes ? qwd->qwr->scope_nodes : "*");
383 buffer_json_member_add_string(wb, "scope_contexts", qwd->qwr->scope_contexts ? qwd->qwr->scope_contexts : "*");
384 + buffer_json_member_add_string(wb, "scope_instances", qwd->qwr->scope_instances ? qwd->qwr->scope_instances : "*");
385 + buffer_json_member_add_string(wb, "scope_labels", qwd->qwr->scope_labels ? qwd->qwr->scope_labels : "*");
386 buffer_json_object_close(wb);
387
388 buffer_json_member_add_object(wb, "selectors");
@@ -387,8 +395,8 @@ static void results_header_to_json_v2(DICTIONARY *results __maybe_unused, BUFFER
395 buffer_json_object_close(wb);
396
397 buffer_json_member_add_object(wb, "window");
390 - buffer_json_member_add_time_t(wb, "after", qwd->qwr->after);
391 - buffer_json_member_add_time_t(wb, "before", qwd->qwr->before);
398 + buffer_json_member_add_time_t_formatted(wb, "after", qwd->qwr->after, options & RRDR_OPTION_RFC3339);
399 + buffer_json_member_add_time_t_formatted(wb, "before", qwd->qwr->before, options & RRDR_OPTION_RFC3339);
400 buffer_json_member_add_uint64(wb, "points", qwd->qwr->points);
401 if(qwd->qwr->options & RRDR_OPTION_SELECTED_TIER)
402 buffer_json_member_add_uint64(wb, "tier", qwd->qwr->tier);
@@ -398,8 +406,8 @@ static void results_header_to_json_v2(DICTIONARY *results __maybe_unused, BUFFER
406
407 if(method == WEIGHTS_METHOD_MC_KS2 || method == WEIGHTS_METHOD_MC_VOLUME) {
408 buffer_json_member_add_object(wb, "baseline");
401 - buffer_json_member_add_time_t(wb, "baseline_after", qwd->qwr->baseline_after);
402 - buffer_json_member_add_time_t(wb, "baseline_before", qwd->qwr->baseline_before);
409 + buffer_json_member_add_time_t_formatted(wb, "baseline_after", qwd->qwr->baseline_after, options & RRDR_OPTION_RFC3339);
410 + buffer_json_member_add_time_t_formatted(wb, "baseline_before", qwd->qwr->baseline_before, options & RRDR_OPTION_RFC3339);
411 buffer_json_object_close(wb);
412 }
413
@@ -433,16 +441,16 @@ static void results_header_to_json_v2(DICTIONARY *results __maybe_unused, BUFFER
441 buffer_json_member_add_string(wb, "time_group", time_grouping_tostring(group));
442
443 buffer_json_member_add_object(wb, "window");
436 - buffer_json_member_add_time_t(wb, "after", after);
437 - buffer_json_member_add_time_t(wb, "before", before);
444 + buffer_json_member_add_time_t_formatted(wb, "after", after, options & RRDR_OPTION_RFC3339);
445 + buffer_json_member_add_time_t_formatted(wb, "before", before, options & RRDR_OPTION_RFC3339);
446 buffer_json_member_add_time_t(wb, "duration", before - after);
447 buffer_json_member_add_uint64(wb, "points", points);
448 buffer_json_object_close(wb);
449
450 if(method == WEIGHTS_METHOD_MC_KS2 || method == WEIGHTS_METHOD_MC_VOLUME) {
451 buffer_json_member_add_object(wb, "baseline");
444 - buffer_json_member_add_time_t(wb, "after", baseline_after);
445 - buffer_json_member_add_time_t(wb, "before", baseline_before);
452 + buffer_json_member_add_time_t_formatted(wb, "after", baseline_after, options & RRDR_OPTION_RFC3339);
453 + buffer_json_member_add_time_t_formatted(wb, "before", baseline_before, options & RRDR_OPTION_RFC3339);
454 buffer_json_member_add_time_t(wb, "duration", baseline_before - baseline_after);
455 buffer_json_member_add_uint64(wb, "points", points << shifts);
456 buffer_json_object_close(wb);
@@ -934,7 +942,7 @@ static size_t registered_results_to_json_multinode_no_group_by(
942
943 buffer_json_object_close(wb); //dictionaries
944
937 - buffer_json_agents_v2(wb, &qwd->timings, 0, false, true);
945 + buffer_json_agents_v2(wb, &qwd->timings, 0, false, true, rrdr_options_to_contexts_options(options));
946 buffer_json_member_add_uint64(wb, "correlated_dimensions", total_dimensions);
947 buffer_json_member_add_uint64(wb, "total_dimensions_count", examined_dimensions);
948 buffer_json_finalize(wb);
@@ -1071,7 +1079,7 @@ static size_t registered_results_to_json_multinode_group_by(
1079 dfe_done(aw);
1080 buffer_json_array_close(wb); // result
1081
1074 - buffer_json_agents_v2(wb, &qwd->timings, 0, false, true);
1082 + buffer_json_agents_v2(wb, &qwd->timings, 0, false, true, rrdr_options_to_contexts_options(options));
1083 buffer_json_member_add_uint64(wb, "correlated_dimensions", total_dimensions);
1084 buffer_json_member_add_uint64(wb, "total_dimensions_count", examined_dimensions);
1085 buffer_json_finalize(wb);
@@ -1516,6 +1524,9 @@ static void rrdset_weights_multi_dimensional_value(struct query_weights_data *qw
1524 .version = 1,
1525 .scope_nodes = qwd->qwr->scope_nodes,
1526 .scope_contexts = qwd->qwr->scope_contexts,
1527 + .scope_instances = qwd->qwr->scope_instances,
1528 + .scope_labels = qwd->qwr->scope_labels,
1529 + .scope_dimensions = qwd->qwr->scope_dimensions,
1530 .nodes = qwd->qwr->nodes,
1531 .contexts = qwd->qwr->contexts,
1532 .instances = qwd->qwr->instances,
@@ -1552,6 +1563,8 @@ static void rrdset_weights_multi_dimensional_value(struct query_weights_data *qw
1563
1564 size_t queries = 0;
1565 for(size_t d = 0; d < r->d ;d++) {
1566 + qwd->examined_dimensions++;
1567 +
1568 if(!rrdr_dimension_should_be_exposed(r->od[d], qwd->qwr->options))
1569 continue;
1570
@@ -1570,8 +1583,9 @@ static void rrdset_weights_multi_dimensional_value(struct query_weights_data *qw
1583 QUERY_CONTEXT *qc = query_context(r->internal.qt, qm->link.query_context_id);
1584 QUERY_NODE *qn = query_node(r->internal.qt, qm->link.query_node_id);
1585
1573 - register_result(qwd->results, qn->rrdhost, qc->rca, qi->ria, qd->rma, qv.value, 0, &qv.sp,
1574 - NULL, &qwd->stats, qwd->register_zero, qm->duration_ut);
1586 + register_result(qwd->results, qn->rrdhost, qc->rca, qi->ria, qd->rma, qv.value, 0,
1587 + &r->internal.qt->query.array[d].query_points, NULL,
1588 + &qwd->stats, qwd->register_zero, qm->duration_ut);
1589 }
1590
1591 queries++;
@@ -1669,6 +1683,198 @@ static size_t spread_results_evenly(DICTIONARY *results, WEIGHTS_STATS *stats) {
1683 return dimensions;
1684 }
1685
1686 +// ----------------------------------------------------------------------------
1687 +// MCP format output
1688 +
1689 +// Comparator for sorting results by value (descending order - highest scores first)
1690 +static int registered_results_value_compare(const DICTIONARY_ITEM **item1, const DICTIONARY_ITEM **item2) {
1691 + struct register_result *r1 = dictionary_acquired_item_value(*item1);
1692 + struct register_result *r2 = dictionary_acquired_item_value(*item2);
1693 +
1694 + // Sort by value in descending order (highest first)
1695 + if (r1->value < r2->value) return 1;
1696 + if (r1->value > r2->value) return -1;
1697 + return 0;
1698 +}
1699 +
1700 +// Callback for sorted dictionary walkthrough
1701 +struct mcp_output_state {
1702 + BUFFER *wb;
1703 + WEIGHTS_METHOD method;
1704 + size_t count;
1705 + size_t limit;
1706 +};
1707 +
1708 +static int registered_results_to_json_mcp_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data) {
1709 + struct mcp_output_state *state = (struct mcp_output_state *)data;
1710 + struct register_result *t = (struct register_result *)value;
1711 +
1712 + // Check if we've reached the cardinality limit
1713 + if (state->count >= state->limit)
1714 + return -1; // Stop iteration
1715 +
1716 + BUFFER *wb = state->wb;
1717 +
1718 + buffer_json_add_array_item_array(wb); // Start row array
1719 +
1720 + // Add score/value based on method
1721 + switch(state->method) {
1722 + case WEIGHTS_METHOD_MC_KS2:
1723 + case WEIGHTS_METHOD_MC_VOLUME:
1724 + buffer_json_add_array_item_double(wb, t->value);
1725 + break;
1726 +
1727 + case WEIGHTS_METHOD_ANOMALY_RATE:
1728 + // For anomaly rate, the value is already a percentage
1729 + buffer_json_add_array_item_double(wb, t->value);
1730 + break;
1731 +
1732 + case WEIGHTS_METHOD_VALUE:
1733 + // For CV or other aggregations
1734 + buffer_json_add_array_item_double(wb, t->value);
1735 + break;
1736 + }
1737 +
1738 + // Add the 5 statistical values
1739 + // 1. Min
1740 + if(storage_point_is_unset(t->highlighted) || storage_point_is_gap(t->highlighted))
1741 + buffer_json_add_array_item_double(wb, NAN);
1742 + else
1743 + buffer_json_add_array_item_double(wb, t->highlighted.min);
1744 +
1745 + // 2. Max
1746 + if(storage_point_is_unset(t->highlighted) || storage_point_is_gap(t->highlighted))
1747 + buffer_json_add_array_item_double(wb, NAN);
1748 + else
1749 + buffer_json_add_array_item_double(wb, t->highlighted.max);
1750 +
1751 + // 3. Average
1752 + if(storage_point_is_unset(t->highlighted) || storage_point_is_gap(t->highlighted) || t->highlighted.count == 0)
1753 + buffer_json_add_array_item_double(wb, NAN);
1754 + else
1755 + buffer_json_add_array_item_double(wb, t->highlighted.sum / (NETDATA_DOUBLE)t->highlighted.count);
1756 +
1757 + // 4. Number of samples in window
1758 + buffer_json_add_array_item_uint64(wb, t->highlighted.count);
1759 +
1760 + // 5. Number of anomalous samples in window
1761 + buffer_json_add_array_item_double(wb, t->highlighted.anomaly_count);
1762 +
1763 + // Add metadata
1764 + // Add node name
1765 + buffer_json_add_array_item_string(wb, rrdhost_hostname(t->host));
1766 +
1767 + // Add context
1768 + buffer_json_add_array_item_string(wb, rrdcontext_acquired_id(t->rca));
1769 +
1770 + // Add instance
1771 + buffer_json_add_array_item_string(wb, rrdinstance_acquired_id(t->ria));
1772 +
1773 + // Add dimension
1774 + buffer_json_add_array_item_string(wb, rrdmetric_acquired_name(t->rma));
1775 +
1776 + // Add labels (as object or null)
1777 + RRDLABELS *labels = rrdinstance_acquired_labels(t->ria);
1778 + if(labels && rrdlabels_entries(labels) > 0) {
1779 + buffer_json_add_array_item_object(wb);
1780 + rrdlabels_to_buffer_json_members(labels, wb);
1781 + buffer_json_object_close(wb);
1782 + }
1783 + else {
1784 + buffer_json_add_array_item_string(wb, NULL);
1785 + }
1786 +
1787 + buffer_json_array_close(wb); // End row array
1788 +
1789 + state->count++;
1790 + return 0; // Continue iteration
1791 +}
1792 +
1793 +static size_t registered_results_to_json_mcp(
1794 + DICTIONARY *results, BUFFER *wb,
1795 + time_t after __maybe_unused, time_t before __maybe_unused,
1796 + time_t baseline_after __maybe_unused, time_t baseline_before __maybe_unused,
1797 + size_t points __maybe_unused, WEIGHTS_METHOD method,
1798 + RRDR_TIME_GROUPING group __maybe_unused, RRDR_OPTIONS options, uint32_t shifts __maybe_unused,
1799 + size_t examined_dimensions __maybe_unused, struct query_weights_data *qwd,
1800 + WEIGHTS_STATS *stats __maybe_unused,
1801 + struct query_versions *versions __maybe_unused) {
1802 +
1803 + buffer_json_initialize(wb, "\"", "\"", 0, true, (options & RRDR_OPTION_MINIFY) ? BUFFER_JSON_OPTIONS_MINIFY : BUFFER_JSON_OPTIONS_DEFAULT);
1804 +
1805 + // Add columns array based on method
1806 + buffer_json_member_add_array(wb, "columns");
1807 +
1808 + switch(method) {
1809 + case WEIGHTS_METHOD_MC_KS2:
1810 + buffer_json_add_array_item_string(wb, "KS2 Score");
1811 + break;
1812 +
1813 + case WEIGHTS_METHOD_MC_VOLUME:
1814 + buffer_json_add_array_item_string(wb, "Volume Score");
1815 + break;
1816 +
1817 + case WEIGHTS_METHOD_ANOMALY_RATE:
1818 + buffer_json_add_array_item_string(wb, "Anomaly Rate");
1819 + break;
1820 +
1821 + case WEIGHTS_METHOD_VALUE:
1822 + buffer_json_add_array_item_string(wb, "Coefficient of Variation");
1823 + break;
1824 + }
1825 +
1826 + // Common statistical columns for all methods
1827 + buffer_json_add_array_item_string(wb, "Minimum Sample Value");
1828 + buffer_json_add_array_item_string(wb, "Maximum Sample Value");
1829 + buffer_json_add_array_item_string(wb, "Average Sample Value");
1830 + buffer_json_add_array_item_string(wb, "# of Samples in Window");
1831 + buffer_json_add_array_item_string(wb, "# of Anomalous Samples in Window");
1832 +
1833 + // Metadata columns
1834 + buffer_json_add_array_item_string(wb, "Hostname");
1835 + buffer_json_add_array_item_string(wb, "Context / Metric Name");
1836 + buffer_json_add_array_item_string(wb, "Metrics Instance");
1837 + buffer_json_add_array_item_string(wb, "Dimension");
1838 + buffer_json_add_array_item_string(wb, "Instance Labels");
1839 +
1840 + buffer_json_array_close(wb); // columns
1841 +
1842 + // Add results array
1843 + buffer_json_member_add_array(wb, "results");
1844 +
1845 + // Get cardinality limit from query weights data
1846 + size_t cardinality_limit = qwd && qwd->qwr ? qwd->qwr->cardinality_limit : 50;
1847 + if (cardinality_limit < 30) cardinality_limit = 30;
1848 +
1849 + // Set up state for callback
1850 + struct mcp_output_state state = {
1851 + .wb = wb,
1852 + .method = method,
1853 + .count = 0,
1854 + .limit = cardinality_limit
1855 + };
1856 +
1857 + // Walk through dictionary in sorted order (by value descending)
1858 + dictionary_sorted_walkthrough_rw(results, 'r', registered_results_to_json_mcp_callback, &state, registered_results_value_compare);
1859 +
1860 + buffer_json_array_close(wb); // results
1861 +
1862 + // Add metadata
1863 + buffer_json_member_add_object(wb, "metadata");
1864 + buffer_json_member_add_uint64(wb, "total_time_series_analyzed", examined_dimensions);
1865 + buffer_json_member_add_uint64(wb, "total_time_series_returned", state.count);
1866 + buffer_json_member_add_string(wb, "method", weights_method_to_string(method));
1867 + if (state.count >= cardinality_limit) {
1868 + buffer_json_member_add_uint64(wb, "cardinality_limit", cardinality_limit);
1869 + buffer_json_member_add_boolean(wb, "truncated", true);
1870 + }
1871 + buffer_json_object_close(wb); // metadata
1872 +
1873 + buffer_json_finalize(wb);
1874 +
1875 + return state.count;
1876 +}
1877 +
1878 // ----------------------------------------------------------------------------
1879 // The main function
1880
@@ -1765,9 +1971,12 @@ static ssize_t weights_do_context_callback(void *data, RRDCONTEXT_ACQUIRED *rca,
1971 return 0;
1972
1973 ssize_t ret = weights_foreach_rrdmetric_in_context(rca,
1974 + qwd->scope_instances_sp,
1975 + qwd->scope_labels_pa,
1976 + qwd->scope_dimensions_sp,
1977 qwd->instances_sp,
1978 NULL,
1770 - qwd->labels_sp,
1979 + qwd->labels_pa,
1980 qwd->alerts_sp,
1981 qwd->dimensions_sp,
1982 true, true, qwd->qwr->version,
@@ -1808,12 +2017,17 @@ int web_api_v12_weights(BUFFER *wb, QUERY_WEIGHTS_REQUEST *qwr) {
2017
2018 .scope_nodes_sp = string_to_simple_pattern(qwr->scope_nodes),
2019 .scope_contexts_sp = string_to_simple_pattern(qwr->scope_contexts),
2020 + .scope_instances_sp = string_to_simple_pattern(qwr->scope_instances),
2021 + .scope_labels_sp = string_to_simple_pattern(qwr->scope_labels),
2022 + .scope_dimensions_sp = string_to_simple_pattern(qwr->scope_dimensions),
2023 .nodes_sp = string_to_simple_pattern(qwr->nodes),
2024 .contexts_sp = string_to_simple_pattern(qwr->contexts),
2025 .instances_sp = string_to_simple_pattern(qwr->instances),
2026 .dimensions_sp = string_to_simple_pattern(qwr->dimensions),
2027 .labels_sp = string_to_simple_pattern(qwr->labels),
2028 .alerts_sp = string_to_simple_pattern(qwr->alerts),
2029 + .scope_labels_pa = NULL,
2030 + .labels_pa = NULL,
2031 .timeout_us = qwr->timeout_ms * USEC_PER_MS,
2032 .timed_out = false,
2033 .examined_dimensions = 0,
@@ -1825,6 +2039,12 @@ int web_api_v12_weights(BUFFER *wb, QUERY_WEIGHTS_REQUEST *qwr) {
2039 .received_ut = now_monotonic_usec(),
2040 }
2041 };
2042 +
2043 + // Pre-compile pattern arrays for labels
2044 + if(qwd.scope_labels_sp)
2045 + qwd.scope_labels_pa = pattern_array_add_simple_pattern(NULL, qwd.scope_labels_sp, ':');
2046 + if(qwd.labels_sp)
2047 + qwd.labels_pa = pattern_array_add_simple_pattern(NULL, qwd.labels_sp, ':');
2048
2049 if(!rrdr_relative_window_to_absolute_query(&qwr->after, &qwr->before, NULL, false))
2050 buffer_no_cacheable(wb);
@@ -1910,6 +2130,10 @@ int web_api_v12_weights(BUFFER *wb, QUERY_WEIGHTS_REQUEST *qwr) {
2130 weights_do_node_callback(&qwd, qwr->host, true);
2131 else {
2132 if((qwd.qwr->method == WEIGHTS_METHOD_VALUE || qwd.qwr->method == WEIGHTS_METHOD_ANOMALY_RATE) && (qwd.contexts_sp || qwd.scope_contexts_sp)) {
2133 +
2134 + if(qwd.qwr->format == WEIGHTS_FORMAT_MCP && qwd.qwr->method == WEIGHTS_METHOD_ANOMALY_RATE)
2135 + qwd.qwr->options |= RRDR_OPTION_ANOMALY_BIT;
2136 +
2137 rrdset_weights_multi_dimensional_value(&qwd);
2138 }
2139 else {
@@ -1940,7 +2164,9 @@ int web_api_v12_weights(BUFFER *wb, QUERY_WEIGHTS_REQUEST *qwr) {
2164 if(!qwd.register_zero)
2165 qwr->options |= RRDR_OPTION_NONZERO;
2166
1943 - if(!(qwr->options & RRDR_OPTION_RETURN_RAW) && qwr->method != WEIGHTS_METHOD_VALUE)
2167 + if(!(qwr->options & RRDR_OPTION_RETURN_RAW) &&
2168 + qwr->method != WEIGHTS_METHOD_VALUE &&
2169 + qwr->format != WEIGHTS_FORMAT_MCP)
2170 spread_results_evenly(qwd.results, &qwd.stats);
2171
2172 usec_t ended_usec = qwd.timings.executed_ut = now_monotonic_usec();
@@ -1972,6 +2198,17 @@ int web_api_v12_weights(BUFFER *wb, QUERY_WEIGHTS_REQUEST *qwr) {
2198 ended_usec - qwd.timings.received_ut, &qwd.stats);
2199 break;
2200
2201 + case WEIGHTS_FORMAT_MCP:
2202 + added_dimensions =
2203 + registered_results_to_json_mcp(
2204 + qwd.results, wb,
2205 + qwr->after, qwr->before,
2206 + qwr->baseline_after, qwr->baseline_before,
2207 + qwr->points, qwr->method, qwr->time_group_method, qwr->options, qwd.shifts,
2208 + qwd.examined_dimensions,
2209 + &qwd, &qwd.stats, &qwd.versions);
2210 + break;
2211 +
2212 default:
2213 case WEIGHTS_FORMAT_MULTINODE:
2214 // we don't support these groupings in weights
@@ -2007,12 +2244,18 @@ int web_api_v12_weights(BUFFER *wb, QUERY_WEIGHTS_REQUEST *qwr) {
2244 cleanup:
2245 simple_pattern_free(qwd.scope_nodes_sp);
2246 simple_pattern_free(qwd.scope_contexts_sp);
2247 + simple_pattern_free(qwd.scope_instances_sp);
2248 + simple_pattern_free(qwd.scope_labels_sp);
2249 + simple_pattern_free(qwd.scope_dimensions_sp);
2250 simple_pattern_free(qwd.nodes_sp);
2251 simple_pattern_free(qwd.contexts_sp);
2252 simple_pattern_free(qwd.instances_sp);
2253 simple_pattern_free(qwd.dimensions_sp);
2254 simple_pattern_free(qwd.labels_sp);
2255 simple_pattern_free(qwd.alerts_sp);
2256 +
2257 + pattern_array_free(qwd.scope_labels_pa);
2258 + pattern_array_free(qwd.labels_pa);
2259
2260 register_result_destroy(qwd.results);
2261
src/web/api/queries/weights.h
+5
@@ -16,6 +16,7 @@ typedef enum {
16 WEIGHTS_FORMAT_CHARTS = 1,
17 WEIGHTS_FORMAT_CONTEXTS = 2,
18 WEIGHTS_FORMAT_MULTINODE = 3,
19 + WEIGHTS_FORMAT_MCP = 4,
20 } WEIGHTS_FORMAT;
21
22 extern int metric_correlations_version;
@@ -27,6 +28,9 @@ typedef struct query_weights_request {
28 RRDHOST *host;
29 const char *scope_nodes;
30 const char *scope_contexts;
31 + const char *scope_instances;
32 + const char *scope_labels;
33 + const char *scope_dimensions;
34 const char *nodes;
35 const char *contexts;
36 const char *instances;
@@ -52,6 +56,7 @@ typedef struct query_weights_request {
56 RRDR_OPTIONS options;
57 size_t tier;
58 time_t timeout_ms;
59 + size_t cardinality_limit;
60
61 weights_interrupt_callback_t interrupt_callback;
62 void *interrupt_callback_data;
src/web/api/request_source.c deleted
-97
@@ -1,97 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -#include "request_source.h"
4 -
5 -bool request_source_is_cloud(const char *source) {
6 - return source && *source && strstartswith(source, "method=NC,");
7 -}
8 -
9 -void web_client_api_request_vX_source_to_buffer(struct web_client *w, BUFFER *source) {
10 - if(web_client_flag_check(w, WEB_CLIENT_FLAG_AUTH_CLOUD))
11 - buffer_sprintf(source, "method=NC");
12 - else if(web_client_flag_check(w, WEB_CLIENT_FLAG_AUTH_BEARER))
13 - buffer_sprintf(source, "method=api-bearer");
14 - else
15 - buffer_sprintf(source, "method=api");
16 -
17 - if(web_client_flag_check(w, WEB_CLIENT_FLAG_AUTH_GOD))
18 - buffer_strcat(source, ",role=god");
19 - else
20 - buffer_sprintf(source, ",role=%s", http_id2user_role(w->user_role));
21 -
22 - buffer_sprintf(source, ",permissions="HTTP_ACCESS_FORMAT, (HTTP_ACCESS_FORMAT_CAST)w->access);
23 -
24 - if(w->auth.client_name[0])
25 - buffer_sprintf(source, ",user=%s", w->auth.client_name);
26 -
27 - if(!uuid_is_null(w->auth.cloud_account_id)) {
28 - char uuid_str[UUID_COMPACT_STR_LEN];
29 - uuid_unparse_lower_compact(w->auth.cloud_account_id, uuid_str);
30 - buffer_sprintf(source, ",account=%s", uuid_str);
31 - }
32 -
33 - if(w->client_ip[0])
34 - buffer_sprintf(source, ",ip=%s", w->client_ip);
35 -
36 - if(w->forwarded_for)
37 - buffer_sprintf(source, ",forwarded_for=%s", w->forwarded_for);
38 -}
39 -
40 -bool parse_request_source(const char *src, PARSED_REQUEST_SOURCE *parsed) {
41 - char *buf, *token, *saveptr;
42 -
43 - if (!src || !parsed)
44 - return false;
45 -
46 - // Initialize the structure.
47 - memset(parsed, 0, sizeof(*parsed));
48 -
49 - // Duplicate the source string because strtok_r modifies it.
50 - buf = strdupz(src);
51 - token = strtok_r(buf, ",", &saveptr);
52 - while (token) {
53 - char *equal_sign = strchr(token, '=');
54 - if (equal_sign) {
55 - *equal_sign = '\0';
56 - const char *key = token;
57 - const char *value = equal_sign + 1;
58 -
59 - if (strcmp(key, "method") == 0) {
60 - if (strcmp(value, "NC") == 0)
61 - parsed->auth |= WEB_CLIENT_FLAG_AUTH_CLOUD;
62 - else if (strcmp(value, "api-bearer") == 0)
63 - parsed->auth |= WEB_CLIENT_FLAG_AUTH_BEARER;
64 - // "api" does not add an extra flag.
65 - }
66 - else if (strcmp(key, "role") == 0) {
67 - if (strcmp(value, "god") == 0)
68 - parsed->auth |= WEB_CLIENT_FLAG_AUTH_GOD;
69 - else
70 - parsed->user_role = http_user_role2id(value);
71 - }
72 - else if (strcmp(key, "permissions") == 0)
73 - parsed->access = http_access_from_hex_str(value);
74 -
75 - else if (strcmp(key, "user") == 0) {
76 - strncpy(parsed->client_name, value, CLOUD_CLIENT_NAME_LENGTH - 1);
77 - parsed->client_name[CLOUD_CLIENT_NAME_LENGTH - 1] = '\0';
78 - }
79 - else if (strcmp(key, "account") == 0) {
80 - if (uuid_parse(value, parsed->cloud_account_id.uuid) != 0)
81 - parsed->cloud_account_id = UUID_ZERO;
82 - }
83 - else if (strcmp(key, "ip") == 0) {
84 - strncpy(parsed->client_ip, value, INET6_ADDRSTRLEN - 1);
85 - parsed->client_ip[INET6_ADDRSTRLEN - 1] = '\0';
86 - }
87 - else if (strcmp(key, "forwarded_for") == 0) {
88 - strncpy(parsed->forwarded_for, value, INET6_ADDRSTRLEN - 1);
89 - parsed->forwarded_for[INET6_ADDRSTRLEN - 1] = '\0';
90 - }
91 - }
92 - token = strtok_r(NULL, ",", &saveptr);
93 - }
94 -
95 - freez(buf);
96 - return true;
97 -}
src/web/api/request_source.h deleted
-24
@@ -1,24 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -#ifndef NETDATA_REQUEST_SOURCE_H
4 -#define NETDATA_REQUEST_SOURCE_H
5 -
6 -#include "web_api.h"
7 -
8 -typedef struct {
9 - char client_ip[INET6_ADDRSTRLEN];
10 - char forwarded_for[INET6_ADDRSTRLEN];
11 - char client_name[CLOUD_CLIENT_NAME_LENGTH];
12 - ND_UUID cloud_account_id;
13 - WEB_CLIENT_FLAGS auth;
14 - HTTP_USER_ROLE user_role;
15 - HTTP_ACCESS access;
16 -} PARSED_REQUEST_SOURCE;
17 -
18 -bool parse_request_source(const char *src, PARSED_REQUEST_SOURCE *parsed);
19 -
20 -bool request_source_is_cloud(const char *source);
21 -
22 -void web_client_api_request_vX_source_to_buffer(struct web_client *w, BUFFER *source);
23 -
24 -#endif //NETDATA_REQUEST_SOURCE_H
src/web/api/v1/api_v1_allmetrics.c
+1 -1
@@ -194,7 +194,7 @@ void rrd_stats_api_v1_charts_allmetrics_json(RRDHOST *host, const char *filter_s
194 int api_v1_allmetrics(RRDHOST *host, struct web_client *w, char *url) {
195 int format = ALLMETRICS_SHELL;
196 const char *filter = NULL;
197 - const char *prometheus_server = w->client_ip;
197 + const char *prometheus_server = w->user_auth.client_ip;
198
199 uint32_t prometheus_exporting_options;
200 if (prometheus_exporter_instance)
src/web/api/v1/api_v1_config.c
+2 -2
@@ -78,10 +78,10 @@ int api_v1_config(RRDHOST *host, struct web_client *w, char *url __maybe_unused)
78 }
79
80 CLEAN_BUFFER *source = buffer_create(100, NULL);
81 - web_client_api_request_vX_source_to_buffer(w, source);
81 + user_auth_to_source_buffer(&w->user_auth, source);
82
83 buffer_flush(w->response.data);
84 - int code = rrd_function_run(host, w->response.data, timeout, w->access, cmd,
84 + int code = rrd_function_run(host, w->response.data, timeout, w->user_auth.access, cmd,
85 true, transaction,
86 NULL, NULL,
87 web_client_progress_functions_update, w,
src/web/api/v1/api_v1_data.c
+7
@@ -29,7 +29,9 @@ int api_v1_data(RRDHOST *host, struct web_client *w, char *url) {
29 char *chart_label_key = NULL;
30 char *chart_labels_filter = NULL;
31 char *group_options = NULL;
32 + char *cardinality_limit_str = NULL;
33 size_t tier = 0;
34 + size_t cardinality_limit = 0;
35 RRDR_TIME_GROUPING group = RRDR_GROUPING_AVERAGE;
36 DATASOURCE_FORMAT format = DATASOURCE_JSON;
37 RRDR_OPTIONS options = 0;
@@ -63,6 +65,7 @@ int api_v1_data(RRDHOST *host, struct web_client *w, char *url) {
65 else if(!strcmp(name, "timeout")) timeout_str = value;
66 else if(!strcmp(name, "gtime")) group_time_str = value;
67 else if(!strcmp(name, "group_options")) group_options = value;
68 + else if(!strcmp(name, "cardinality_limit")) cardinality_limit_str = value;
69 else if(!strcmp(name, "group")) {
70 group = time_grouping_parse(value, RRDR_GROUPING_AVERAGE);
71 }
@@ -146,6 +149,9 @@ int api_v1_data(RRDHOST *host, struct web_client *w, char *url) {
149 int points = (points_str && *points_str)?str2i(points_str):0;
150 int timeout = (timeout_str && *timeout_str)?str2i(timeout_str): 0;
151 long group_time = (group_time_str && *group_time_str)?str2l(group_time_str):0;
152 +
153 + if (cardinality_limit_str && *cardinality_limit_str)
154 + cardinality_limit = str2ul(cardinality_limit_str);
155
156 QUERY_TARGET_REQUEST qtr = {
157 .version = 1,
@@ -169,6 +175,7 @@ int api_v1_data(RRDHOST *host, struct web_client *w, char *url) {
175 .labels = chart_labels_filter,
176 .query_source = QUERY_SOURCE_API_DATA,
177 .priority = STORAGE_PRIORITY_NORMAL,
178 + .cardinality_limit = cardinality_limit,
179 .interrupt_callback = web_client_interrupt_callback,
180 .interrupt_callback_data = w,
181 .transaction = &w->transaction,
src/web/api/v1/api_v1_function.c
+2 -2
@@ -34,9 +34,9 @@ int api_v1_function(RRDHOST *host, struct web_client *w, char *url) {
34 uuid_unparse_lower_compact(w->transaction, transaction);
35
36 CLEAN_BUFFER *source = buffer_create(100, NULL);
37 - web_client_api_request_vX_source_to_buffer(w, source);
37 + user_auth_to_source_buffer(&w->user_auth, source);
38
39 - return rrd_function_run(host, wb, timeout, w->access, function, true, transaction,
39 + return rrd_function_run(host, wb, timeout, w->user_auth.access, function, true, transaction,
40 NULL, NULL,
41 web_client_progress_functions_update, w,
42 web_client_interrupt_callback, w, w->payload,
src/web/api/v2/api_v2_alert_transitions.c
+1 -1
@@ -3,5 +3,5 @@
3 #include "api_v2_calls.h"
4
5 int api_v2_alert_transitions(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 - return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_ALERT_TRANSITIONS | CONTEXTS_V2_NODES);
6 + return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_ALERT_TRANSITIONS | CONTEXTS_V2_NODES, 0);
7 }
src/web/api/v2/api_v2_alerts.c
+1 -1
@@ -3,5 +3,5 @@
3 #include "api_v2_calls.h"
4
5 int api_v2_alerts(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 - return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_ALERTS | CONTEXTS_V2_NODES);
6 + return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_ALERTS | CONTEXTS_V2_NODES, CONTEXTS_OPTION_SUMMARY);
7 }
src/web/api/v2/api_v2_bearer.c
+4 -4
@@ -132,8 +132,8 @@ int api_v2_bearer_get_token(RRDHOST *host, struct web_client *w, char *url) {
132 claim_id,
133 machine_guid,
134 node_id,
135 - w->user_role,
136 - w->access,
137 - w->auth.cloud_account_id,
138 - w->auth.client_name);
135 + w->user_auth.user_role,
136 + w->user_auth.access,
137 + w->user_auth.cloud_account_id.uuid,
138 + w->user_auth.client_name);
139 }
src/web/api/v2/api_v2_calls.h
+2 -1
@@ -8,11 +8,12 @@
8 int api_v2_info(RRDHOST *host, struct web_client *w, char *url);
9
10 int api_v2_data(RRDHOST *host, struct web_client *w, char *url);
11 +int api_v3_data(RRDHOST *host, struct web_client *w, char *url);
12 int api_v2_weights(RRDHOST *host, struct web_client *w, char *url);
13
14 int api_v2_alert_config(RRDHOST *host, struct web_client *w, char *url);
15
15 -int api_v2_contexts_internal(RRDHOST *host, struct web_client *w, char *url, CONTEXTS_V2_MODE mode);
16 +int api_v2_contexts_internal(RRDHOST *host, struct web_client *w, char *url, CONTEXTS_V2_MODE mode, CONTEXTS_OPTIONS options);
17 int api_v2_contexts(RRDHOST *host, struct web_client *w, char *url);
18 int api_v2_alert_transitions(RRDHOST *host, struct web_client *w, char *url);
19 int api_v2_alerts(RRDHOST *host, struct web_client *w, char *url);
src/web/api/v2/api_v2_claim.c
+1 -1
@@ -158,7 +158,7 @@ static int claim_json_response(BUFFER *wb, CLAIM_RESPONSE response, const char *
158 claim_add_user_info_command(wb);
159 }
160
161 - buffer_json_agents_v2(wb, NULL, now_s, false, false);
161 + buffer_json_agents_v2(wb, NULL, now_s, false, false, 0);
162 buffer_json_finalize(wb);
163
164 return (response == CLAIM_RESP_ERROR) ? HTTP_RESP_BAD_REQUEST : HTTP_RESP_OK;
src/web/api/v2/api_v2_contexts.c
+10 -5
@@ -4,8 +4,10 @@
4
5 // --------------------------------------------------------------------------------------------------------------------
6
7 -int api_v2_contexts_internal(RRDHOST *host __maybe_unused, struct web_client *w, char *url, CONTEXTS_V2_MODE mode) {
8 - struct api_v2_contexts_request req = { 0 };
7 +int api_v2_contexts_internal(RRDHOST *host __maybe_unused, struct web_client *w, char *url, CONTEXTS_V2_MODE mode, CONTEXTS_OPTIONS options) {
8 + struct api_v2_contexts_request req = {
9 + .options = options,
10 + };
11
12 while(url) {
13 char *value = strsep_skip_consecutive_separators(&url, "&");
@@ -29,13 +31,15 @@ int api_v2_contexts_internal(RRDHOST *host __maybe_unused, struct web_client *w,
31 else if((mode & CONTEXTS_V2_SEARCH) && !strcmp(name, "q"))
32 req.q = value;
33 else if(!strcmp(name, "options"))
32 - req.options = contexts_options_str_to_id(value);
34 + req.options |= contexts_options_str_to_id(value);
35 else if(!strcmp(name, "after"))
36 req.after = str2l(value);
37 else if(!strcmp(name, "before"))
38 req.before = str2l(value);
39 else if(!strcmp(name, "timeout"))
40 req.timeout_ms = str2l(value);
41 + else if(!strcmp(name, "cardinality") || !strcmp(name, "cardinality_limit"))
42 + req.cardinality_limit = str2ul(value);
43 else if(mode & (CONTEXTS_V2_ALERTS | CONTEXTS_V2_ALERT_TRANSITIONS)) {
44 if (!strcmp(name, "alert"))
45 req.alerts.alert = value;
@@ -73,6 +77,7 @@ int api_v2_contexts_internal(RRDHOST *host __maybe_unused, struct web_client *w,
77
78 int api_v2_contexts(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
79 return api_v2_contexts_internal(
76 - host, w, url, CONTEXTS_V2_CONTEXTS | CONTEXTS_V2_NODES | CONTEXTS_V2_AGENTS | CONTEXTS_V2_VERSIONS);
80 + host, w, url, CONTEXTS_V2_CONTEXTS | CONTEXTS_V2_NODES | CONTEXTS_V2_AGENTS | CONTEXTS_V2_VERSIONS,
81 + CONTEXTS_OPTION_PRIORITIES | CONTEXTS_OPTION_RETENTION | CONTEXTS_OPTION_LIVENESS |
82 + CONTEXTS_OPTION_FAMILY | CONTEXTS_OPTION_UNITS);
83 }
78 -
src/web/api/v2/api_v2_data.c
+28 -3
@@ -17,14 +17,14 @@ __attribute__((constructor)) void initialize_group_by_keys(void) {
17 }
18 }
19
20 -int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
20 +static int api_v23_data_internal(RRDHOST *host __maybe_unused, struct web_client *w, char *url, size_t version) {
21 usec_t received_ut = now_monotonic_usec();
22
23 int ret = HTTP_RESP_BAD_REQUEST;
24
25 buffer_flush(w->response.data);
26
27 - char *google_version = "0.6",
27 + char *google_version = "0.6",
28 *google_reqId = "0",
29 *google_sig = "0",
30 *google_out = "json",
@@ -35,6 +35,9 @@ int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
35
36 char *scope_nodes = NULL;
37 char *scope_contexts = NULL;
38 + char *scope_instances = NULL;
39 + char *scope_labels = NULL;
40 + char *scope_dimensions = NULL;
41 char *nodes = NULL;
42 char *contexts = NULL;
43 char *instances = NULL;
@@ -48,7 +51,9 @@ int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
51 char *alerts = NULL;
52 char *time_group_options = NULL;
53 char *tier_str = NULL;
54 + char *cardinality_limit_str = NULL;
55 size_t tier = 0;
56 + size_t cardinality_limit = 0;
57 RRDR_TIME_GROUPING time_group = RRDR_GROUPING_AVERAGE;
58 DATASOURCE_FORMAT format = DATASOURCE_JSON2;
59 RRDR_OPTIONS options = RRDR_OPTION_VIRTUAL_POINTS | RRDR_OPTION_JSON_WRAP | RRDR_OPTION_RETURN_JWAR;
@@ -76,6 +81,9 @@ int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
81
82 if(!strcmp(name, "scope_nodes")) scope_nodes = value;
83 else if(!strcmp(name, "scope_contexts")) scope_contexts = value;
84 + else if(!strcmp(name, "scope_instances")) scope_instances = value;
85 + else if(!strcmp(name, "scope_labels")) scope_labels = value;
86 + else if(!strcmp(name, "scope_dimensions")) scope_dimensions = value;
87 else if(!strcmp(name, "nodes")) nodes = value;
88 else if(!strcmp(name, "contexts")) contexts = value;
89 else if(!strcmp(name, "instances")) instances = value;
@@ -107,6 +115,7 @@ int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
115 else if(!strcmp(name, "time_group_options")) time_group_options = value;
116 else if(!strcmp(name, "time_resampling")) resampling_time_str = value;
117 else if(!strcmp(name, "tier")) tier_str = value;
118 + else if(!strcmp(name, "cardinality_limit")) cardinality_limit_str = value;
119 else if(!strcmp(name, "callback")) responseHandler = value;
120 else if(!strcmp(name, "filename")) outFileName = value;
121 else if(!strcmp(name, "tqx")) {
@@ -186,6 +195,10 @@ int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
195 else
196 tier = 0;
197 }
198 +
199 + if(cardinality_limit_str && *cardinality_limit_str) {
200 + cardinality_limit = str2ul(cardinality_limit_str);
201 + }
202
203 time_t before = (before_str && *before_str)?str2l(before_str):0;
204 time_t after = (after_str && *after_str) ?str2l(after_str):-600;
@@ -194,9 +207,12 @@ int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
207 time_t resampling_time = (resampling_time_str && *resampling_time_str) ? str2l(resampling_time_str) : 0;
208
209 QUERY_TARGET_REQUEST qtr = {
197 - .version = 2,
210 + .version = version,
211 .scope_nodes = scope_nodes,
212 .scope_contexts = scope_contexts,
213 + .scope_instances = scope_instances,
214 + .scope_labels = scope_labels,
215 + .scope_dimensions = scope_dimensions,
216 .after = after,
217 .before = before,
218 .host = NULL,
@@ -219,6 +235,7 @@ int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
235 .query_source = QUERY_SOURCE_API_DATA,
236 .priority = STORAGE_PRIORITY_NORMAL,
237 .received_ut = received_ut,
238 + .cardinality_limit = cardinality_limit,
239
240 .interrupt_callback = web_client_interrupt_callback,
241 .interrupt_callback_data = w,
@@ -301,3 +318,11 @@ cleanup:
318 onewayalloc_destroy(owa);
319 return ret;
320 }
321 +
322 +int api_v2_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
323 + return api_v23_data_internal(host, w, url, 2);
324 +}
325 +
326 +int api_v3_data(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
327 + return api_v23_data_internal(host, w, url, 3);
328 +}
src/web/api/v2/api_v2_functions.c
+1 -1
@@ -4,5 +4,5 @@
4
5 int api_v2_functions(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 return api_v2_contexts_internal(
7 - host, w, url, CONTEXTS_V2_FUNCTIONS | CONTEXTS_V2_NODES | CONTEXTS_V2_AGENTS | CONTEXTS_V2_VERSIONS);
7 + host, w, url, CONTEXTS_V2_FUNCTIONS | CONTEXTS_V2_NODES | CONTEXTS_V2_AGENTS | CONTEXTS_V2_VERSIONS, 0);
8 }
src/web/api/v2/api_v2_info.c
+1 -1
@@ -3,5 +3,5 @@
3 #include "api_v2_calls.h"
4
5 int api_v2_info(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 - return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_AGENTS | CONTEXTS_V2_AGENTS_INFO);
6 + return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_AGENTS | CONTEXTS_V2_AGENTS_INFO, 0);
7 }
src/web/api/v2/api_v2_node_instances.c
+2 -2
@@ -5,6 +5,6 @@
5 int api_v2_node_instances(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 return api_v2_contexts_internal(
7 host, w, url,
8 - CONTEXTS_V2_NODES | CONTEXTS_V2_NODE_INSTANCES | CONTEXTS_V2_AGENTS |
9 - CONTEXTS_V2_AGENTS_INFO | CONTEXTS_V2_VERSIONS);
8 + CONTEXTS_V2_NODES | CONTEXTS_V2_NODE_INSTANCES |
9 + CONTEXTS_V2_AGENTS | CONTEXTS_V2_AGENTS_INFO | CONTEXTS_V2_VERSIONS, 0);
10 }
src/web/api/v2/api_v2_nodes.c
+1 -1
@@ -3,5 +3,5 @@
3 #include "api_v2_calls.h"
4
5 int api_v2_nodes(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 - return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_NODES | CONTEXTS_V2_NODES_INFO);
6 + return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_NODES | CONTEXTS_V2_NODES_INFO, 0);
7 }
src/web/api/v2/api_v2_q.c
+3 -1
@@ -5,5 +5,7 @@
5 int api_v2_q(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 return api_v2_contexts_internal(
7 host, w, url,
8 - CONTEXTS_V2_SEARCH | CONTEXTS_V2_CONTEXTS | CONTEXTS_V2_NODES | CONTEXTS_V2_AGENTS | CONTEXTS_V2_VERSIONS);
8 + CONTEXTS_V2_SEARCH | CONTEXTS_V2_NODES | CONTEXTS_V2_AGENTS | CONTEXTS_V2_VERSIONS,
9 + CONTEXTS_OPTION_FAMILY | CONTEXTS_OPTION_UNITS | CONTEXTS_OPTION_TITLES | CONTEXTS_OPTION_LABELS |
10 + CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_DIMENSIONS);
11 }
src/web/api/v2/api_v2_versions.c
+1 -1
@@ -3,5 +3,5 @@
3 #include "api_v2_calls.h"
4
5 int api_v2_versions(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 - return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_VERSIONS);
6 + return api_v2_contexts_internal(host, w, url, CONTEXTS_V2_VERSIONS, 0);
7 }
src/web/api/v2/api_v2_weights.c
+13 -2
@@ -12,8 +12,9 @@ int web_client_api_request_weights(RRDHOST *host, struct web_client *w, char *ur
12 RRDR_TIME_GROUPING time_group_method = RRDR_GROUPING_AVERAGE;
13 time_t timeout_ms = 0;
14 size_t tier = 0;
15 - const char *time_group_options = NULL, *scope_contexts = NULL, *scope_nodes = NULL, *contexts = NULL, *nodes = NULL,
16 - *instances = NULL, *dimensions = NULL, *labels = NULL, *alerts = NULL;
15 + size_t cardinality_limit = 0;
16 + const char *time_group_options = NULL, *scope_contexts = NULL, *scope_nodes = NULL, *scope_instances = NULL, *scope_labels = NULL, *scope_dimensions = NULL,
17 + *contexts = NULL, *nodes = NULL, *instances = NULL, *dimensions = NULL, *labels = NULL, *alerts = NULL;
18
19 struct group_by_pass group_by = {
20 .group_by = RRDR_GROUP_BY_NONE,
@@ -49,6 +50,9 @@ int web_client_api_request_weights(RRDHOST *host, struct web_client *w, char *ur
50
51 else if (!strcmp(name, "timeout"))
52 timeout_ms = str2l(value);
53 +
54 + else if (!strcmp(name, "cardinality_limit"))
55 + cardinality_limit = str2ul(value);
56
57 else if((api_version == 1 && !strcmp(name, "group")) || (api_version >= 2 && !strcmp(name, "time_group")))
58 time_group_method = time_grouping_parse(value, RRDR_GROUPING_AVERAGE);
@@ -67,6 +71,9 @@ int web_client_api_request_weights(RRDHOST *host, struct web_client *w, char *ur
71
72 else if(api_version >= 2 && !strcmp(name, "scope_nodes")) scope_nodes = value;
73 else if(api_version >= 2 && !strcmp(name, "scope_contexts")) scope_contexts = value;
74 + else if(api_version >= 2 && !strcmp(name, "scope_instances")) scope_instances = value;
75 + else if(api_version >= 2 && !strcmp(name, "scope_labels")) scope_labels = value;
76 + else if(api_version >= 2 && !strcmp(name, "scope_dimensions")) scope_dimensions = value;
77 else if(api_version >= 2 && !strcmp(name, "nodes")) nodes = value;
78 else if(api_version >= 2 && !strcmp(name, "contexts")) contexts = value;
79 else if(api_version >= 2 && !strcmp(name, "instances")) instances = value;
@@ -114,6 +121,9 @@ int web_client_api_request_weights(RRDHOST *host, struct web_client *w, char *ur
121 .host = (api_version == 1) ? NULL : host,
122 .scope_nodes = scope_nodes,
123 .scope_contexts = scope_contexts,
124 + .scope_instances = scope_instances,
125 + .scope_labels = scope_labels,
126 + .scope_dimensions = scope_dimensions,
127 .nodes = nodes,
128 .contexts = contexts,
129 .instances = instances,
@@ -137,6 +147,7 @@ int web_client_api_request_weights(RRDHOST *host, struct web_client *w, char *ur
147 .options = options,
148 .tier = tier,
149 .timeout_ms = timeout_ms,
150 + .cardinality_limit = cardinality_limit,
151
152 .interrupt_callback = web_client_interrupt_callback,
153 .interrupt_callback_data = w,
src/web/api/v3/api_v3_calls.h
+2 -2
@@ -5,8 +5,8 @@
5
6 #include "../web_api_v3.h"
7
8 -int api_v2_contexts_internal(RRDHOST *host, struct web_client *w, char *url, CONTEXTS_V2_MODE mode);
9 -#define api_v3_contexts_internal(host, w, url, mode) api_v2_contexts_internal(host, w, url, mode)
8 +int api_v2_contexts_internal(RRDHOST *host, struct web_client *w, char *url, CONTEXTS_V2_MODE mode, CONTEXTS_OPTIONS options);
9 +#define api_v3_contexts_internal(host, w, url, mode, options) api_v2_contexts_internal(host, w, url, mode, options)
10
11 int api_v3_settings(RRDHOST *host, struct web_client *w, char *url);
12 int api_v3_me(RRDHOST *host, struct web_client *w, char *url);
src/web/api/v3/api_v3_me.c
+9 -8
@@ -8,29 +8,30 @@ int api_v3_me(RRDHOST *host __maybe_unused, struct web_client *w, char *url __ma
8 buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
9
10 const char *auth;
11 - switch(web_client_flag_check(w, WEB_CLIENT_FLAG_AUTH_CLOUD|WEB_CLIENT_FLAG_AUTH_BEARER|WEB_CLIENT_FLAG_AUTH_GOD)) {
12 - case WEB_CLIENT_FLAG_AUTH_CLOUD:
11 + switch(w->user_auth.method) {
12 + case USER_AUTH_METHOD_CLOUD:
13 auth = "cloud";
14 break;
15
16 - case WEB_CLIENT_FLAG_AUTH_BEARER:
16 + case USER_AUTH_METHOD_BEARER:
17 auth = "bearer";
18 break;
19
20 - case WEB_CLIENT_FLAG_AUTH_GOD:
20 + case USER_AUTH_METHOD_GOD:
21 auth = "god";
22 break;
23
24 default:
25 + case USER_AUTH_METHOD_NONE:
26 auth = "none";
27 break;
28 }
29 buffer_json_member_add_string(wb, "auth", auth);
30
30 - buffer_json_member_add_uuid(wb, "cloud_account_id", w->auth.cloud_account_id);
31 - buffer_json_member_add_string(wb, "client_name", w->auth.client_name);
32 - http_access2buffer_json_array(wb, "access", w->access);
33 - buffer_json_member_add_string(wb, "user_role", http_id2user_role(w->user_role));
31 + buffer_json_member_add_uuid(wb, "cloud_account_id", w->user_auth.cloud_account_id.uuid);
32 + buffer_json_member_add_string(wb, "client_name", w->user_auth.client_name);
33 + http_access2buffer_json_array(wb, "access", w->user_auth.access);
34 + buffer_json_member_add_string(wb, "user_role", http_id2user_role(w->user_auth.user_role));
35
36 buffer_json_finalize(wb);
37 return HTTP_RESP_OK;
src/web/api/v3/api_v3_settings.c
+1 -1
@@ -257,7 +257,7 @@ int api_v3_settings(RRDHOST *host, struct web_client *w, char *url) {
257 "Settings API is only allowed for the agent node.",
258 HTTP_RESP_BAD_REQUEST);
259
260 - if(web_client_flags_check_auth(w) != WEB_CLIENT_FLAG_AUTH_BEARER && strcmp(file, "default") != 0)
260 + if(w->user_auth.method != USER_AUTH_METHOD_BEARER && strcmp(file, "default") != 0)
261 return rrd_call_function_error(
262 w->response.data,
263 "Only the 'default' settings file is allowed for anonymous users",
src/web/api/v3/api_v3_stream_path.c
+1 -1
@@ -3,5 +3,5 @@
3 #include "api_v3_calls.h"
4
5 int api_v3_stream_path(RRDHOST *host __maybe_unused, struct web_client *w, char *url) {
6 - return api_v3_contexts_internal(host, w, url, CONTEXTS_V2_NODES | CONTEXTS_V2_NODES_STREAM_PATH);
6 + return api_v3_contexts_internal(host, w, url, CONTEXTS_V2_NODES | CONTEXTS_V2_NODES_STREAM_PATH, 0);
7 }
src/web/api/web_api.c
+15 -9
@@ -8,26 +8,30 @@ void host_labels2json(RRDHOST *host, BUFFER *wb, const char *key) {
8 buffer_json_object_close(wb);
9 }
10
11 -int web_client_api_request_vX(RRDHOST *host, struct web_client *w, char *url_path_endpoint, struct web_api_command *api_commands) {
12 - buffer_no_cacheable(w->response.data);
13 -
14 - internal_fatal(web_client_flags_check_auth(w) && !(w->access & HTTP_ACCESS_SIGNED_ID),
11 +void web_client_ensure_proper_authorization(struct web_client *w) {
12 + internal_fatal(w->user_auth.method != USER_AUTH_METHOD_NONE && !(w->user_auth.access & HTTP_ACCESS_SIGNED_ID),
13 "signed-in permission should be set, but is missing");
14
17 - internal_fatal(!web_client_flags_check_auth(w) && (w->access & HTTP_ACCESS_SIGNED_ID),
15 + internal_fatal(w->user_auth.method == USER_AUTH_METHOD_NONE && (w->user_auth.access & HTTP_ACCESS_SIGNED_ID),
16 "signed-in permission is set, but it shouldn't");
17
18 #ifdef NETDATA_GOD_MODE
21 - web_client_set_permissions(w, HTTP_ACCESS_ALL, HTTP_USER_ROLE_ADMIN, WEB_CLIENT_FLAG_AUTH_GOD);
19 + web_client_set_permissions(w, HTTP_ACCESS_ALL, HTTP_USER_ROLE_ADMIN, USER_AUTH_METHOD_GOD);
20 #else
23 - if(!web_client_flags_check_auth(w)) {
21 + if(w->user_auth.method == USER_AUTH_METHOD_NONE) {
22 web_client_set_permissions(
23 w,
24 (netdata_is_protected_by_bearer) ? HTTP_ACCESS_NONE : HTTP_ACCESS_ANONYMOUS_DATA,
25 (netdata_is_protected_by_bearer) ? HTTP_USER_ROLE_NONE : HTTP_USER_ROLE_ANY,
28 - 0);
26 + USER_AUTH_METHOD_NONE);
27 }
28 #endif
29 +}
30 +
31 +int web_client_api_request_vX(RRDHOST *host, struct web_client *w, char *url_path_endpoint, struct web_api_command *api_commands) {
32 + buffer_no_cacheable(w->response.data);
33 +
34 + web_client_ensure_proper_authorization(w);
35
36 if(unlikely(!url_path_endpoint || !*url_path_endpoint)) {
37 buffer_flush(w->response.data);
@@ -63,7 +67,7 @@ int web_client_api_request_vX(RRDHOST *host, struct web_client *w, char *url_pat
67 return web_client_permission_denied_acl(w);
68
69 bool permissions_allows =
66 - http_access_user_has_enough_access_level_for_endpoint(w->access, api_commands[i].access);
70 + http_access_user_has_enough_access_level_for_endpoint(w->user_auth.access, api_commands[i].access);
71 if(!permissions_allows)
72 return web_client_permission_denied(w);
73
@@ -112,6 +116,8 @@ RRDCONTEXT_TO_JSON_OPTIONS rrdcontext_to_json_parse_options(char *o) {
116 options |= RRDCONTEXT_OPTION_DEEPSCAN;
117 else if(!strcmp(tok, "hidden"))
118 options |= RRDCONTEXT_OPTION_SHOW_HIDDEN;
119 + else if(!strcmp(tok, "rfc3339"))
120 + options |= RRDCONTEXT_OPTION_RFC3339;
121 }
122
123 return options;
src/web/api/web_api.h
+3 -1
@@ -16,7 +16,7 @@ struct web_client;
16 #include "web/api/http_auth.h"
17 #include "web/api/formatters/rrd2json.h"
18 #include "web/api/queries/weights.h"
19 -#include "web/api/request_source.h"
19 +#include "libnetdata/user-auth/user-auth.h"
20
21 void nd_web_api_init(void);
22
@@ -53,6 +53,8 @@ bool web_client_interrupt_callback(void *data);
53 char *format_value_and_unit(char *value_string, size_t value_string_len,
54 NETDATA_DOUBLE value, const char *units, int precision);
55
56 +void web_client_ensure_proper_authorization(struct web_client *w);
57 +
58 #include "web_api_v1.h"
59 #include "web_api_v2.h"
60 #include "web_api_v3.h"
src/web/api/web_api_v3.c
+1 -1
@@ -12,7 +12,7 @@ static struct web_api_command api_commands_v3[] = {
12 .hash = 0,
13 .acl = HTTP_ACL_METRICS,
14 .access = HTTP_ACCESS_ANONYMOUS_DATA,
15 - .callback = api_v2_data,
15 + .callback = api_v3_data,
16 .allow_subpaths = 0
17 },
18 // badges
src/web/mcp/README.md new
+349
@@ -0,0 +1,349 @@
1 +# Netdata Model Context Protocol (MCP) Integration
2 +
3 +Netdata Agents (and soon Netdata Cloud) provide a Model Context Protocol (MCP) server that enables AI assistants like Claude or Cursos to interact with your infrastructure monitoring data. This integration allows AI assistants to access metrics, logs, alerts, and live system information (processes, services, containers, VMs, network connections, etc), acting as a capable DevOps/SRE/SysAdmin assistant.
4 +
5 +## Overview
6 +
7 +The AI assistants have different visibility on your infrastructure, depending on where in a Netdata hierarchy they are connected:
8 +
9 + - **Netdata Cloud**: (not yet available) AI assistants connected to Netdata Cloud will have full visibility across all nodes in your infrastructure.
10 + - **Netdata Parent Node**: AI assistants connected to a Netdata parent node will have visibility across all child nodes connected to that parent.
11 + - **Netdata Child Node**: AI assistants connected to a Netdata child node will only have visibility into that specific node.
12 + - **Netdata Standalone Node**: AI assistants connected to a standalone Netdata node will only have visibility into that specific node.
13 +
14 +## Supported AI Assistants
15 +
16 +You can use Netdata with the following AI assistants:
17 +
18 +- [Claude Desktop](https://claude.ai/download): supports flat-fee usage for unlimited access
19 +- [Claude Code](https://claude.ai/code): supports flat-fee usage for unlimited access
20 +- [Cursor](https://www.cursor.com/): supports flat-fee usage for unlimited access. Enables Netdata use with multiple AI assistants, including Claude, ChatGPT, and Gemini.
21 +
22 +Probably more: Check the [MCP documentation](https://modelcontextprotocol.io/clients) for a full list of supported AI assistants.
23 +
24 +All these AI assistants need local access to the MCP servers. This means that the application you run locally on your computer (Claude Desktop, Cursor, etc) needs to be able to connect to the Netdata using `stdio` communication. However, since your Netdata runs remotely on a server, you need a bridge to convert the `stdio` communication to `WebSocket` communication. Netdata provides bridges in multiple languages (Node.js, Python, Go) to facilitate this.
25 +
26 +Once MCP is integrated into Netdata Cloud, Web-based AI assistants will also be supported. For Web-based AI assistants, the backend of the assistant connects to a publicly accessible MCP server (i.e. Netdata Cloud) to access infrastructure observability data, without needing a bridge.
27 +
28 +## Installation
29 +
30 +The MCP server is built into Netdata and requires no additional installation. Just ensure you have a recent version of Netdata installed.
31 +
32 +To use the MCP integration of Netdata with AI clients, you need to configure them and bridge them to the Netdata MCP server.
33 +
34 +## Configuration of AI Assistants
35 +
36 +The configuration of most AI assistants is done via a configuration file, which is almost identical for all of them.
37 +
38 +```json
39 +{
40 + "mcpServers": {
41 + "netdata": {
42 + "command": "/usr/bin/nd-mcp",
43 + "args": [
44 + "ws://IP_OF_YOUR_NETDATA:19999/mcp?api_key=YOUR_API_KEY"
45 + ]
46 + }
47 + }
48 +}
49 +```
50 +
51 +The program `nd-mcp` is the bridge program that converts `stdio` communication to `WebSocket` communication. This program is part of all Netdata installations, so by installing Netdata on your personal computer (Linux, MacOS, Windows) you will have it available.
52 +
53 +There may be different paths for it, depending on how you installed Netdata:
54 +
55 +- `/usr/bin/nd-mcp`: Linux native packages
56 +- `/opt/netdata/usr/bin/nd-mcp`: Linux static Netdata installations
57 +- `/usr/local/netdata/usr/bin/nd-mcp`: MacOS installations from source
58 +- `C:\\Program Files\\Netdata\\usr\\bin\\nd-mcp.exe`: Windows installations
59 +
60 +You will also need:
61 +
62 +`IP_OF_YOUR_NETDATA`, is the IP address or hostname of the Netdata instance you want to connect to. This will eventually be replaced by the Netdata Cloud URL. For this dev preview, use any Netdata, preferably one of your parent nodes. Remember that the AI assistant will "see" only the nodes that are connected to that Netdata instance.
63 +
64 +`YOUR_API_KEY` is the API key that allows the AI assistant to access sensitive functions like logs and live system information. Just start Netdata and it will automatically generate a random UUID for you. You can find it at:
65 +
66 +```
67 +/var/lib/netdata/mcp_dev_preview_api_key
68 +```
69 +
70 +or, if you installed a static Netdata package, it may be located at:
71 +
72 +```
73 +/opt/netdata/var/lib/netdata/mcp_dev_preview_api_key
74 +```
75 +
76 +
77 +To view your API key:
78 +```bash
79 +sudo cat /var/lib/netdata/mcp_dev_preview_api_key
80 +```
81 +
82 +or
83 +
84 +```bash
85 +sudo cat /opt/netdata/var/lib/netdata/mcp_dev_preview_api_key
86 +```
87 +
88 +### Claude Desktop
89 +
90 +To add Netdata MCP to Claude Desktop:
91 +
92 +1. Open Claude Desktop
93 +2. Navigate to the Developer settings:
94 + - **Windows/Linux**: File → Settings → Developer (or use Ctrl+,)
95 + - **macOS**: Claude → Settings → Developer (or use Cmd+,)
96 +3. Click the "Edit Config" button (below the server list)
97 +4. This will open or show the exact configuration file location
98 +5. Add the configuration mentioned above to that file.
99 +
100 +**Linux Users**: Claude Desktop is available via a community project (https://github.com/fsoft72/claude-desktop-to-appimage). It works best with https://github.com/TheAssassin/AppImageLauncher.
101 +
102 +Once configured correctly, you will need to restart Claude Desktop.
103 +Once restarted, you should see "netdata" appear in Claude Desktop:
104 +- Click the "Search and tools" button (just below the prompt)
105 +- You should see "netdata" listed among the available tools
106 +- If you don't see it, check your configuration and ensure the bridge is accessible
107 +
108 +### Claude Code
109 +
110 +For [Claude Code](https://claude.ai/code), add to your project's root, the file `.mcp.json`, with the contents given above. This file will be automatically detected by Claude Code the next time it starts in that directory.
111 +
112 +Once configured correctly, issue the command `/mcp` to your Claude Code. It should show you the available MCP servers, including "netdata".
113 +
114 +### Cursor
115 +
116 +For [Cursor](https://www.cursor.com/), add the configuration to the MCP settings.
117 +
118 +## Alternative `stdio` to `websocket` Bridges
119 +
120 +We provide 3 different bridges for you to choose the one that best fits your environment:
121 +
122 +1. **Go bridge**: Located at `src/web/mcp/bridges/stdio-golang/`
123 +2. **Node.js bridge**: Located at `src/web/mcp/bridges/stdio-nodejs/`
124 +3. **Python bridge**: Located at `src/web/mcp/bridges/stdio-python/`
125 +
126 +All these bridges should provide exactly the same functionality, so you can choose the one that best fits your environment.
127 +
128 +Each of these directories includes `build.sh` script to install dependencies and prepare the bridge.
129 +The Go bridge provides also a `build.bat` script for Windows users.
130 +
131 +## Capabilities
132 +
133 +The MCP integration provides AI assistants with access to:
134 +
135 +### Infrastructure Discovery
136 +- **Nodes information**: Complete visibility across all connected nodes in your infrastructure
137 + - Hardware specifications, OS details, virtualization info
138 + - Streaming configuration and parent-child relationships
139 + - Connection status and data collection capabilities
140 +- **Metrics discovery**: All metrics collected by your Netdata installation
141 + - System metrics: CPU, memory, disks, network interfaces
142 + - Application metrics: databases, web servers, containers
143 + - Hardware metrics: IPMI sensors, GPU, temperature sensors
144 + - Custom metrics: StatsD, logs-based metrics
145 +
146 +### Metrics and Analytics
147 +- **Time-series queries**: Powerful data aggregation and analysis
148 + - Multiple grouping options: by dimension, instance, node, or label
149 + - Aggregation methods: sum, average, min, max, percentages
150 + - Time aggregations: average, min, max, median, percentile, etc.
151 +- **Anomaly detection**: ML-powered anomaly detection across all metrics
152 + - Real-time anomaly rates (0-100% of time anomalous)
153 + - Per-metric and per-dimension anomaly tracking
154 +- **Correlation analysis**: Find metrics that changed during incidents
155 + - Compare problem periods with baseline periods
156 + - Statistical and volume-based correlation methods
157 +- **Variability analysis**: Identify unstable or fluctuating metrics
158 +
159 +### Live System Information (requires API key and claimed agent)
160 +- **Processes**: Detailed process information including:
161 + - CPU usage, memory consumption, I/O statistics
162 + - File descriptors, page faults, parent-child relationships
163 + - Container-aware process tracking
164 +- **Network connections**: Active connections with:
165 + - Protocol details, states, addresses, ports
166 + - Performance metrics per connection
167 +- **Systemd services & units**: Service health and resource usage
168 +- **Mount points**: Filesystem usage, capacity, and inode statistics
169 +- **Block devices**: I/O performance, latency, and utilization
170 +- **Containers & VMs**: Resource usage across containerized workloads
171 +- **Network interfaces**: Traffic rates, packets, drops, link status
172 +- **Streaming status**: Real-time replication and ML synchronization
173 +
174 +### Logs Access (requires API key and claimed agent)
175 +- **systemd-journal**: Comprehensive log access including:
176 + - Local system logs, user logs, and namespaces
177 + - Remote system logs from connected nodes
178 + - Advanced filtering and search capabilities
179 + - Historical log data based on retention
180 +- **Windows events**: Query Windows event logs (on Windows systems)
181 +
182 +### Alerts and Monitoring
183 +- **Active alerts**: Currently raised warnings and critical alerts
184 + - Detailed alert information including values, timestamps, and context
185 + - Alert classification by type, component, and severity
186 +- **Alert history**: Complete alert state tracking
187 + - All states: critical, warning, clear, undefined, uninitialized
188 + - Alert transitions with timestamps and values
189 +- **Alert metadata**: Recipients, configurations, and thresholds
190 +
191 +### Available Metric Categories
192 +The integration provides access to all metrics categories collected by Netdata including:
193 +- Core system: CPU, memory, disks, network, processes
194 +- Containers: Docker, cgroups, systemd services
195 +- Databases: MySQL, PostgreSQL, Redis, MongoDB
196 +- Web servers: Apache, Nginx, LiteSpeed
197 +- Hardware: IPMI, GPUs, temperature sensors, SMART
198 +- Network services: DNS, DHCP, VPN, firewalls
199 +- Applications: Custom StatsD metrics, logs-based metrics
200 +- And any other metrics collected by your Netdata installation
201 +
202 +## Security Considerations
203 +
204 +- The MCP integration currently provides **read-only** access to Netdata
205 +- Dynamic configuration is not exposed - AI assistants cannot read or modify Netdata settings
206 +- API key is required for accessing sensitive functions (logs and live data)
207 +- For production use, ensure your Netdata agent is claimed to Netdata Cloud
208 +
209 +## Usage Examples
210 +
211 +Once configured, you can ask questions like:
212 +
213 +### Infrastructure Overview
214 +- "Show me all connected nodes and their status"
215 +- "What metrics are available for my database servers?"
216 +- "Provide an observability coverage report for my infrastructure"
217 +- "Which nodes are offline or having connection issues?"
218 +
219 +### Performance Analysis
220 +- "What are the top CPU-consuming processes across all my servers?"
221 +- "Show me network interface utilization across all nodes"
222 +- "What's the memory usage trend for my database servers?"
223 +- "List all block devices and their I/O performance"
224 +- "Which of my nodes have disk backlog issues?"
225 +- "Show me container resource usage statistics"
226 +
227 +### Anomaly Detection and Troubleshooting
228 +- "Which metrics are showing anomalous behavior in the last hour?"
229 +- "Find metrics that changed significantly during the outage at 2 PM"
230 +- "What are the most unstable metrics in my infrastructure?"
231 +- "Analyze the correlation between disk I/O and application response time"
232 +
233 +### Alerts and Monitoring
234 +- "Are there any critical alerts currently active?"
235 +- "Show me all alert transitions in the last 24 hours"
236 +- "Which systems have disk space warnings?"
237 +- "What alerts fired and cleared during the night?"
238 +
239 +### System Logs and Events
240 +- "Show me systemd journal logs for failed services"
241 +- "Search for authentication failures in the last hour"
242 +- "Display kernel errors from all nodes"
243 +- "Find all logs related to out-of-memory conditions"
244 +
245 +### Live System State
246 +- "List all systemd services and their status"
247 +- "Show me active network connections on the web servers"
248 +- "What's the current streaming replication status?"
249 +- "Display mount points with low available space"
250 +
251 +## Troubleshooting
252 +
253 +1. **Connection refused**: Ensure Netdata is running and accessible at the specified URL
254 +2. **Bridge not found**: Verify the bridge path is correct and dependencies are installed
255 +3. **Authentication errors**: Verify the API key is correct and the agent is claimed
256 +4. **Missing data**: Check that the Netdata agent has the required collectors enabled
257 +5. **Limited access**: Without API key or unclaimed agent, functions and logs won't be available
258 +
259 +## FAQ
260 +
261 +- **Q: Can I use MCP with other AI assistants?**
262 + - A: Yes, MCP supports multiple AI assistants. Check the [MCP documentation](https://modelcontextprotocol.io/clients) for a full list.
263 +
264 +- **Q: Do I need to run a bridge on my local machine?**
265 +- A: Yes, the bridge converts `stdio` communication to `WebSocket` for remote access to Netdata. The bridge is run on your local machine (personal computer) to connect to the Netdata instance.
266 +
267 +- **Q: How do I find my API key?**
268 + - A: The API key is automatically generated by Netdata and stored in `/var/lib/netdata/mcp_dev_preview_api_key` or `/opt/netdata/var/lib/netdata/mcp_dev_preview_api_key` on the Netdata Agent you will connect to. Use `sudo cat` to view it.
269 +
270 +- **Q: Can I use MCP with Netdata Cloud?**
271 + - A: Yes, once MCP is integrated into Netdata Cloud, you will be able to use it with web-based AI assistants without needing a bridge.
272 +
273 +- **Q: What data can I access with MCP?**
274 + - A: You can access metrics, logs, alerts, live system information (processes, services, containers, network connections), and more.
275 +
276 +- **Q: Can I use MCP with my existing Netdata installation?**
277 + - A: Yes, as long as you have a recent version of Netdata installed, you can use the MCP integration without any additional installation.
278 +
279 +- **Q: Is MCP secure?**
280 + - A: Yes, MCP currently provides read-only access. Sensitive functions like logs and live system information require an API key, and the agent should be claimed to Netdata Cloud for production use.
281 +
282 +- **Q: Will my observability data be exposed to AI companies?**
283 + - A: Yes, but it depends on the AI assistant you use and the subscription you have. For example, Claude promises that your data will not be used to train their models for certain subscriptions, and Cursor allows you to use multiple AI assistants. Always check the privacy policies of the AI assistant you choose.
284 +
285 +- **Q: Are the responses of AI assistants accurate?**
286 + - A: AI assistants like Claude are designed to provide accurate and relevant responses based on the data they have access to. However, they may not always be perfect, or they may have not checked all the aspects before giving answers. It's important to verify critical information.
287 +
288 +## Best Practices
289 +
290 +### AI Assistants sampling data
291 +
292 +Sometimes, when you ask generic questions about your infrastructure, AI assistants do a simple sampling on a few nodes of the infrastructure, instead of querying all nodes. In Netdata we have provided the tools to properly do that, but AI assistants may not use them.
293 +
294 +Examples:
295 +
296 +Q: "which are the top processes/containers/VMs/services running on my servers?"
297 +
298 +The AI assistant may respond with a list of processes/containers/VMs/services from a few nodes, instead of querying all nodes.
299 +
300 +The proper way in Netdata is to query:
301 +
302 + - `app.*` charts/contexts for `processes`, which will return the processes running on all nodes grouped by category.
303 + - `systemd.*` to get the services running on all nodes.
304 + - `cgroup.*` to get the all the containers and VMs on all nodes.
305 +
306 +For all such queries, Netdata responses return cardinality information (much like the NIDL charts on your Netdata dashboard), so the AI assistant could get a much better picture instead of sampling data. When you notice that, you could ask the AI assistant to find the answer using more generic queries.
307 +
308 +### AI Assistants missing newer Netdata features
309 +
310 +Sometimes you ask AI assistants about features that have been recently added to Netdata (eg logs, or windows capabilities), and the AI assistant instead of checking what is available via their MCP connection, they say that Netdata does not support that feature. Answering "check your MCP tools, features, functions" is usually enough for the AI assistant to check the available features and start using them.
311 +
312 +### AI Assistants not using MCP at all
313 +
314 +Sometimes you need instruct them to use their MCP connection. So instead of saying "check the performance of my production db", you can say "use netdata to check the performance of my production db". This way, the AI assistant will use its MCP connection to query the Netdata instance and provide you with the relevant information.
315 +
316 +### Use AI Assistants to do your DevOps/SRE/SysAdmin "laundry"
317 +
318 +Our advice is to use AI assistants to do "your laundry": Give them specific tasks, check the queries they did to get that information, and when possible ask them to cross-check their answers using a different tool/source. AI assistants usually rush to make conclusions, so **challenge them** and they will go deeper and correct themselves. Remember that you always need to verify their answers, especially for critical tasks.
319 +
320 +### Multiple Netdata MCP servers for a single AI assistant
321 +
322 +If you need to configure multiple MCP servers, you can add them under the `mcpServers` section with different names. Example:
323 +
324 +```json
325 +{
326 + "mcpServers": {
327 + "netdata-production": {
328 + "command": "/usr/bin/nd-mcp",
329 + "args": [
330 + "ws://IP_OF_YOUR_NETDATA:19999/mcp?api_key=YOUR_API_KEY"
331 + ]
332 + },
333 + "netdata-testing": {
334 + "command": "/usr/bin/nd-mcp",
335 + "args": [
336 + "ws://IP_OF_YOUR_NETDATA:19999/mcp?api_key=YOUR_API_KEY"
337 + ]
338 + }
339 + }
340 +}
341 +```
342 +
343 +However, when multiple netdata MCP servers are configured, all AI assistants have difficulties to determine which one to use:
344 +
345 +- **Claude Desktop**: it seems there is no way to instruct it to use the right one. There is an enable/disable toggle for each MCP server, which however does not work properly. So, it is best to configure only one MCP server at a time.
346 +- **Cursor**: Similarly, it is impossible to instruct it to use the right one. However, there is a toggle and it works properly, but you still need to ensure that the Netdata server you want to use is the only one enabled.
347 +- **Claude Code**: this project has a different philosophy: you can have a different `.mcp.json` file in each project directory (the current directory from which you run it), so you can have different configurations for each project/directory. Since **Claude Code** also supports a `Claude.md` file with default instructions to the AI assistant, you can have different directories with different instructions and configurations, so you can use multiple Netdata MCP servers by spawning multiple Claude Code instances in different directories.
348 +
349 +For more information about Netdata, visit [netdata.cloud](https://netdata.cloud)
src/web/mcp/TODO-LIST.md new
+418
@@ -0,0 +1,418 @@
1 +# MCP Implementation Plan
2 +
3 +## Overview
4 +
5 +This document outlines the complete plan for implementing the Model Context Protocol (MCP) system with clean separation between transport and business logic, supporting both HTTP and WebSocket transports.
6 +
7 +## Architecture Overview
8 +
9 +### Core Design Principles
10 +1. **Transport-agnostic MCP core** - Business logic separated from transport protocols
11 +2. **Registry-based tool system** - Single source of truth for all MCP tools
12 +3. **Netdata-compatible authorization** - Reuse existing HTTP_ACL and HTTP_ACCESS system
13 +4. **Multi-buffer responses** - Support ordered responses using libnetdata double-linked lists
14 +5. **Clean job-based execution** - Each request becomes a structured job
15 +
16 +## 1. Core MCP Architecture Refactoring
17 +
18 +### A. Job-Based Request Processing
19 +
20 +#### MCP_REQ_JOB Structure (Transport-Agnostic)
21 +```c
22 +typedef struct mcp_req_job {
23 + // Job identification
24 + nd_uuid_t job_id;
25 + char job_id_str[UUID_STR_LEN];
26 +
27 + // Request data (pure, no transport context)
28 + struct json_object *params; // Parsed JSON parameters
29 + const char *tool_name; // Tool to execute
30 + USER_AUTH *auth; // Authentication context
31 +
32 + // Response data (ordered list using libnetdata double-linked lists)
33 + MCP_RESPONSE_BUFFER *response_buffers; // Head of double-linked list
34 +
35 + // Status and metadata
36 + int status_code; // Overall job status
37 + const char *error_message; // Error description if failed
38 + bool completed; // Job completion status
39 + usec_t created_usec; // Creation timestamp
40 + usec_t completed_usec; // Completion timestamp
41 +
42 + // Pagination support
43 + const char *next_cursor; // For paginated responses
44 +} MCP_REQ_JOB;
45 +```
46 +
47 +#### MCP_RESPONSE_BUFFER Structure (Using BUFFER's Built-in HTTP Metadata)
48 +```c
49 +typedef struct mcp_response_buffer {
50 + BUFFER *buffer; // Uses existing BUFFER with HTTP metadata built-in
51 + const char *response_type; // "text", "error", "data", etc.
52 +
53 + // Double-linked list support using libnetdata macros
54 + struct mcp_response_buffer *prev;
55 + struct mcp_response_buffer *next;
56 +} MCP_RESPONSE_BUFFER;
57 +```
58 +
59 +### B. Adapter-Specific Job Structures
60 +
61 +#### HTTP Adapter Job
62 +```c
63 +typedef struct mcp_http_adapter_job {
64 + MCP_REQ_JOB req; // Core MCP job (no MCP_CLIENT field!)
65 +
66 + // HTTP-specific data
67 + struct web_client *web_client; // HTTP client context
68 + const char *url_path; // Original URL path
69 + const char *query_string; // URL query parameters
70 +} MCP_HTTP_ADAPTER_JOB;
71 +```
72 +
73 +#### WebSocket/JSON-RPC Adapter Job
74 +```c
75 +typedef struct mcp_jsonrpc_adapter_job {
76 + MCP_REQ_JOB req; // Core MCP job
77 +
78 + // JSON-RPC/WebSocket specific data
79 + MCP_CLIENT *mcpc; // WebSocket client context (adapter manages this)
80 + uint64_t jsonrpc_id; // JSON-RPC request ID
81 + const char *jsonrpc_method; // Original method name
82 +} MCP_JSONRPC_ADAPTER_JOB;
83 +```
84 +
85 +**Status**:
86 +- [ ] Implement MCP_REQ_JOB structure
87 +- [ ] Implement MCP_RESPONSE_BUFFER with libnetdata double-linked list support
88 +- [ ] Create adapter job structures
89 +- [ ] Implement job lifecycle management functions
90 +
91 +## 2. Registry-Based Tool System
92 +
93 +### A. Tool Registry Structure (Following web_api_command Pattern)
94 +
95 +```c
96 +typedef struct mcp_tool_registry_entry {
97 + // Tool identification (similar to web_api_command)
98 + const char *name; // Tool name (e.g., "execute_function")
99 + uint32_t hash; // Hash for fast lookup (like api_commands_v3)
100 +
101 + // Authorization (following Netdata pattern exactly)
102 + HTTP_ACL acl; // ACL requirements (e.g., HTTP_ACL_FUNCTIONS)
103 + HTTP_ACCESS access; // Access level requirements
104 +
105 + // Execution
106 + int (*execute)(MCP_REQ_JOB *job); // Function pointer (similar to callback)
107 +
108 + // MCP-specific metadata
109 + MCP_NAMESPACE namespace; // Which MCP namespace
110 + const char *title; // Human-readable title
111 + const char *description; // Tool description
112 + const char *input_schema_json; // JSON schema for parameters (static string)
113 +
114 + // Feature flags
115 + bool supports_pagination; // Whether tool supports cursor pagination
116 + bool supports_streaming; // Whether tool supports streaming responses (future)
117 +
118 + // Caching hints for adapters
119 + bool cacheable_schema; // Whether schema responses can be cached
120 + time_t schema_cache_duration; // How long to cache schema responses
121 +} MCP_TOOL_REGISTRY_ENTRY;
122 +```
123 +
124 +### B. Registry Implementation
125 +
126 +#### Global Static Registry (Following api_commands_v3 Pattern)
127 +```c
128 +MCP_TOOL_REGISTRY_ENTRY mcp_tools_registry[] = {
129 + // Function execution tools
130 + {
131 + .name = "execute_function",
132 + .hash = 0, // Will be calculated on init like api_commands_v3
133 + .acl = HTTP_ACL_FUNCTIONS, // Same as existing function APIs
134 + .access = HTTP_ACCESS_ANONYMOUS_DATA, // Same as api_v1_function
135 + .execute = mcp_tool_execute_function,
136 + .namespace = MCP_NAMESPACE_TOOLS,
137 + .title = "Execute Netdata Function",
138 + .description = "Execute live data collection functions on nodes",
139 + .input_schema_json = MCP_EXECUTE_FUNCTION_SCHEMA_JSON,
140 + .supports_pagination = true,
141 + .supports_streaming = false,
142 + .cacheable_schema = true,
143 + .schema_cache_duration = 3600,
144 + },
145 + // ... more tools
146 +
147 + // Terminator (like api_commands_v3)
148 + { .name = NULL }
149 +};
150 +```
151 +
152 +#### Registry Access Functions
153 +```c
154 +// Initialize registry (calculate hashes like web_client_api_request_v3)
155 +void mcp_tools_registry_init(void);
156 +
157 +// Tool lookup (similar to web_client_api_request_vX)
158 +const MCP_TOOL_REGISTRY_ENTRY *mcp_find_tool(const char *tool_name);
159 +
160 +// Get tools by namespace
161 +const MCP_TOOL_REGISTRY_ENTRY **mcp_get_tools_by_namespace(MCP_NAMESPACE namespace, size_t *count);
162 +```
163 +
164 +**Status**:
165 +- [ ] Define MCP_TOOL_REGISTRY_ENTRY structure
166 +- [ ] Implement static registry array with all current tools
167 +- [ ] Implement registry initialization and lookup functions
168 +- [ ] Add authorization checking using HTTP_ACL/HTTP_ACCESS
169 +
170 +## 3. Transport Adapters
171 +
172 +### A. HTTP Adapter (Integrated with Netdata Web Server)
173 +
174 +#### HTTP Route Registration
175 +```c
176 +// HTTP adapter decides its own URL structure
177 +int mcp_http_adapter_init_routes(void) {
178 + // Direct tool execution endpoints
179 + web_client_api_request_v3_register("/api/v3/mcp/execute_function", mcp_http_handle_execute_function);
180 + web_client_api_request_v3_register("/api/v3/mcp/query_metrics", mcp_http_handle_query_metrics);
181 +
182 + // Generic endpoints using registry
183 + web_client_api_request_v3_register("/api/v3/mcp/tools", mcp_http_handle_tools_list);
184 + web_client_api_request_v3_register("/api/v3/mcp/tools/*/call", mcp_http_handle_tool_call);
185 + web_client_api_request_v3_register("/api/v3/mcp/tools/*/schema", mcp_http_handle_tool_schema);
186 +
187 + return 0;
188 +}
189 +```
190 +
191 +#### Authorization Integration (Following Netdata Pattern Exactly)
192 +```c
193 +// Generic tool execution using registry (like web_client_api_request_vX)
194 +int mcp_http_handle_tool_call(RRDHOST *host, struct web_client *w, char *url) {
195 + const char *tool_name = extract_tool_name_from_url(url);
196 +
197 + // Look up in registry
198 + const MCP_TOOL_REGISTRY_ENTRY *tool = mcp_find_tool(tool_name);
199 + if (!tool) {
200 + return web_client_api_request_v1_info_fill_buffer(host, w, "Tool not found");
201 + }
202 +
203 + // Check ACL and access (following Netdata pattern exactly)
204 + if(tool->acl != HTTP_ACL_NOCHECK) {
205 + if(!(w->acl & tool->acl)) {
206 + web_client_permission_denied_acl(w);
207 + return HTTP_RESP_FORBIDDEN;
208 + }
209 +
210 + if(tool->access != HTTP_ACCESS_NONE) {
211 + if(!web_client_can_access_with_auth(w, tool->access)) {
212 + web_client_permission_denied_access(w, tool->access);
213 + return HTTP_ACCESS_PERMISSION_DENIED_HTTP_CODE(tool->access);
214 + }
215 + }
216 + }
217 +
218 + // Execute tool
219 + // ... implementation
220 +}
221 +```
222 +
223 +**Status**:
224 +- [ ] Implement HTTP route registration
225 +- [ ] Implement HTTP request parsing (JSON body to params)
226 +- [ ] Implement HTTP response conversion (BUFFER list to HTTP JSON)
227 +- [ ] Integrate with existing Netdata authorization system
228 +- [ ] Add HTTP-specific error handling
229 +
230 +### B. WebSocket/JSON-RPC Adapter (Manages MCP_CLIENT)
231 +
232 +#### Adapter Responsibilities
233 +- **MCP_CLIENT management** (WebSocket connections, stdio pipes)
234 +- **JSON-RPC protocol** wrapping/unwrapping
235 +- **Ping/pong handling** - MCP core not involved
236 +- **Client notifications**
237 +- **Connection lifecycle**
238 +
239 +#### JSON-RPC Implementation
240 +```c
241 +void mcp_jsonrpc_handle_tools_call(MCP_CLIENT *mcpc, struct json_object *request) {
242 + struct json_object *params = json_object_object_get(request, "params");
243 + const char *tool_name = json_object_get_string(json_object_object_get(params, "name"));
244 +
245 + // Look up in registry (same as HTTP)
246 + const MCP_TOOL_REGISTRY_ENTRY *tool = mcp_find_tool(tool_name);
247 + if (!tool) {
248 + mcp_jsonrpc_send_error(mcpc, extract_id_from_request(request), -32601, "Tool not found");
249 + return;
250 + }
251 +
252 + // Check ACL and access (same authorization as HTTP!)
253 + // ... authorization code
254 +
255 + // Execute (same execution path as HTTP)
256 + // ... execution code
257 +}
258 +
259 +// Adapter handles ping separately
260 +void mcp_jsonrpc_handle_ping(MCP_CLIENT *mcpc, struct json_object *request) {
261 + // Adapter handles ping/pong - MCP core not involved
262 + uint64_t id = extract_id_from_request(request);
263 + mcp_jsonrpc_send_pong_response(mcpc, id);
264 +}
265 +```
266 +
267 +**Status**:
268 +- [ ] Extract JSON-RPC code from current MCP implementation
269 +- [ ] Move MCP_CLIENT management to WebSocket adapter
270 +- [ ] Implement JSON-RPC request/response conversion
271 +- [ ] Handle ping/pong and notifications in adapter
272 +- [ ] Ensure same authorization as HTTP adapter
273 +
274 +## 4. Core MCP Interface (Transport-Agnostic)
275 +
276 +### A. Core Execution Function
277 +```c
278 +// Main MCP execution function - completely transport agnostic
279 +int mcp_execute_tool(MCP_REQ_JOB *job);
280 +
281 +// Response buffer management using libnetdata double-linked list macros
282 +MCP_RESPONSE_BUFFER *mcp_job_add_response_buffer(MCP_REQ_JOB *job, BUFFER *buffer, const char *response_type);
283 +void mcp_job_prepend_response_buffer(MCP_REQ_JOB *job, MCP_RESPONSE_BUFFER *item);
284 +void mcp_job_append_response_buffer(MCP_REQ_JOB *job, MCP_RESPONSE_BUFFER *item);
285 +
286 +// Helper functions for common response types
287 +MCP_RESPONSE_BUFFER *mcp_job_add_text_response(MCP_REQ_JOB *job, const char *text, int http_status, HTTP_CONTENT_TYPE content_type);
288 +MCP_RESPONSE_BUFFER *mcp_job_add_json_response(MCP_REQ_JOB *job, BUFFER *json_buffer);
289 +MCP_RESPONSE_BUFFER *mcp_job_add_error_response(MCP_REQ_JOB *job, const char *error_msg, int http_status);
290 +```
291 +
292 +### B. Tool Implementation Interface
293 +```c
294 +// Tool function signature - no client context needed
295 +typedef int (*mcp_tool_func_t)(MCP_REQ_JOB *job);
296 +
297 +// Example simplified tool implementation
298 +int mcp_tool_execute_function(MCP_REQ_JOB *job) {
299 + // Pure business logic - no transport awareness
300 + const char *node = json_object_get_string(json_object_object_get(job->params, "node"));
301 +
302 + if (!node) {
303 + mcp_job_add_error_response(job, "Missing required parameter: node", 400);
304 + return -1;
305 + }
306 +
307 + // Execute function using existing BUFFER with built-in HTTP metadata
308 + BUFFER *result = buffer_create(0, NULL);
309 + buffer_set_content_type(result, CT_APPLICATION_JSON);
310 + buffer_cacheable(result);
311 + result->expires = now_monotonic_usec() + 300 * USEC_PER_SEC;
312 +
313 + int status = execute_netdata_function(node, function, result);
314 +
315 + if (status == 0) {
316 + mcp_job_add_response_buffer(job, result, "text");
317 + } else {
318 + mcp_job_add_error_response(job, "Function execution failed", 500);
319 + buffer_free(result);
320 + }
321 +
322 + return status;
323 +}
324 +```
325 +
326 +**Status**:
327 +- [ ] Implement core execution function
328 +- [ ] Implement response buffer management with double-linked lists
329 +- [ ] Update all existing tools to use new job interface
330 +- [ ] Remove transport dependencies from tool implementations
331 +
332 +## 5. Directory Structure
333 +
334 +```
335 +src/web/mcp/
336 +├── core/
337 +│ ├── mcp-registry.c/h # Global tool registry
338 +│ ├── mcp-job.c/h # Job management
339 +│ ├── mcp-response.c/h # Response management
340 +│ ├── mcp-tools-*.c/h # Individual tool implementations
341 +│ └── mcp-core.c/h # Core execution (tool lookup + execute)
342 +├── adapters/
343 +│ ├── websocket/
344 +│ │ ├── mcp-jsonrpc-adapter.c/h # tools/list, tools/call implementation
345 +│ │ └── mcp-client.c/h # MCP_CLIENT management
346 +│ └── http/
347 +│ └── mcp-http-adapter.c/h # HTTP routes using registry
348 +├── schemas/
349 +│ ├── execute_function.json # Static schema definitions
350 +│ ├── query_metrics.json
351 +│ └── list_metrics.json
352 +└── mcp.h # Public interfaces
353 +```
354 +
355 +**Status**:
356 +- [ ] Create directory structure
357 +- [ ] Move existing code to appropriate locations
358 +- [ ] Update build system for new structure
359 +
360 +## 6. Logs Tools Implementation (Advanced Features)
361 +
362 +### Progressive Logs Discovery Workflow
363 +
364 +#### Specialized Logs Tools
365 +1. **`list_logs_sources`** - Discovery: "What logs are available?"
366 +2. **`list_logs_fields`** - Schema discovery: "What fields exist in this source?"
367 +3. **`list_logs_fields_values`** - Facets analysis: "What values can these fields have and how common are they?"
368 +4. **`query_logs`** - Actual log retrieval with all the power of the current system
369 +
370 +#### Implementation Strategy
371 +- **Tools 1-2**: Use the `info=true` calls to logs functions to get metadata
372 +- **Tool 3**: Call logs functions with specific facets parameters, return just the facets counts
373 +- **Tool 4**: Current full query capability
374 +
375 +**Status**:
376 +- [x] Design the 4-tool architecture
377 +- [ ] Implement `list_logs_sources`
378 +- [ ] Implement `list_logs_fields`
379 +- [ ] Implement `list_logs_fields_values` (the key facets tool)
380 +- [ ] Implement `query_logs`
381 +- [ ] Update `execute_function` to exclude logs functions
382 +- [ ] Test the complete workflow
383 +
384 +## 7. Implementation Phases
385 +
386 +### Phase 1: Core Infrastructure (Priority: High)
387 +1. **MCP_REQ_JOB and response buffer structures**
388 +2. **Registry system with authorization**
389 +3. **Core execution function**
390 +4. **Basic HTTP adapter**
391 +
392 +### Phase 2: Transport Separation (Priority: High)
393 +1. **Extract JSON-RPC from WebSocket adapter**
394 +2. **Update all existing tools to use job interface**
395 +3. **Implement multi-buffer response system**
396 +4. **Complete HTTP adapter with full feature parity**
397 +
398 +### Phase 3: Advanced Features (Priority: Medium)
399 +1. **Specialized logs tools**
400 +2. **Enhanced error handling and status reporting**
401 +3. **Performance optimizations**
402 +4. **Comprehensive testing**
403 +
404 +### Phase 4: Future Enhancements (Priority: Low)
405 +1. **Streaming support for long-running operations**
406 +2. **Additional MCP namespaces (resources, prompts)**
407 +3. **Advanced caching strategies**
408 +4. **Monitoring and metrics**
409 +
410 +## Benefits of This Architecture
411 +
412 +1. ✅ **Code Reuse**: All existing MCP tools work unchanged after refactoring
413 +2. ✅ **Consistency**: Same functionality via HTTP and WebSocket
414 +3. ✅ **Integration**: Native part of Netdata web server
415 +4. ✅ **Authorization**: Reuses existing HTTP_ACL/HTTP_ACCESS system
416 +5. ✅ **Maintenance**: Single codebase for all MCP logic
417 +6. ✅ **Performance**: No extra proxy/adapter process
418 +7. ✅ **Scalability**: Clean separation enables easy addition of new tools and transports
\ No newline at end of file
src/web/mcp/adapters/mcp-websocket.c
+48 -14
@@ -22,6 +22,9 @@ int mcp_websocket_send_buffer(struct websocket_server_client *wsc, BUFFER *buffe
22 const char *text = buffer_tostring(buffer);
23 if (!text || !*text) return -1;
24
25 + // Log the raw outgoing message
26 + netdata_log_debug(D_MCP, "SND: %s", text);
27 +
28 return websocket_protocol_send_text(wsc, text);
29 }
30
@@ -30,6 +33,10 @@ static MCP_CLIENT *mcp_websocket_create_context(struct websocket_server_client *
33 if (!wsc) return NULL;
34
35 MCP_CLIENT *ctx = mcp_create_client(MCP_TRANSPORT_WEBSOCKET, wsc);
36 + if (ctx) {
37 + // Set pointer to the websocket client's user_auth
38 + ctx->user_auth = &wsc->user_auth;
39 + }
40 mcp_websocket_set_context(wsc, ctx);
41
42 return ctx;
@@ -51,19 +58,28 @@ void mcp_websocket_on_connect(struct websocket_server_client *wsc) {
58
59 // WebSocket message handler for MCP - receives message and routes to MCP
60 void mcp_websocket_on_message(struct websocket_server_client *wsc, const char *message, size_t length, WEBSOCKET_OPCODE opcode) {
54 - if (!wsc || !message || length == 0) return;
61 + if (!wsc || !message || length == 0)
62 + return;
63 +
64 + // Log the raw incoming message
65 + netdata_log_debug(D_MCP, "RCV: %s", message);
66
67 // Only handle text messages
68 if (opcode != WS_OPCODE_TEXT) {
58 - websocket_debug(wsc, "Ignoring binary message");
69 + websocket_error(wsc, "Ignoring binary message - mcp supports only TEXT messages");
70 + return;
71 + }
72 +
73 + // Silently ignore standalone "PING" messages (legacy MCP client behavior)
74 + if (length == 4 && strncmp(message, "PING", 4) == 0) {
75 + websocket_debug(wsc, "Ignoring legacy PING message");
76 return;
77 }
78
79 // Get the MCP context
63 - MCP_CLIENT *ctx = mcp_websocket_get_context(wsc);
64 - if (!ctx) {
65 - websocket_debug(wsc, "MCP context not found");
66 - websocket_protocol_send_close(wsc, WS_CLOSE_INTERNAL_ERROR, "MCP context not found");
80 + MCP_CLIENT *mcpc = mcp_websocket_get_context(wsc);
81 + if (!mcpc) {
82 + websocket_error(wsc, "MCP context not found");
83 return;
84 }
85
@@ -73,15 +89,35 @@ void mcp_websocket_on_message(struct websocket_server_client *wsc, const char *m
89 request = json_tokener_parse_verbose(message, &jerr);
90
91 if (!request || jerr != json_tokener_success) {
76 - websocket_debug(wsc, "Failed to parse JSON-RPC request: %s", json_tokener_error_desc(jerr));
77 - CLEAN_BUFFER *b = buffer_create(0, NULL);
78 - mcp_jsonrpc_error(b, NULL, 0, -32700);
79 - mcp_websocket_send_buffer(wsc, b);
92 + // Log the full error with payload for debugging
93 + websocket_error(wsc, "Failed to parse JSON-RPC request: %s | Payload (length=%zu): '%.*s'",
94 + json_tokener_error_desc(jerr),
95 + length,
96 + (int)(length > 1000 ? 1000 : length), // Limit to 1000 chars in log
97 + message);
98 +
99 + // Also log the hex dump of first few bytes to catch non-printable characters
100 + if (length > 0) {
101 + char hex_dump[256];
102 + size_t hex_len = 0;
103 + size_t bytes_to_dump = (length > 32) ? 32 : length;
104 +
105 + for (size_t i = 0; i < bytes_to_dump && hex_len < sizeof(hex_dump) - 6; i++) {
106 + hex_len += snprintf(hex_dump + hex_len, sizeof(hex_dump) - hex_len,
107 + "%02X ", (unsigned char)message[i]);
108 + }
109 + if (bytes_to_dump < length) {
110 + hex_len += snprintf(hex_dump + hex_len, sizeof(hex_dump) - hex_len, "...");
111 + }
112 +
113 + websocket_error(wsc, "First %zu bytes hex dump: %s", bytes_to_dump, hex_dump);
114 + }
115 +
116 return;
117 }
118
119 // Pass the request to the MCP handler
84 - mcp_handle_request(ctx, request);
120 + mcp_handle_request(mcpc, request);
121
122 // Free the request object
123 json_object_put(request);
@@ -117,8 +153,6 @@ void mcp_websocket_on_disconnect(struct websocket_server_client *wsc) {
153
154 // Register WebSocket callbacks for MCP
155 void mcp_websocket_adapter_initialize(void) {
120 - // Initialize the MCP subsystem
156 mcp_initialize_subsystem();
122 -
157 netdata_log_info("MCP WebSocket adapter initialized");
124 -}
\ No newline at end of file
158 +}
src/web/mcp/adapters/mcp-websocket.h
-4
@@ -23,8 +23,4 @@ int mcp_websocket_send_buffer(struct websocket_server_client *wsc, BUFFER *buffe
23 MCP_CLIENT *mcp_websocket_get_context(struct websocket_server_client *wsc);
24 void mcp_websocket_set_context(struct websocket_server_client *wsc, MCP_CLIENT *ctx);
25
26 -// Convenience wrappers for sending responses
27 -void mcp_websocket_send_error_response(struct websocket_server_client *wsc, int code, const char *message, uint64_t id);
28 -void mcp_websocket_send_success_response(struct websocket_server_client *wsc, struct json_object *result, uint64_t id);
29 -
26 #endif // NETDATA_MCP_ADAPTER_WEBSOCKET_H
\ No newline at end of file
src/web/mcp/bridges/README.md new
+59
@@ -0,0 +1,59 @@
1 +# Netdata MCP Bridges
2 +
3 +This directory contains bridge programs that convert MCP stdio communication to Netdata's MCP over WebSocket. These bridges allow you to use standard I/O based AI models and tools with Netdata's Model Context Protocol.
4 +
5 +## Available Bridges
6 +
7 +Three implementations are provided to accommodate different environments, each in its own subdirectory:
8 +
9 +- [`stdio-python/`](stdio-python/) - Python implementation
10 +- [`stdio-nodejs/`](stdio-nodejs/) - JavaScript (Node.js) implementation
11 +- [`stdio-golang/`](stdio-golang/) - Go implementation
12 +
13 +Each subdirectory contains:
14 +- The bridge implementation
15 +- A README.md with detailed installation and usage instructions specific to that implementation
16 +
17 +## Common Usage
18 +
19 +All bridges follow the same usage pattern, accepting a WebSocket URL as their only parameter:
20 +
21 +```
22 +<bridge-program> ws://<ip>:19999/mcp
23 +```
24 +
25 +Where `<ip>` is either `localhost` or the IP address where a Netdata instance is listening.
26 +
27 +The bridge reads from standard input and writes to standard output, allowing you to pipe AI model output through it to connect with Netdata MCP.
28 +
29 +## Claude Desktop Configuration
30 +
31 +To use Claude Desktop with Netdata's MCP, you can configure it to use one of these bridges. Here's how to set it up:
32 +
33 +1. Select and install one of the bridge implementations from the subdirectories
34 +2. In Claude Desktop settings, configure the Custom Command option to point to your chosen bridge
35 +3. If your Netdata instance is running on a different machine, replace `localhost` with the appropriate IP address
36 +
37 +This configuration allows Claude Desktop to communicate directly with your Netdata instance, providing it with contextual information from your monitoring environment.
38 +
39 +For detailed instructions specific to each implementation, refer to the README.md in the corresponding subdirectory.
40 +
41 +## Connection Reliability
42 +
43 +All bridge implementations include robust connection handling features:
44 +
45 +- **Automatic Reconnection**: If the WebSocket connection is lost for any reason, the bridges will automatically attempt to reconnect
46 +- **Exponential Backoff**: Reconnection attempts use exponential backoff with jitter to avoid overwhelming the server
47 +- **Message Queuing**: Messages sent while disconnected are queued and delivered once reconnected
48 +- **Connection Status Logging**: All bridges log connection status to stderr for monitoring
49 +
50 +These features ensure that temporary network issues or Netdata server restarts don't disrupt your AI model's integration, providing a resilient connection that automatically recovers from failures.
51 +
52 +## Protocol Compatibility
53 +
54 +Netdata MCP implements the JSON-RPC 2.0 protocol for communication. The bridges pass messages directly between the AI model's standard I/O and the WebSocket connection without modification.
55 +
56 +Important compatibility notes:
57 +- Netdata MCP expects JSON-RPC 2.0 compatible messages
58 +- Messages that don't conform to the JSON-RPC 2.0 format are silently ignored
59 +- Ensure your AI model or tool produces properly formatted JSON-RPC 2.0 messages for full compatibility
src/web/mcp/bridges/stdio-golang/.gitignore new
+19
@@ -0,0 +1,19 @@
1 +# Binary
2 +nd-mcp
3 +
4 +# Build cache and directories
5 +/bin/
6 +/pkg/
7 +
8 +# Debug files
9 +*.debug
10 +*.test
11 +debug
12 +debug.test
13 +
14 +# IDE specific files
15 +.idea/
16 +.vscode/
17 +*.swp
18 +*.swo
19 +*~
\ No newline at end of file
src/web/mcp/bridges/stdio-golang/README.md new
+97
@@ -0,0 +1,97 @@
1 +# Netdata MCP Bridge - Go Implementation
2 +
3 +This Go bridge converts MCP stdio communication to Netdata's MCP over WebSocket.
4 +
5 +## Requirements
6 +
7 +- Go 1.16+
8 +- nhooyr.io/websocket package
9 +
10 +## Installation
11 +
12 +The easiest way to build the Go bridge is to use the included build script:
13 +
14 +```bash
15 +# Make the build script executable (if needed)
16 +chmod +x build.sh
17 +
18 +# Run the build script
19 +./build.sh
20 +```
21 +
22 +The script will:
23 +1. Check if Go is installed
24 +2. Initialize a Go module if needed
25 +3. Add required dependencies
26 +4. Build the binary as `nd-mcp`
27 +
28 +Alternatively, you can build it manually:
29 +
30 +```bash
31 +# Initialize a new module in this directory (only needed once)
32 +go mod init netdata/nd-mcp-bridge
33 +
34 +# Add dependencies
35 +go get nhooyr.io/websocket
36 +go mod tidy
37 +
38 +# Build the binary
39 +go build -o nd-mcp nd-mcp.go
40 +```
41 +
42 +## Usage
43 +
44 +```bash
45 +./nd-mcp ws://<ip>:19999/mcp
46 +```
47 +
48 +Where `<ip>` is either `localhost` or the IP address where a Netdata instance is listening.
49 +
50 +## Example with Claude Desktop
51 +
52 +To use this bridge with Claude Desktop:
53 +
54 +1. In Claude Desktop settings, configure the Custom Command option:
55 +
56 +```bash
57 +/path/to/stdio-golang/nd-mcp ws://localhost:19999/mcp
58 +```
59 +
60 +2. If your Netdata instance is running on a different machine, replace `localhost` with the appropriate IP address.
61 +
62 +## How It Works
63 +
64 +The bridge:
65 +1. Establishes a WebSocket connection to the specified Netdata MCP endpoint
66 +2. Reads from standard input and sends to the WebSocket
67 +3. Receives messages from the WebSocket and writes to standard output
68 +4. Handles both directions simultaneously
69 +5. Automatically reconnects if the connection is lost, with exponential backoff
70 +
71 +## Connection Reliability
72 +
73 +This bridge implements robust connection handling:
74 +
75 +- **Automatic Reconnection**: If the WebSocket connection is lost, the bridge will automatically attempt to reconnect
76 +- **Exponential Backoff**: Reconnection attempts use exponential backoff with jitter to avoid overwhelming the server
77 +- **Message Queuing**: Messages sent while disconnected are queued and delivered once reconnected
78 +- **Connection Status Logging**: The bridge logs connection status to stderr for monitoring
79 +- **Thread-Safe Operation**: Uses goroutines and mutexes to ensure thread-safe operation
80 +
81 +The reconnection algorithm starts with a 1-second delay and doubles the wait time with each attempt, up to a maximum of 60 seconds. Random jitter is added to prevent connection storms from multiple clients reconnecting simultaneously.
82 +
83 +## Protocol Compatibility
84 +
85 +- Netdata MCP implements the JSON-RPC 2.0 protocol
86 +- Messages that don't conform to the JSON-RPC 2.0 format are silently ignored
87 +- The bridge passes messages directly without any modification
88 +
89 +## Implementation Notes
90 +
91 +This implementation:
92 +- Uses the nhooyr.io/websocket library for WebSocket communication
93 +- Explicitly sets WebSocket headers including Sec-WebSocket-Key and Sec-WebSocket-Version
94 +- Implements proper WebSocket handshake to ensure compatibility with Netdata's WebSocket server
95 +- Sends and receives raw text messages directly (without JSON serialization/deserialization)
96 +- Preserves the exact format of JSON-RPC 2.0 messages without adding extra quotes or escaping
97 +- Uses Go's concurrency model with channels for efficient message processing
\ No newline at end of file
src/web/mcp/bridges/stdio-golang/build.bat new
+66
@@ -0,0 +1,66 @@
1 +@echo off
2 +setlocal enabledelayedexpansion
3 +
4 +REM Get the directory where the batch file is located
5 +set "SCRIPT_DIR=%~dp0"
6 +cd /d "%SCRIPT_DIR%"
7 +
8 +echo Working directory: %SCRIPT_DIR%
9 +
10 +REM Check if Go is installed
11 +go version >nul 2>&1
12 +if !errorlevel! neq 0 (
13 + echo Error: Go is not installed or not in the PATH
14 + echo Please install Go from https://golang.org/doc/install
15 + echo.
16 + echo After installation, make sure to:
17 + echo 1. Restart your command prompt
18 + echo 2. Verify Go is in your PATH by running: go version
19 + pause
20 + exit /b 1
21 +)
22 +
23 +echo Go is installed:
24 +go version
25 +echo Creating Go module for Netdata MCP bridge...
26 +
27 +REM Initialize Go module if it doesn't exist
28 +if not exist "go.mod" (
29 + go mod init netdata/nd-mcp-bridge
30 + echo Initialized new Go module
31 +) else (
32 + echo Go module already exists
33 +)
34 +
35 +REM Add required dependencies
36 +echo Adding dependencies...
37 +go get nhooyr.io/websocket
38 +go mod tidy
39 +
40 +REM Build the binary
41 +echo Building nd-mcp.exe binary...
42 +go build -o nd-mcp.exe nd-mcp.go
43 +
44 +if !errorlevel! neq 0 (
45 + echo Build failed!
46 + pause
47 + exit /b 1
48 +)
49 +
50 +REM Get the full path to the executable
51 +set "EXECUTABLE_PATH=%SCRIPT_DIR%nd-mcp.exe"
52 +
53 +echo.
54 +echo Build complete!
55 +echo.
56 +echo You can now run the bridge using the full path:
57 +echo "%EXECUTABLE_PATH%" ws://ip:19999/mcp
58 +echo.
59 +echo Or from this directory:
60 +echo nd-mcp.exe ws://ip:19999/mcp
61 +echo.
62 +echo Example usage:
63 +echo nd-mcp.exe ws://localhost:19999/mcp
64 +echo nd-mcp.exe ws://192.168.1.100:19999/mcp
65 +echo.
66 +pause
\ No newline at end of file
src/web/mcp/bridges/stdio-golang/build.sh new
+49
@@ -0,0 +1,49 @@
1 +#!/bin/bash
2 +set -e
3 +
4 +# Get the directory where the script is located
5 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6 +cd "$SCRIPT_DIR"
7 +
8 +echo "Working directory: $SCRIPT_DIR"
9 +
10 +# Check if Go is installed
11 +if ! command -v go &> /dev/null; then
12 + echo "Error: Go is not installed or not in the PATH"
13 + echo "Please install Go from https://golang.org/doc/install"
14 + exit 1
15 +fi
16 +
17 +echo "Go is installed: $(go version)"
18 +echo "Creating Go module for Netdata MCP bridge..."
19 +
20 +# Initialize Go module if it doesn't exist
21 +if [ ! -f "go.mod" ]; then
22 + go mod init netdata/nd-mcp-bridge
23 + echo "Initialized new Go module"
24 +else
25 + echo "Go module already exists"
26 +fi
27 +
28 +# Add required dependencies
29 +echo "Adding dependencies..."
30 +go get nhooyr.io/websocket
31 +go mod tidy
32 +
33 +# Build the binary
34 +echo "Building nd-mcp binary..."
35 +go build -o nd-mcp nd-mcp.go
36 +
37 +# Make binary executable
38 +chmod +x nd-mcp
39 +
40 +# Get the full path to the executable
41 +EXECUTABLE_PATH="$SCRIPT_DIR/nd-mcp"
42 +
43 +echo "Build complete!"
44 +echo ""
45 +echo "You can now run the bridge using the full path:"
46 +echo "$EXECUTABLE_PATH ws://ip:19999/mcp"
47 +echo ""
48 +echo "Or from this directory:"
49 +echo "./nd-mcp ws://ip:19999/mcp"
\ No newline at end of file
src/web/mcp/bridges/stdio-golang/go.mod new
+5
@@ -0,0 +1,5 @@
1 +module netdata/nd-mcp-bridge
2 +
3 +go 1.24.0
4 +
5 +require nhooyr.io/websocket v1.8.17
src/web/mcp/bridges/stdio-golang/go.sum new
+2
@@ -0,0 +1,2 @@
1 +nhooyr.io/websocket v1.8.17 h1:KEVeLJkUywCKVsnLIDlD/5gtayKp8VoCkksHCGGfT9Y=
2 +nhooyr.io/websocket v1.8.17/go.mod h1:rN9OFWIUwuxg4fR5tELlYC04bXYowCP9GX47ivo2l+c=
src/web/mcp/bridges/stdio-golang/nd-mcp.go new
+616
@@ -0,0 +1,616 @@
1 +package main
2 +
3 +import (
4 + "bufio"
5 + "context"
6 + "crypto/rand"
7 + "encoding/base64"
8 + "encoding/json"
9 + "fmt"
10 + "log"
11 + "math"
12 + "math/big"
13 + "net/http"
14 + "os"
15 + "os/signal"
16 + "sync"
17 + "syscall"
18 + "time"
19 +
20 + "nhooyr.io/websocket"
21 +)
22 +
23 +// Connection states
24 +const (
25 + stateDisconnected = iota
26 + stateConnecting
27 + stateConnected
28 +)
29 +
30 +// WebSocket message types (these are defined in the websocket package but we need to define them here)
31 +const (
32 + MessageText = websocket.MessageText
33 + MessageBinary = websocket.MessageBinary
34 +)
35 +
36 +// JSON-RPC 2.0 related structures
37 +type JsonRpcMessage struct {
38 + JsonRpc string `json:"jsonrpc"`
39 + Id interface{} `json:"id,omitempty"`
40 + Method string `json:"method,omitempty"`
41 + Result interface{} `json:"result,omitempty"`
42 + Error *JsonRpcError `json:"error,omitempty"`
43 +}
44 +
45 +type JsonRpcError struct {
46 + Code int `json:"code"`
47 + Message string `json:"message"`
48 + Data interface{} `json:"data,omitempty"`
49 +}
50 +
51 +// ConnectionTimeout is the timeout for initial connection attempts
52 +const ConnectionTimeout = 5 * time.Second
53 +
54 +// PendingRequest represents a message waiting for the connection to be established
55 +type PendingRequest struct {
56 + ID interface{}
57 + Message string
58 + Timer *time.Timer
59 +}
60 +
61 +func generateWebSocketKey() string {
62 + key := make([]byte, 16)
63 + _, err := rand.Read(key)
64 + if err != nil {
65 + log.Fatalf("failed to generate WebSocket key: %v", err)
66 + }
67 + return base64.StdEncoding.EncodeToString(key)
68 +}
69 +
70 +func main() {
71 + // Get program name for logs
72 + programName := "nd-mcp-golang"
73 + if len(os.Args) > 0 {
74 + programName = os.Args[0]
75 + }
76 +
77 + if len(os.Args) != 2 {
78 + fmt.Fprintf(os.Stderr, "%s: Usage: %s ws://host/path\n", programName, programName)
79 + os.Exit(1)
80 + }
81 +
82 + // Set up channels for communication
83 + stdinCh := make(chan string, 100) // Buffer stdin messages
84 + reconnectCh := make(chan struct{}, 1) // Signal for immediate reconnection
85 + doneCh := make(chan struct{}) // Signal for program termination
86 + stdinClosedCh := make(chan struct{}) // Signal that stdin is closed
87 +
88 + // Global state
89 + var state int // Connection state
90 + var stateMu sync.Mutex
91 + var messageQueueMu sync.Mutex
92 + messageQueue := []string{}
93 + stdinActive := true
94 +
95 + // Pending requests that are waiting for connection to be established
96 + var pendingMu sync.Mutex
97 + pendingRequests := make(map[interface{}]*PendingRequest)
98 +
99 + // Parse a JSON-RPC message and extract the ID and method
100 + parseJsonRpcMessage := func(message string) (interface{}, string) {
101 + var msg JsonRpcMessage
102 + err := json.Unmarshal([]byte(message), &msg)
103 + if err != nil || msg.JsonRpc != "2.0" {
104 + return nil, ""
105 + }
106 + return msg.Id, msg.Method
107 + }
108 +
109 + // Create a JSON-RPC error response
110 + createJsonRpcError := func(id interface{}, code int, message string, data interface{}) string {
111 + response := JsonRpcMessage{
112 + JsonRpc: "2.0",
113 + Id: id,
114 + Error: &JsonRpcError{
115 + Code: code,
116 + Message: message,
117 + Data: data,
118 + },
119 + }
120 + responseJson, err := json.Marshal(response)
121 + if err != nil {
122 + fmt.Fprintf(os.Stderr, "%s: ERROR: Failed to marshal error response: %v\n", programName, err)
123 + return fmt.Sprintf("{\"jsonrpc\":\"2.0\",\"id\":%v,\"error\":{\"code\":%d,\"message\":\"%s\"}}", id, code, message)
124 + }
125 + return string(responseJson)
126 + }
127 +
128 + // Handle a request timeout for connection establishment ONLY
129 + handleRequestTimeout := func(msgId interface{}) {
130 + stateMu.Lock()
131 + currentState := state
132 + stateMu.Unlock()
133 +
134 + // This timeout ONLY applies if the connection is not established yet
135 + if currentState != stateConnected {
136 + pendingMu.Lock()
137 + if req, exists := pendingRequests[msgId]; exists {
138 + fmt.Fprintf(os.Stderr, "%s: Connection timeout for request ID %v, sending error response\n", programName, msgId)
139 +
140 + // Create and send error response
141 + errorResponse := createJsonRpcError(
142 + msgId,
143 + -32000, // Server error code
144 + "MCP server connection failed",
145 + map[string]string{"details": "Could not establish connection to Netdata within timeout period"},
146 + )
147 +
148 + fmt.Println(errorResponse)
149 +
150 + // Get the original message
151 + originalMessage := req.Message
152 +
153 + // Remove request from pending
154 + delete(pendingRequests, msgId)
155 + pendingMu.Unlock()
156 +
157 + // Also remove from messageQueue if it exists there
158 + messageQueueMu.Lock()
159 + for i, msg := range messageQueue {
160 + if msg == originalMessage {
161 + // Remove from queue by creating a new slice without this element
162 + messageQueue = append(messageQueue[:i], messageQueue[i+1:]...)
163 + fmt.Fprintf(os.Stderr, "%s: Removed timed-out request from message queue\n", programName)
164 + break
165 + }
166 + }
167 + messageQueueMu.Unlock()
168 + return
169 + }
170 + pendingMu.Unlock()
171 + } else {
172 + // The connection was established before the timeout - just clean up
173 + pendingMu.Lock()
174 + if _, exists := pendingRequests[msgId]; exists {
175 + delete(pendingRequests, msgId)
176 + }
177 + pendingMu.Unlock()
178 + }
179 + }
180 +
181 + // Set up signal handling
182 + sigCh := make(chan os.Signal, 1)
183 + signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
184 + go func() {
185 + sig := <-sigCh
186 + fmt.Fprintf(os.Stderr, "%s: Received signal %v, shutting down\n", programName, sig)
187 + close(doneCh)
188 + }()
189 +
190 + // Start reading from stdin in a separate goroutine
191 + go func() {
192 + scanner := bufio.NewScanner(os.Stdin)
193 + scannerRunning := true
194 +
195 + // Make scanner buffer larger to handle large messages
196 + const maxScanBufferSize = 1024 * 1024 // 1MB
197 + buf := make([]byte, maxScanBufferSize)
198 + scanner.Buffer(buf, maxScanBufferSize)
199 +
200 + for scannerRunning && scanner.Scan() {
201 + text := scanner.Text()
202 +
203 + // Parse for JSON-RPC ID
204 + msgId, _ := parseJsonRpcMessage(text)
205 +
206 + // Check connection state
207 + stateMu.Lock()
208 + currentState := state
209 + stateMu.Unlock()
210 +
211 + if currentState != stateConnected && msgId != nil {
212 + // Store as pending request with timeout
213 + pendingMu.Lock()
214 + req := &PendingRequest{
215 + ID: msgId,
216 + Message: text,
217 + Timer: time.AfterFunc(ConnectionTimeout, func() { handleRequestTimeout(msgId) }),
218 + }
219 + pendingRequests[msgId] = req
220 + pendingMu.Unlock()
221 +
222 + fmt.Fprintf(os.Stderr, "%s: Received request with ID %v, setting response timeout\n", programName, msgId)
223 + }
224 +
225 + // Queue the message
226 + stdinCh <- text
227 +
228 + // If we're not connected, trigger immediate reconnection
229 + if currentState != stateConnected {
230 + select {
231 + case reconnectCh <- struct{}{}:
232 + fmt.Fprintf(os.Stderr, "%s: Received stdin data, attempting immediate reconnection\n", programName)
233 + default:
234 + // Channel already has reconnection request, ignore
235 + }
236 + }
237 + }
238 +
239 + // Check for scanner error
240 + if err := scanner.Err(); err != nil {
241 + fmt.Fprintf(os.Stderr, "%s: ERROR: stdin read error: %v\n", programName, err)
242 + } else {
243 + fmt.Fprintf(os.Stderr, "%s: End of stdin\n", programName)
244 + }
245 +
246 + // Signal that stdin is closed
247 + stdinActive = false
248 + close(stdinClosedCh)
249 + }()
250 +
251 + // Reconnection parameters
252 + baseDelay := 1 * time.Second
253 + maxDelay := 60 * time.Second
254 + attempt := 0
255 +
256 + // Timer for reconnection backoff
257 + var timer *time.Timer
258 +
259 + // Main connection loop
260 + for {
261 + select {
262 + case <-doneCh:
263 + // Program termination requested
264 + return
265 + case <-stdinClosedCh:
266 + // Stdin closed, continue running until websocket disconnects
267 + // and then exit on the next reconnection attempt
268 + stdinClosedCh = nil // Prevent duplicate handling
269 + case <-reconnectCh:
270 + // Immediate reconnection requested (e.g. from stdin activity)
271 + if timer != nil {
272 + timer.Stop()
273 + }
274 +
275 + // Only proceed with immediate reconnection if we're not already connecting/connected
276 + stateMu.Lock()
277 + currentState := state
278 + stateMu.Unlock()
279 +
280 + if currentState == stateDisconnected {
281 + // Reset the reconnection timer
282 + attempt = 0
283 + // Fall through to connection attempt
284 + } else {
285 + // Already connecting or connected
286 + continue
287 + }
288 + default:
289 + // Calculate backoff delay with jitter
290 + if attempt > 0 {
291 + // Check if stdin is closed and we're disconnected - if so, exit
292 + if !stdinActive {
293 + fmt.Fprintf(os.Stderr, "%s: Stdin closed and disconnected, exiting\n", programName)
294 + return
295 + }
296 +
297 + delaySeconds := math.Min(float64(maxDelay.Seconds()),
298 + float64(baseDelay.Seconds())*math.Pow(2, float64(attempt-1))*(0.5+jitter()))
299 + delay := time.Duration(delaySeconds * float64(time.Second))
300 +
301 + fmt.Fprintf(os.Stderr, "%s: Reconnecting in %.1f seconds (attempt %d)...\n",
302 + programName, delaySeconds, attempt)
303 +
304 + // Create timer and wait for it to expire, or for signals
305 + timer = time.NewTimer(delay)
306 +
307 + select {
308 + case <-timer.C:
309 + // Timer expired, continue to connection attempt
310 + case <-reconnectCh:
311 + // Immediate reconnection requested
312 + timer.Stop()
313 + case <-doneCh:
314 + // Program termination requested
315 + timer.Stop()
316 + return
317 + case <-stdinClosedCh:
318 + // Stdin closed
319 + timer.Stop()
320 + stdinClosedCh = nil
321 + continue
322 + }
323 + }
324 + }
325 +
326 + // Update state to connecting
327 + stateMu.Lock()
328 + state = stateConnecting
329 + stateMu.Unlock()
330 +
331 + // Set up connection context with cancellation
332 + ctx, cancel := context.WithCancel(context.Background())
333 +
334 + // Set up connection timeout
335 + connectionCtx, connectionCancel := context.WithTimeout(ctx, 15*time.Second)
336 + defer connectionCancel()
337 +
338 + fmt.Fprintf(os.Stderr, "%s: Connecting to %s...\n", programName, os.Args[1])
339 +
340 + // Create a custom header with the WebSocket key
341 + header := http.Header{}
342 + header.Set("Sec-WebSocket-Key", generateWebSocketKey())
343 + header.Set("Sec-WebSocket-Version", "13")
344 +
345 + // Connect to WebSocket
346 + conn, _, err := websocket.Dial(connectionCtx, os.Args[1], &websocket.DialOptions{
347 + CompressionMode: websocket.CompressionContextTakeover,
348 + HTTPHeader: header,
349 + })
350 +
351 + // Connection failed
352 + if err != nil {
353 + fmt.Fprintf(os.Stderr, "%s: ERROR: websocket connection failed: %v\n", programName, err)
354 + cancel()
355 +
356 + // Increment attempt counter and try again
357 + attempt++
358 +
359 + // Update state to disconnected
360 + stateMu.Lock()
361 + state = stateDisconnected
362 + stateMu.Unlock()
363 + continue
364 + }
365 +
366 + // Increase the read limit to match Netdata's WS_MAX_INCOMING_FRAME_SIZE (16MB)
367 + conn.SetReadLimit(16 * 1024 * 1024)
368 +
369 + // Update state to connected
370 + stateMu.Lock()
371 + state = stateConnected
372 + stateMu.Unlock()
373 +
374 + fmt.Fprintf(os.Stderr, "%s: Connected\n", programName)
375 + attempt = 0 // Reset attempt counter on successful connection
376 +
377 + // Process any queued messages, including pending requests
378 + messageQueueMu.Lock()
379 + if len(messageQueue) > 0 {
380 + fmt.Fprintf(os.Stderr, "%s: Sending %d queued message(s)\n", programName, len(messageQueue))
381 + for _, msg := range messageQueue {
382 + if err := conn.Write(ctx, websocket.MessageText, []byte(msg)); err != nil {
383 + fmt.Fprintf(os.Stderr, "%s: ERROR: failed to send queued message: %v\n", programName, err)
384 + } else {
385 + // Check if this was a pending request with an ID
386 + msgId, _ := parseJsonRpcMessage(msg)
387 + if msgId != nil {
388 + pendingMu.Lock()
389 + if req, exists := pendingRequests[msgId]; exists {
390 + // Stop the timer
391 + if req.Timer != nil {
392 + req.Timer.Stop()
393 + }
394 + delete(pendingRequests, msgId)
395 + }
396 + pendingMu.Unlock()
397 + }
398 + }
399 + }
400 + messageQueue = nil // Clear the queue
401 + }
402 + messageQueueMu.Unlock()
403 +
404 + // Memory management: Limit size of pendingRequests to prevent memory leaks
405 + pendingMu.Lock()
406 + if len(pendingRequests) > 1000 {
407 + fmt.Fprintf(os.Stderr, "%s: Too many pending requests (%d), cleaning up older ones\n", programName, len(pendingRequests))
408 + // Remove older entries (this is a bit tricky in Go without order guarantee)
409 + // For simplicity, we'll just clear it all in this extreme case
410 + pendingRequests = make(map[interface{}]*PendingRequest)
411 + }
412 + pendingMu.Unlock()
413 +
414 + // Signal for connection closure
415 + connClosed := make(chan struct{})
416 +
417 + // Create wait group for goroutines
418 + var wg sync.WaitGroup
419 +
420 + // Goroutine for handling termination signals during connection
421 + wg.Add(1)
422 + go func() {
423 + defer wg.Done()
424 + select {
425 + case <-ctx.Done():
426 + return
427 + case <-doneCh:
428 + // Clean up the connection
429 + conn.Close(websocket.StatusNormalClosure, "Shutdown requested")
430 + cancel()
431 + return
432 + case <-stdinClosedCh:
433 + // Stdin closed, but keep connection active
434 + stdinClosedCh = nil
435 + return
436 + }
437 + }()
438 +
439 + // Goroutine for sending messages to websocket
440 + wg.Add(1)
441 + go func() {
442 + defer wg.Done()
443 + for {
444 + select {
445 + case <-ctx.Done():
446 + return
447 + case <-connClosed:
448 + return
449 + case msg, ok := <-stdinCh:
450 + if !ok {
451 + // Stdin channel was closed
452 + return
453 + }
454 +
455 + err := conn.Write(ctx, websocket.MessageText, []byte(msg))
456 + if err != nil {
457 + // Check if this was a message with ID
458 + msgId, _ := parseJsonRpcMessage(msg)
459 +
460 + select {
461 + case <-connClosed:
462 + // Connection already known to be closed, queue the message
463 + messageQueueMu.Lock()
464 + messageQueue = append(messageQueue, msg)
465 + messageQueueMu.Unlock()
466 +
467 + // If the message had an ID, keep the pending status
468 + if msgId != nil {
469 + pendingMu.Lock()
470 + if _, exists := pendingRequests[msgId]; !exists {
471 + // Create a new pending request with timeout if it doesn't exist
472 + req := &PendingRequest{
473 + ID: msgId,
474 + Message: msg,
475 + Timer: time.AfterFunc(ConnectionTimeout, func() { handleRequestTimeout(msgId) }),
476 + }
477 + pendingRequests[msgId] = req
478 + }
479 + pendingMu.Unlock()
480 + }
481 + default:
482 + fmt.Fprintf(os.Stderr, "%s: ERROR: write to websocket: %v\n", programName, err)
483 + messageQueueMu.Lock()
484 + messageQueue = append(messageQueue, msg)
485 + messageQueueMu.Unlock()
486 +
487 + // Same ID handling as above
488 + if msgId != nil {
489 + pendingMu.Lock()
490 + if _, exists := pendingRequests[msgId]; !exists {
491 + req := &PendingRequest{
492 + ID: msgId,
493 + Message: msg,
494 + Timer: time.AfterFunc(ConnectionTimeout, func() { handleRequestTimeout(msgId) }),
495 + }
496 + pendingRequests[msgId] = req
497 + }
498 + pendingMu.Unlock()
499 + }
500 + }
501 + } else {
502 + // Message was sent successfully, remove from pending if it had an ID
503 + msgId, _ := parseJsonRpcMessage(msg)
504 + if msgId != nil {
505 + pendingMu.Lock()
506 + if req, exists := pendingRequests[msgId]; exists {
507 + // Stop the connection establishment timer (if any)
508 + if req.Timer != nil {
509 + req.Timer.Stop()
510 + req.Timer = nil // Clear timer to prevent memory leaks
511 + }
512 + // Keep in pendingRequests until we get a response
513 + }
514 + pendingMu.Unlock()
515 + }
516 + }
517 + }
518 + }
519 + }()
520 +
521 + // Goroutine for receiving messages from websocket
522 + wg.Add(1)
523 + go func() {
524 + defer wg.Done()
525 + defer close(connClosed)
526 +
527 + for {
528 + messageType, message, err := conn.Read(ctx)
529 + if err != nil {
530 + // Don't report error if context was cancelled
531 + if ctx.Err() == nil {
532 + fmt.Fprintf(os.Stderr, "%s: ERROR: read from websocket: %v\n", programName, err)
533 + }
534 + return
535 + }
536 +
537 + if messageType == websocket.MessageText {
538 + fmt.Println(string(message))
539 +
540 + // Check if this is a response to a request with an ID
541 + var response JsonRpcMessage
542 + if err := json.Unmarshal(message, &response); err == nil && response.JsonRpc == "2.0" && response.Id != nil {
543 + pendingMu.Lock()
544 + if req, exists := pendingRequests[response.Id]; exists {
545 + // Stop the timer
546 + if req.Timer != nil {
547 + req.Timer.Stop()
548 + }
549 + delete(pendingRequests, response.Id)
550 + }
551 + pendingMu.Unlock()
552 + }
553 + }
554 + }
555 + }()
556 +
557 + // Set up periodic pinger to keep connection alive
558 + pingInterval := 30 * time.Second
559 + wg.Add(1)
560 + go func() {
561 + defer wg.Done()
562 + ticker := time.NewTicker(pingInterval)
563 + defer ticker.Stop()
564 +
565 + for {
566 + select {
567 + case <-ticker.C:
568 + // Send ping using a text message with a special ping format
569 + err := conn.Write(ctx, websocket.MessageText, []byte("PING"))
570 + if err != nil {
571 + fmt.Fprintf(os.Stderr, "%s: ERROR: ping failed: %v\n", programName, err)
572 + // Connection will be closed by the read loop
573 + }
574 + case <-ctx.Done():
575 + return
576 + case <-connClosed:
577 + return
578 + }
579 + }
580 + }()
581 +
582 + // Wait for connection closed
583 + <-connClosed
584 +
585 + // Update state to disconnected
586 + stateMu.Lock()
587 + state = stateDisconnected
588 + stateMu.Unlock()
589 +
590 + // Clean up
591 + cancel()
592 + wg.Wait()
593 +
594 + // Don't automatically reconnect if stdin closed and no messages in queue
595 + messageQueueMu.Lock()
596 + hasMessages := len(messageQueue) > 0
597 + messageQueueMu.Unlock()
598 +
599 + if !stdinActive && !hasMessages {
600 + fmt.Fprintf(os.Stderr, "%s: Stdin closed and no pending messages, exiting\n", programName)
601 + return
602 + }
603 +
604 + // Increment attempt counter for next reconnection
605 + attempt++
606 + }
607 +}
608 +
609 +// jitter returns a random value between 0.0 and 1.0
610 +func jitter() float64 {
611 + n, err := rand.Int(rand.Reader, big.NewInt(1000))
612 + if err != nil {
613 + return 0.5 // Fall back to 0.5 if there's an error
614 + }
615 + return float64(n.Int64()) / 1000.0
616 +}
\ No newline at end of file
src/web/mcp/bridges/stdio-nodejs/.gitignore new
+33
@@ -0,0 +1,33 @@
1 +# Dependency directories
2 +node_modules/
3 +npm-debug.log*
4 +yarn-debug.log*
5 +yarn-error.log*
6 +package-lock.json
7 +yarn.lock
8 +
9 +# Generated files
10 +package.json
11 +
12 +# Optional npm cache directory
13 +.npm
14 +
15 +# Optional eslint cache
16 +.eslintcache
17 +
18 +# Output of 'npm pack'
19 +*.tgz
20 +
21 +# dotenv environment variable files
22 +.env
23 +.env.local
24 +.env.development.local
25 +.env.test.local
26 +.env.production.local
27 +
28 +# IDE specific files
29 +.idea/
30 +.vscode/
31 +*.swp
32 +*.swo
33 +*~
\ No newline at end of file
src/web/mcp/bridges/stdio-nodejs/README.md new
+90
@@ -0,0 +1,90 @@
1 +# Netdata MCP Bridge - Node.js Implementation
2 +
3 +This Node.js bridge converts MCP stdio communication to Netdata's MCP over WebSocket.
4 +
5 +## Requirements
6 +
7 +- Node.js 14+
8 +- ws library
9 +
10 +## Installation
11 +
12 +The easiest way to set up the Node.js bridge is to use the included build script:
13 +
14 +```bash
15 +# Make the build script executable (if needed)
16 +chmod +x build.sh
17 +
18 +# Run the build script
19 +./build.sh
20 +```
21 +
22 +The script will:
23 +1. Check if Node.js is installed
24 +2. Create a package.json file if it doesn't exist
25 +3. Install the required ws dependency
26 +
27 +Alternatively, you can set it up manually:
28 +
29 +```bash
30 +# Create a package.json file (optional)
31 +npm init -y
32 +
33 +# Install the WebSocket library
34 +npm install ws --save
35 +```
36 +
37 +## Usage
38 +
39 +The script can be run directly as an executable (the shebang line will use the system's Node.js):
40 +
41 +```bash
42 +./nd-mcp.js ws://<ip>:19999/mcp
43 +```
44 +
45 +Or explicitly with Node.js:
46 +
47 +```bash
48 +node nd-mcp.js ws://<ip>:19999/mcp
49 +```
50 +
51 +Where `<ip>` is either `localhost` or the IP address where a Netdata instance is listening.
52 +
53 +## Example with Claude Desktop
54 +
55 +To use this bridge with Claude Desktop:
56 +
57 +1. In Claude Desktop settings, configure the Custom Command option:
58 +
59 +```bash
60 +node /path/to/stdio-nodejs/nd-mcp.js ws://localhost:19999/mcp
61 +```
62 +
63 +2. If your Netdata instance is running on a different machine, replace `localhost` with the appropriate IP address.
64 +
65 +## How It Works
66 +
67 +The bridge:
68 +1. Establishes a WebSocket connection to the specified Netdata MCP endpoint
69 +2. Reads from standard input and sends to the WebSocket
70 +3. Receives messages from the WebSocket and writes to standard output
71 +4. Handles both directions simultaneously
72 +5. Automatically reconnects if the connection is lost, with exponential backoff
73 +
74 +## Connection Reliability
75 +
76 +This bridge implements robust connection handling:
77 +
78 +- **Automatic Reconnection**: If the WebSocket connection is lost, the bridge will automatically attempt to reconnect
79 +- **Exponential Backoff**: Reconnection attempts use exponential backoff with jitter to avoid overwhelming the server
80 +- **Message Queuing**: Messages sent while disconnected are queued and delivered once reconnected
81 +- **Connection Status Logging**: The bridge logs connection status to stderr for monitoring
82 +- **SIGINT Handling**: Properly closes the WebSocket connection on CTRL+C
83 +
84 +The reconnection algorithm starts with a 1-second delay and doubles the wait time with each attempt, up to a maximum of 60 seconds. Random jitter is added to prevent connection storms.
85 +
86 +## Protocol Compatibility
87 +
88 +- Netdata MCP implements the JSON-RPC 2.0 protocol
89 +- Messages that don't conform to the JSON-RPC 2.0 format are silently ignored
90 +- The bridge passes messages directly without any modification
\ No newline at end of file
src/web/mcp/bridges/stdio-nodejs/build.sh new
+60
@@ -0,0 +1,60 @@
1 +#!/bin/bash
2 +set -e
3 +
4 +# Get the directory where the script is located
5 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6 +cd "$SCRIPT_DIR"
7 +
8 +echo "Working directory: $SCRIPT_DIR"
9 +
10 +# Check if Node.js is installed
11 +if ! command -v node &> /dev/null; then
12 + echo "Error: Node.js is not installed or not in the PATH"
13 + echo "Please install Node.js from https://nodejs.org/"
14 + exit 1
15 +fi
16 +
17 +echo "Node.js is installed: $(node --version)"
18 +echo "npm version: $(npm --version)"
19 +
20 +# Check if package.json exists, if not create it
21 +if [ ! -f "package.json" ]; then
22 + echo "Creating package.json..."
23 + npm init -y
24 +
25 + # Update package description and add script
26 + tmp=$(mktemp)
27 + jq '.description = "Netdata MCP WebSocket bridge for Node.js" |
28 + .scripts.start = "node nd-mcp.js" |
29 + .private = true' package.json > "$tmp" && mv "$tmp" package.json
30 +
31 + echo "Created package.json"
32 +else
33 + echo "package.json already exists"
34 +fi
35 +
36 +# Install dependencies if not already installed
37 +if [ ! -d "node_modules" ]; then
38 + echo "Installing dependencies..."
39 + npm install ws --save
40 +else
41 + echo "Dependencies already installed"
42 + echo "To reinstall dependencies, delete the node_modules directory and run this script again"
43 +fi
44 +
45 +# Get the full path to the script
46 +SCRIPT_PATH="$SCRIPT_DIR/nd-mcp.js"
47 +
48 +# Make sure the script is executable
49 +chmod +x "$SCRIPT_PATH"
50 +
51 +echo "Build complete!"
52 +echo ""
53 +echo "You can now run the bridge using the full path:"
54 +echo "$SCRIPT_PATH ws://ip:19999/mcp"
55 +echo ""
56 +echo "Or using Node.js with the full path:"
57 +echo "node $SCRIPT_PATH ws://ip:19999/mcp"
58 +echo ""
59 +echo "Or from this directory:"
60 +echo "./nd-mcp.js ws://ip:19999/mcp"
\ No newline at end of file
src/web/mcp/bridges/stdio-nodejs/nd-mcp.js new
+385
@@ -0,0 +1,385 @@
1 +#!/usr/bin/env node
2 +
3 +const WebSocket = require('ws');
4 +const path = require('path');
5 +
6 +// Get program name for logs
7 +const PROGRAM_NAME = path.basename(process.argv[1] || 'nd-mcp-nodejs');
8 +
9 +if (process.argv.length !== 3) {
10 + console.error(`${PROGRAM_NAME}: Usage: ${PROGRAM_NAME} ws://host/path`);
11 + process.exit(1);
12 +}
13 +
14 +// Reconnection settings
15 +const MAX_RECONNECT_DELAY_MS = 60000; // 60 seconds
16 +const BASE_DELAY_MS = 1000; // 1 second
17 +const CONNECTION_TIMEOUT_MS = 5000; // 5 seconds timeout for initial connection
18 +let reconnectAttempt = 0;
19 +let ws = null;
20 +let messageQueue = [];
21 +let reconnectTimeout = null;
22 +let stdinActive = true;
23 +let connectingInProgress = false;
24 +let pendingRequests = new Map(); // Store pending requests by their IDs
25 +
26 +// Parse JSON-RPC message and extract ID if present
27 +function parseJsonRpcMessage(message) {
28 + try {
29 + const data = JSON.parse(message);
30 + if (data && typeof data === 'object' && data.jsonrpc === '2.0') {
31 + return [data.id, data.method];
32 + }
33 + return [null, null];
34 + } catch (err) {
35 + return [null, null];
36 + }
37 +}
38 +
39 +// Create a JSON-RPC error response
40 +function createJsonRpcError(id, code, message, data = null) {
41 + const response = {
42 + jsonrpc: '2.0',
43 + id: id,
44 + error: {
45 + code: code,
46 + message: message
47 + }
48 + };
49 +
50 + if (data !== null) {
51 + response.error.data = data;
52 + }
53 +
54 + return JSON.stringify(response);
55 +}
56 +
57 +// Handle request timeout for connection establishment only
58 +function handleRequestTimeout(msgId) {
59 + // This timeout ONLY applies if we're still not connected after waiting
60 + if (!ws || ws.readyState !== WebSocket.OPEN) {
61 + if (pendingRequests.has(msgId)) {
62 + console.error(`${PROGRAM_NAME}: Connection timeout for request ID ${msgId}, sending error response`);
63 +
64 + // Create and send error response
65 + const errorResponse = createJsonRpcError(
66 + msgId,
67 + -32000, // Server error code
68 + "MCP server connection failed",
69 + { details: "Could not establish connection to Netdata within timeout period" }
70 + );
71 +
72 + try {
73 + process.stdout.write(errorResponse + "\n");
74 + } catch (err) {
75 + console.error(`${PROGRAM_NAME}: ERROR: Failed to write error response to stdout: ${err.message}`);
76 + }
77 +
78 + // Get the original message from pendingRequests
79 + const originalMessage = pendingRequests.get(msgId);
80 +
81 + // Remove from pending requests
82 + pendingRequests.delete(msgId);
83 +
84 + // Also remove from messageQueue if it exists there
85 + const msgIndex = messageQueue.indexOf(originalMessage);
86 + if (msgIndex !== -1) {
87 + messageQueue.splice(msgIndex, 1);
88 + console.error(`${PROGRAM_NAME}: Removed timed-out request from message queue`);
89 + }
90 + }
91 + } else {
92 + // If we get here, it means the connection was established before the timeout
93 + // Just clear the pending request without any action
94 + if (pendingRequests.has(msgId)) {
95 + // Connection succeeded in time, just remove the tracking
96 + pendingRequests.delete(msgId);
97 + }
98 + }
99 +}
100 +
101 +// Set up stdin processing once
102 +process.stdin.setEncoding('utf8');
103 +process.stdin.on('data', (data) => {
104 + try {
105 + // Flag to check if we received actual data
106 + let hasContent = false;
107 +
108 + data.split(/\r?\n/).forEach(line => {
109 + if (line.trim() !== '') {
110 + hasContent = true;
111 +
112 + // Parse the message for JSON-RPC ID
113 + const [msgId, method] = parseJsonRpcMessage(line);
114 +
115 + if (ws && ws.readyState === WebSocket.OPEN) {
116 + // If connected, send immediately and track requests with IDs
117 + ws.send(line);
118 +
119 + // If this is a request with an ID, track it (but don't set a timeout)
120 + if (msgId !== null && msgId !== undefined) {
121 + pendingRequests.set(msgId, line);
122 + }
123 + } else {
124 + // Queue the message if not connected
125 + messageQueue.push(line);
126 +
127 + // If this is a request with an ID, set a timeout for CONNECTION establishment
128 + if (msgId !== null && msgId !== undefined) {
129 + console.error(`${PROGRAM_NAME}: Received request with ID ${msgId}, setting connection timeout`);
130 + pendingRequests.set(msgId, line);
131 +
132 + // Set timeout to send error if connection not established quickly
133 + // This ONLY applies to connection establishment, not request processing
134 + setTimeout(() => handleRequestTimeout(msgId), CONNECTION_TIMEOUT_MS);
135 + }
136 + }
137 + }
138 + });
139 +
140 + // If disconnected and we got content, try to reconnect immediately
141 + if (hasContent && (!ws || ws.readyState !== WebSocket.OPEN)) {
142 + console.error(`${PROGRAM_NAME}: Received stdin data, attempting immediate reconnection`);
143 + // Clear any pending reconnect timeout
144 + if (reconnectTimeout) {
145 + clearTimeout(reconnectTimeout);
146 + reconnectTimeout = null;
147 + }
148 +
149 + // Only attempt connection if not already connecting
150 + if (!connectingInProgress) {
151 + connect(true); // immediate=true
152 + }
153 + }
154 + } catch (err) {
155 + console.error(`${PROGRAM_NAME}: ERROR: Failed to process data: ${err.message}`);
156 + }
157 +});
158 +
159 +process.stdin.on('error', (err) => {
160 + console.error(`${PROGRAM_NAME}: ERROR: stdin: ${err.message}`);
161 + stdinActive = false;
162 + process.exit(1);
163 +});
164 +
165 +process.stdin.on('end', () => {
166 + console.error(`${PROGRAM_NAME}: End of stdin, will exit when WebSocket disconnects`);
167 + stdinActive = false;
168 +
169 + // If WebSocket is not active, exit immediately
170 + if (!ws || ws.readyState !== WebSocket.OPEN) {
171 + process.exit(0);
172 + }
173 +});
174 +
175 +function connect(immediate = false) {
176 + // Clear any existing reconnect timeout
177 + if (reconnectTimeout) {
178 + clearTimeout(reconnectTimeout);
179 + reconnectTimeout = null;
180 + }
181 +
182 + // If stdin is no longer active (closed/error) and we're disconnected, exit
183 + if (!stdinActive && (!ws || ws.readyState !== WebSocket.OPEN)) {
184 + console.error(`${PROGRAM_NAME}: Stdin closed and WebSocket disconnected, exiting`);
185 + process.exit(0);
186 + }
187 +
188 + if (immediate || reconnectAttempt === 0) {
189 + attemptConnection();
190 + } else {
191 + // Exponential backoff with jitter
192 + const delay = Math.min(
193 + MAX_RECONNECT_DELAY_MS,
194 + BASE_DELAY_MS * Math.pow(2, reconnectAttempt - 1) * (0.5 + Math.random())
195 + );
196 +
197 + console.error(`${PROGRAM_NAME}: Reconnecting in ${(delay/1000).toFixed(1)} seconds (attempt ${reconnectAttempt+1})...`);
198 + reconnectTimeout = setTimeout(() => {
199 + reconnectTimeout = null;
200 + attemptConnection();
201 + }, delay);
202 + }
203 +}
204 +
205 +function attemptConnection() {
206 + // Prevent multiple concurrent connection attempts
207 + if (connectingInProgress) {
208 + return;
209 + }
210 +
211 + connectingInProgress = true;
212 + console.error(`${PROGRAM_NAME}: Connecting to ${process.argv[2]}...`);
213 +
214 + // Close any existing websocket
215 + if (ws) {
216 + try {
217 + ws.terminate();
218 + } catch (e) {
219 + // Ignore errors
220 + }
221 + ws = null;
222 + }
223 +
224 + const wsOptions = {
225 + perMessageDeflate: true,
226 + maxPayload: 16 * 1024 * 1024, // 16MB to match Netdata's limits
227 + handshakeTimeout: 10000, // 10 seconds for initial handshake
228 + followRedirects: true, // Follow HTTP redirects
229 + // Keep the connection alive with pings
230 + pingInterval: 30000, // 30 seconds
231 + pingTimeout: 10000 // 10 seconds to wait for pong
232 + };
233 +
234 + ws = new WebSocket(process.argv[2], wsOptions);
235 +
236 + // Set a timeout for initial connection
237 + const connectionTimeout = setTimeout(() => {
238 + if (ws && ws.readyState !== WebSocket.OPEN) {
239 + console.error(`${PROGRAM_NAME}: Connection timeout, closing...`);
240 + ws.terminate();
241 + }
242 + }, 15000); // 15 seconds connection timeout
243 +
244 + ws.on('open', () => {
245 + clearTimeout(connectionTimeout);
246 + console.error(`${PROGRAM_NAME}: Connected`);
247 + connectingInProgress = false;
248 + reconnectAttempt = 0; // Reset counter on successful connection
249 +
250 + // Process any queued messages
251 + if (messageQueue.length > 0) {
252 + console.error(`${PROGRAM_NAME}: Sending ${messageQueue.length} queued message(s)`);
253 + const queueCopy = [...messageQueue];
254 + messageQueue = [];
255 + queueCopy.forEach(msg => {
256 + try {
257 + ws.send(msg);
258 +
259 + // If this was a pending request with an ID, remove it from pending
260 + const [msgId] = parseJsonRpcMessage(msg);
261 + if (msgId !== null && pendingRequests.has(msgId)) {
262 + pendingRequests.delete(msgId);
263 + }
264 +
265 + // Ensure we don't let the message queue grow indefinitely
266 + if (messageQueue.length > 1000) {
267 + console.error(`${PROGRAM_NAME}: Message queue too large (${messageQueue.length}), trimming older messages`);
268 + messageQueue = messageQueue.slice(-500); // Keep only the 500 most recent messages
269 + }
270 + } catch (err) {
271 + console.error(`${PROGRAM_NAME}: ERROR: Failed to send queued message: ${err.message}`);
272 + // Re-queue the message if connection is still open
273 + if (ws.readyState === WebSocket.OPEN) {
274 + messageQueue.push(msg);
275 + }
276 + }
277 + });
278 + }
279 + });
280 +
281 + ws.on('message', (message) => {
282 + try {
283 + process.stdout.write(message + "\n");
284 +
285 + // If this is a response to a request, check if we can remove it from pending
286 + try {
287 + const data = JSON.parse(message);
288 + if (data && typeof data === 'object' && data.jsonrpc === '2.0' && data.id !== undefined) {
289 + if (pendingRequests.has(data.id)) {
290 + pendingRequests.delete(data.id);
291 + }
292 + }
293 + } catch (err) {
294 + // Ignore parsing errors
295 + }
296 +
297 + // Memory management: Don't let pendingRequests grow indefinitely
298 + if (pendingRequests.size > 1000) {
299 + console.error(`${PROGRAM_NAME}: Too many pending requests (${pendingRequests.size}), clearing older ones`);
300 + // Since Map iteration is in insertion order, we keep the newest entries
301 + const entries = Array.from(pendingRequests.entries());
302 + pendingRequests.clear();
303 + entries.slice(-500).forEach(([id, msg]) => pendingRequests.set(id, msg));
304 + }
305 + } catch (err) {
306 + console.error(`${PROGRAM_NAME}: ERROR: Failed to write to stdout: ${err.message}`);
307 + // If stdout is broken, exit
308 + process.exit(1);
309 + }
310 + });
311 +
312 + ws.on('close', (code, reason) => {
313 + clearTimeout(connectionTimeout);
314 + console.error(`${PROGRAM_NAME}: WebSocket closed with code ${code}${reason ? ': ' + reason : ''}`);
315 + connectingInProgress = false;
316 +
317 + // Only increment reconnect attempt if we're still supposed to be running
318 + if (stdinActive) {
319 + reconnectAttempt++;
320 + connect();
321 + } else {
322 + console.error(`${PROGRAM_NAME}: Stdin closed, exiting`);
323 + process.exit(0);
324 + }
325 + });
326 +
327 + ws.on('error', (err) => {
328 + console.error(`${PROGRAM_NAME}: ERROR: WebSocket error: ${err.message}`);
329 + connectingInProgress = false;
330 + // The close event should handle reconnection
331 + });
332 +
333 + // Verify the connection is healthy periodically
334 + ws.on('pong', () => {
335 + // Connection is alive
336 + });
337 +}
338 +
339 +// Handle process termination
340 +process.on('SIGINT', () => {
341 + console.error(`${PROGRAM_NAME}: Received SIGINT, shutting down`);
342 + // Clear any pending reconnect
343 + if (reconnectTimeout) {
344 + clearTimeout(reconnectTimeout);
345 + reconnectTimeout = null;
346 + }
347 +
348 + if (ws) {
349 + try {
350 + ws.close(1000, 'Normal closure');
351 + } catch (e) {
352 + // Ignore errors during cleanup
353 + try {
354 + ws.terminate();
355 + } catch (e) {
356 + // Last resort
357 + }
358 + }
359 + }
360 + process.exit(0);
361 +});
362 +
363 +process.on('SIGTERM', () => {
364 + console.error(`${PROGRAM_NAME}: Received SIGTERM, shutting down`);
365 + // Same cleanup as SIGINT
366 + if (reconnectTimeout) {
367 + clearTimeout(reconnectTimeout);
368 + }
369 +
370 + if (ws) {
371 + try {
372 + ws.close(1000, 'Normal closure');
373 + } catch (e) {
374 + try {
375 + ws.terminate();
376 + } catch (e) {
377 + // Ignore
378 + }
379 + }
380 + }
381 + process.exit(0);
382 +});
383 +
384 +// Start the connection process
385 +connect();
\ No newline at end of file
src/web/mcp/bridges/stdio-python/.gitignore new
+48
@@ -0,0 +1,48 @@
1 +# Virtual environment
2 +venv/
3 +env/
4 +ENV/
5 +.env/
6 +.venv/
7 +
8 +# Generated pip files
9 +requirements.txt
10 +pip-log.txt
11 +
12 +# Byte-compiled / optimized / DLL files
13 +__pycache__/
14 +*.py[cod]
15 +*$py.class
16 +*.so
17 +.Python
18 +
19 +# Distribution / packaging
20 +dist/
21 +build/
22 +*.egg-info/
23 +.installed.cfg
24 +*.egg
25 +
26 +# Unit test / coverage reports
27 +htmlcov/
28 +.tox/
29 +.coverage
30 +.coverage.*
31 +.cache
32 +nosetests.xml
33 +coverage.xml
34 +*.cover
35 +.hypothesis/
36 +
37 +# Jupyter Notebook
38 +.ipynb_checkpoints
39 +
40 +# IDE specific files
41 +.idea/
42 +.vscode/
43 +*.swp
44 +*.swo
45 +*~
46 +.spyderproject
47 +.spyproject
48 +.ropeproject
\ No newline at end of file
src/web/mcp/bridges/stdio-python/README.md new
+115
@@ -0,0 +1,115 @@
1 +# Netdata MCP Bridge - Python Implementation
2 +
3 +This Python bridge converts MCP stdio communication to Netdata's MCP over WebSocket.
4 +
5 +## Requirements
6 +
7 +- Python 3.7+
8 +- websockets library
9 +
10 +## Installation
11 +
12 +The easiest way to set up the Python bridge is to use the included build script, which creates a virtual environment (recommended for most Linux distributions where pip is restricted to virtual environments):
13 +
14 +```bash
15 +# Make the build script executable (if needed)
16 +chmod +x build.sh
17 +
18 +# Run the build script
19 +./build.sh
20 +```
21 +
22 +The script will:
23 +1. Check if Python 3 is installed
24 +2. Create a virtual environment in the `venv` directory if it doesn't exist
25 +3. Install the required websockets dependency in the virtual environment
26 +4. Generate a requirements.txt file for reproducibility
27 +
28 +### Running with Virtual Environment
29 +
30 +After running the build script:
31 +
32 +```bash
33 +# Activate the virtual environment
34 +source venv/bin/activate
35 +
36 +# Run the bridge
37 +python nd-mcp.py ws://localhost:19999/mcp
38 +
39 +# When finished, deactivate the virtual environment
40 +deactivate
41 +```
42 +
43 +Or run it directly with the virtual environment's Python:
44 +
45 +```bash
46 +./venv/bin/python nd-mcp.py ws://localhost:19999/mcp
47 +```
48 +
49 +### Manual Installation (without virtual environment)
50 +
51 +If your system allows global pip installations:
52 +
53 +```bash
54 +pip install websockets
55 +```
56 +
57 +## Usage
58 +
59 +The script can be run directly as an executable (the shebang line will use the system's Python):
60 +
61 +```bash
62 +./nd-mcp.py ws://<ip>:19999/mcp
63 +```
64 +
65 +Or explicitly with Python:
66 +
67 +```bash
68 +python nd-mcp.py ws://<ip>:19999/mcp
69 +```
70 +
71 +When using a virtual environment, run it with the environment's Python:
72 +
73 +```bash
74 +./venv/bin/python nd-mcp.py ws://<ip>:19999/mcp
75 +```
76 +
77 +Where `<ip>` is either `localhost` or the IP address where a Netdata instance is listening.
78 +
79 +## Example with Claude Desktop
80 +
81 +To use this bridge with Claude Desktop:
82 +
83 +1. In Claude Desktop settings, configure the Custom Command option:
84 +
85 +```bash
86 +python /path/to/stdio-python/nd-mcp.py ws://localhost:19999/mcp
87 +```
88 +
89 +2. If your Netdata instance is running on a different machine, replace `localhost` with the appropriate IP address.
90 +
91 +## How It Works
92 +
93 +The bridge:
94 +1. Establishes a WebSocket connection to the specified Netdata MCP endpoint
95 +2. Reads from standard input and sends to the WebSocket
96 +3. Receives messages from the WebSocket and writes to standard output
97 +4. Handles both directions simultaneously
98 +5. Automatically reconnects if the connection is lost, with exponential backoff
99 +
100 +## Connection Reliability
101 +
102 +This bridge implements robust connection handling:
103 +
104 +- **Automatic Reconnection**: If the WebSocket connection is lost, the bridge will automatically attempt to reconnect
105 +- **Exponential Backoff**: Reconnection attempts use exponential backoff with jitter to avoid overwhelming the server
106 +- **Message Queuing**: Messages sent while disconnected are queued and delivered once reconnected
107 +- **Connection Status Logging**: The bridge logs connection status to stderr for monitoring
108 +
109 +The reconnection algorithm starts with a 1-second delay and doubles the wait time with each attempt, up to a maximum of 60 seconds. Random jitter is added to prevent connection storms.
110 +
111 +## Protocol Compatibility
112 +
113 +- Netdata MCP implements the JSON-RPC 2.0 protocol
114 +- Messages that don't conform to the JSON-RPC 2.0 format are silently ignored
115 +- The bridge passes messages directly without any modification
\ No newline at end of file
src/web/mcp/bridges/stdio-python/build.sh new
+70
@@ -0,0 +1,70 @@
1 +#!/bin/bash
2 +set -e
3 +
4 +# Get the directory where the script is located
5 +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
6 +cd "$SCRIPT_DIR"
7 +
8 +echo "Working directory: $SCRIPT_DIR"
9 +
10 +# Check if Python is installed
11 +if ! command -v python3 &> /dev/null; then
12 + echo "Error: Python 3 is not installed or not in the PATH"
13 + echo "Please install Python 3 from https://www.python.org/ or through your package manager"
14 + exit 1
15 +fi
16 +
17 +echo "Python is installed: $(python3 --version)"
18 +
19 +# Create virtual environment if it doesn't exist
20 +if [ ! -d "venv" ]; then
21 + echo "Creating virtual environment..."
22 + python3 -m venv venv
23 + echo "Virtual environment created at $SCRIPT_DIR/venv"
24 +else
25 + echo "Virtual environment already exists at $SCRIPT_DIR/venv"
26 +fi
27 +
28 +# Activate virtual environment
29 +echo "Activating virtual environment..."
30 +source venv/bin/activate || { echo "Failed to activate virtual environment"; exit 1; }
31 +
32 +# Check pip version
33 +echo "pip version: $(pip --version)"
34 +
35 +# Install requirements
36 +echo "Installing dependencies..."
37 +pip install websockets
38 +
39 +# Create requirements.txt if it doesn't exist
40 +if [ ! -f "requirements.txt" ]; then
41 + echo "Creating requirements.txt..."
42 + pip freeze > requirements.txt
43 + echo "requirements.txt created"
44 +else
45 + echo "requirements.txt already exists"
46 + echo "To update it, run: source $SCRIPT_DIR/venv/bin/activate && pip freeze > requirements.txt"
47 +fi
48 +
49 +# Get the full path to the script and venv
50 +SCRIPT_PATH="$SCRIPT_DIR/nd-mcp.py"
51 +VENV_PYTHON="$SCRIPT_DIR/venv/bin/python"
52 +VENV_ACTIVATE="$SCRIPT_DIR/venv/bin/activate"
53 +
54 +# Make sure the script is executable
55 +chmod +x "$SCRIPT_PATH"
56 +
57 +echo "Build complete!"
58 +echo ""
59 +echo "To use the bridge with the virtual environment:"
60 +echo ""
61 +echo "Option 1: Activate the environment and run the script"
62 +echo "source $VENV_ACTIVATE"
63 +echo "python $SCRIPT_PATH ws://ip:19999/mcp"
64 +echo "deactivate # when finished"
65 +echo ""
66 +echo "Option 2: Run directly with the virtual environment's Python"
67 +echo "$VENV_PYTHON $SCRIPT_PATH ws://ip:19999/mcp"
68 +echo ""
69 +echo "Option 3: Run the script directly (using the system's Python)"
70 +echo "$SCRIPT_PATH ws://ip:19999/mcp"
\ No newline at end of file
src/web/mcp/bridges/stdio-python/nd-mcp.py new
+349
@@ -0,0 +1,349 @@
1 +#!/usr/bin/env python3
2 +# -*- coding: utf-8 -*-
3 +
4 +import sys
5 +import asyncio
6 +import websockets
7 +import os.path
8 +import random
9 +import time
10 +import signal
11 +import json
12 +
13 +# Get program name for logs
14 +PROGRAM_NAME = os.path.basename(sys.argv[0]) if len(sys.argv) > 0 else "nd-mcp-python"
15 +
16 +# Global flag to track if we're exiting due to a stdin issue
17 +STDIN_ERROR = False
18 +
19 +# Timeout for connection attempts when a message with ID is waiting
20 +CONNECTION_TIMEOUT = 5 # seconds
21 +
22 +# Parse JSON-RPC message and extract ID if present
23 +def parse_jsonrpc_message(message):
24 + try:
25 + data = json.loads(message)
26 + if isinstance(data, dict) and "jsonrpc" in data and data.get("jsonrpc") == "2.0":
27 + return data.get("id"), data.get("method")
28 + return None, None
29 + except json.JSONDecodeError:
30 + return None, None
31 +
32 +# Create a JSON-RPC error response
33 +def create_jsonrpc_error(id, code, message, data=None):
34 + response = {
35 + "jsonrpc": "2.0",
36 + "id": id,
37 + "error": {
38 + "code": code,
39 + "message": message
40 + }
41 + }
42 + if data is not None:
43 + response["error"]["data"] = data
44 + return json.dumps(response)
45 +
46 +async def connect_with_backoff(uri):
47 + max_delay = 60 # Maximum delay between reconnections in seconds
48 + base_delay = 1 # Initial delay in seconds
49 + retry_count = 0
50 +
51 + # Message queue for storing messages during disconnections
52 + stdin_queue = asyncio.Queue()
53 +
54 + # Dictionary to track pending requests with their IDs and timers
55 + pending_requests = {}
56 +
57 + # Set up stdin reader once
58 + async def read_stdin():
59 + global STDIN_ERROR
60 + try:
61 + loop = asyncio.get_running_loop()
62 + reader = asyncio.StreamReader()
63 + await loop.connect_read_pipe(lambda: asyncio.StreamReaderProtocol(reader), sys.stdin)
64 +
65 + while True:
66 + line = await reader.readline()
67 + if not line:
68 + print(f"{PROGRAM_NAME}: End of stdin, exiting...", file=sys.stderr)
69 + STDIN_ERROR = True
70 + # Signal the main event loop to exit
71 + loop.stop()
72 + return
73 +
74 + # Process the received line
75 + line_text = line.decode().strip()
76 + if not line_text:
77 + continue
78 +
79 + # Parse JSON-RPC message
80 + msg_id, msg_method = parse_jsonrpc_message(line_text)
81 +
82 + # Store message in queue
83 + await stdin_queue.put((line_text, msg_id, msg_method))
84 +
85 + # If we're disconnected, check if we need to respond with error
86 + if retry_count > 0:
87 + print(f"{PROGRAM_NAME}: Received stdin data, attempting immediate reconnection", file=sys.stderr)
88 + retry_event.set()
89 +
90 + if msg_id is not None:
91 + # Set timer for this request
92 + pending_requests[msg_id] = {
93 + "message": line_text,
94 + "timer": asyncio.create_task(handle_request_timeout(msg_id, CONNECTION_TIMEOUT))
95 + }
96 +
97 + except Exception as e:
98 + print(f"{PROGRAM_NAME}: ERROR: stdin reader: {e}", file=sys.stderr)
99 + STDIN_ERROR = True
100 + # Signal the main event loop to exit
101 + loop.stop()
102 + return
103 +
104 + # Handler for request timeout - ONLY for connection establishment
105 + async def handle_request_timeout(msg_id, timeout):
106 + await asyncio.sleep(timeout)
107 +
108 + # If we're still disconnected and the request is still pending
109 + if retry_count > 0 and msg_id in pending_requests:
110 + print(f"{PROGRAM_NAME}: Connection timeout for request ID {msg_id}, sending error response", file=sys.stderr)
111 +
112 + # Create and send error response
113 + error_response = create_jsonrpc_error(
114 + msg_id,
115 + -32000, # Server error code
116 + "MCP server connection failed",
117 + {"details": "Could not establish connection to Netdata within timeout period"}
118 + )
119 + print(error_response, flush=True)
120 +
121 + # Mark this message as timed out
122 + if msg_id in pending_requests:
123 + pending_requests[msg_id]["timed_out"] = True
124 +
125 + # We'll keep it in pending_requests so we don't resend it, but mark it as handled
126 + print(f"{PROGRAM_NAME}: Marked request ID {msg_id} as timed out - it will not be sent to server", file=sys.stderr)
127 + elif msg_id in pending_requests:
128 + # Connection was established before timeout, just clean up the timer
129 + print(f"{PROGRAM_NAME}: Connection established before timeout for request ID {msg_id}", file=sys.stderr)
130 +
131 + # Create an event for signaling reconnection
132 + retry_event = asyncio.Event()
133 +
134 + # Start reading stdin in the background
135 + stdin_task = asyncio.create_task(read_stdin())
136 +
137 + while True:
138 + if STDIN_ERROR:
139 + print(f"{PROGRAM_NAME}: Stdin error detected, exiting", file=sys.stderr)
140 + return
141 +
142 + try:
143 + # Calculate backoff delay with jitter
144 + delay = min(max_delay, base_delay * (2 ** retry_count) * (0.5 + random.random()))
145 +
146 + if retry_count > 0:
147 + print(f"{PROGRAM_NAME}: Reconnecting in {delay:.1f} seconds (attempt {retry_count+1})...", file=sys.stderr)
148 +
149 + # Create a wait task, but also break on retry_event being set
150 + try:
151 + # Wait for the delay or until retry_event is set
152 + wait_task = asyncio.create_task(asyncio.sleep(delay))
153 + retry_task = asyncio.create_task(retry_event.wait())
154 +
155 + done, pending = await asyncio.wait(
156 + [wait_task, retry_task],
157 + return_when=asyncio.FIRST_COMPLETED
158 + )
159 +
160 + # Cancel the pending task
161 + for task in pending:
162 + task.cancel()
163 +
164 + # Clear the event
165 + retry_event.clear()
166 +
167 + except asyncio.CancelledError:
168 + pass
169 +
170 + print(f"{PROGRAM_NAME}: Connecting to {uri}...", file=sys.stderr)
171 +
172 + try:
173 + # Connect with timeout
174 + # In newer versions of websockets, connect() is already awaitable
175 + ws = await asyncio.wait_for(
176 + websockets.connect(
177 + uri,
178 + compression='deflate',
179 + max_size=16*1024*1024,
180 + ping_interval=30, # Send keep-alive pings every 30 seconds
181 + ping_timeout=10, # Wait 10 seconds for pong response
182 + close_timeout=5 # Wait 5 seconds for close frame
183 + ),
184 + timeout=15 # 15 second timeout
185 + )
186 + except asyncio.TimeoutError:
187 + raise Exception("Connection timeout")
188 +
189 + print(f"{PROGRAM_NAME}: Connected", file=sys.stderr)
190 + retry_count = 0 # Reset retry counter on successful connection
191 +
192 + # Clear all pending request timers
193 + for req_id, req_data in list(pending_requests.items()):
194 + if "timer" in req_data and not req_data["timer"].done():
195 + req_data["timer"].cancel()
196 +
197 + # We don't clear pending_requests entirely here, we just clear the timers
198 + # and keep tracking the requests until we get responses
199 +
200 + # Memory management: Limit the size of pending_requests to avoid memory leaks
201 + if len(pending_requests) > 1000:
202 + print(f"{PROGRAM_NAME}: Too many pending requests ({len(pending_requests)}), cleaning up", file=sys.stderr)
203 + pending_requests.clear() # In extreme case, clear everything
204 +
205 + # Processor for stdin messages
206 + async def process_stdin():
207 + try:
208 + while True:
209 + # Get message from queue
210 + line_data = await stdin_queue.get()
211 + line, msg_id, _ = line_data
212 +
213 + # Skip messages that have already timed out and received error responses
214 + if msg_id is not None and msg_id in pending_requests and pending_requests[msg_id].get("timed_out", False):
215 + print(f"{PROGRAM_NAME}: Skipping previously timed-out request with ID {msg_id}", file=sys.stderr)
216 + stdin_queue.task_done()
217 + continue
218 +
219 + # If this is a request with an ID, track it (but no timeout since we're connected)
220 + if msg_id is not None and msg_id not in pending_requests:
221 + pending_requests[msg_id] = {
222 + "message": line,
223 + "sent_time": time.time()
224 + }
225 +
226 + try:
227 + # Send to WebSocket
228 + await ws.send(line)
229 + stdin_queue.task_done()
230 + except websockets.exceptions.ConnectionClosed:
231 + # Put the message back in the queue, unless it already timed out
232 + if not (msg_id is not None and msg_id in pending_requests and pending_requests[msg_id].get("timed_out", False)):
233 + await stdin_queue.put(line_data)
234 + # Re-raise to trigger reconnection
235 + raise
236 + except Exception as e:
237 + # In newer websockets versions, we need to check for close attribute differently
238 + try:
239 + # Check if the connection is still open
240 + if hasattr(ws, 'closed') and not ws.closed:
241 + print(f"{PROGRAM_NAME}: ERROR: stdin processor: {e}", file=sys.stderr)
242 + elif hasattr(ws, 'protocol') and not ws.protocol.closed:
243 + print(f"{PROGRAM_NAME}: ERROR: stdin processor: {e}", file=sys.stderr)
244 + else:
245 + print(f"{PROGRAM_NAME}: Websocket connection closed: {e}", file=sys.stderr)
246 + except:
247 + # If we can't check closed state, just log the error
248 + print(f"{PROGRAM_NAME}: ERROR: stdin processor: {e}", file=sys.stderr)
249 + # Don't propagate the exception, let the connection close
250 + # and reconnection will be triggered
251 +
252 + # Processor for WebSocket messages
253 + async def process_websocket():
254 + try:
255 + async for message in ws:
256 + # Forward WebSocket messages to stdout
257 + print(message, flush=True)
258 +
259 + # Check if this is a response to a tracked request
260 + try:
261 + data = json.loads(message)
262 + if isinstance(data, dict) and "jsonrpc" in data and data.get("jsonrpc") == "2.0" and "id" in data:
263 + msg_id = data.get("id")
264 + if msg_id in pending_requests:
265 + del pending_requests[msg_id]
266 + except:
267 + pass
268 +
269 + # Memory management: Limit the size of stdin_queue to avoid memory leaks
270 + if stdin_queue.qsize() > 1000:
271 + print(f"{PROGRAM_NAME}: WARNING: Very large stdin queue ({stdin_queue.qsize()}), this may indicate a problem", file=sys.stderr)
272 + except Exception as e:
273 + # In newer websockets versions, we need to check for close attribute differently
274 + try:
275 + # Check if the connection is still open
276 + if hasattr(ws, 'closed') and not ws.closed:
277 + print(f"{PROGRAM_NAME}: ERROR: websocket processor: {e}", file=sys.stderr)
278 + elif hasattr(ws, 'protocol') and not ws.protocol.closed:
279 + print(f"{PROGRAM_NAME}: ERROR: websocket processor: {e}", file=sys.stderr)
280 + else:
281 + print(f"{PROGRAM_NAME}: Websocket connection closed: {e}", file=sys.stderr)
282 + except:
283 + # If we can't check closed state, just log the error
284 + print(f"{PROGRAM_NAME}: ERROR: websocket processor: {e}", file=sys.stderr)
285 + # Don't propagate the exception, let the connection close
286 + # and reconnection will be triggered
287 +
288 + # Run both tasks concurrently
289 + stdin_processor = asyncio.create_task(process_stdin())
290 + ws_processor = asyncio.create_task(process_websocket())
291 +
292 + # Wait for either task to complete (which means a failure)
293 + done, pending = await asyncio.wait(
294 + [stdin_processor, ws_processor],
295 + return_when=asyncio.FIRST_COMPLETED
296 + )
297 +
298 + # Cancel the pending task
299 + for task in pending:
300 + task.cancel()
301 + try:
302 + await task
303 + except asyncio.CancelledError:
304 + pass
305 +
306 + # Ensure WebSocket is closed
307 + try:
308 + # Check if websocket is already closed
309 + is_closed = False
310 + if hasattr(ws, 'closed'):
311 + is_closed = ws.closed
312 + elif hasattr(ws, 'protocol'):
313 + is_closed = ws.protocol.closed
314 +
315 + if not is_closed:
316 + await ws.close()
317 + except Exception as e:
318 + print(f"{PROGRAM_NAME}: Error closing websocket: {e}", file=sys.stderr)
319 +
320 + except (websockets.exceptions.ConnectionClosed, websockets.exceptions.WebSocketException) as e:
321 + print(f"{PROGRAM_NAME}: WebSocket error: {e}", file=sys.stderr)
322 + retry_count += 1
323 + except Exception as e:
324 + print(f"{PROGRAM_NAME}: Unexpected error: {e}", file=sys.stderr)
325 + retry_count += 1
326 +
327 +def main():
328 + if len(sys.argv) != 2:
329 + print(f"{PROGRAM_NAME}: Usage: {PROGRAM_NAME} ws://host/path", file=sys.stderr)
330 + sys.exit(1)
331 +
332 + # Set up signal handling
333 + def signal_handler(sig, frame):
334 + print(f"{PROGRAM_NAME}: Received signal {sig}, exiting", file=sys.stderr)
335 + sys.exit(0)
336 +
337 + signal.signal(signal.SIGINT, signal_handler)
338 + signal.signal(signal.SIGTERM, signal_handler)
339 +
340 + try:
341 + asyncio.run(connect_with_backoff(sys.argv[1]))
342 + except KeyboardInterrupt:
343 + print(f"{PROGRAM_NAME}: Interrupted by user, exiting", file=sys.stderr)
344 +
345 + if STDIN_ERROR:
346 + print(f"{PROGRAM_NAME}: Exiting due to stdin error", file=sys.stderr)
347 +
348 +if __name__ == "__main__":
349 + main()
\ No newline at end of file
src/web/mcp/mcp-api-key.c new
+152
@@ -0,0 +1,152 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-api-key.h"
4 +#include "claim/claim.h"
5 +#include <fcntl.h>
6 +#include <sys/stat.h>
7 +#include <unistd.h>
8 +#include <limits.h>
9 +
10 +#ifdef NETDATA_MCP_DEV_PREVIEW_API_KEY
11 +
12 +static char mcp_dev_preview_api_key[MCP_DEV_PREVIEW_API_KEY_LENGTH + 1] = "";
13 +
14 +static bool mcp_api_key_generate_and_save(void) {
15 + nd_uuid_t uuid;
16 + uuid_generate_random(uuid);
17 +
18 + // Unparse directly to the destination buffer
19 + uuid_unparse_lower(uuid, mcp_dev_preview_api_key);
20 +
21 + // Construct full path
22 + char path[PATH_MAX];
23 + snprintf(path, sizeof(path), "%s/%s", netdata_configured_varlib_dir, MCP_DEV_PREVIEW_API_KEY_FILENAME);
24 +
25 + // Open file with O_CREAT | O_EXCL to ensure we don't overwrite
26 + int fd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
27 + if (fd == -1) {
28 + netdata_log_error("MCP: Failed to create API key file %s: %s",
29 + path, strerror(errno));
30 + return false;
31 + }
32 +
33 + // Write the UUID with newline
34 + char buffer[MCP_DEV_PREVIEW_API_KEY_LENGTH + 2]; // +1 for newline, +1 for null
35 + snprintf(buffer, sizeof(buffer), "%s\n", mcp_dev_preview_api_key);
36 +
37 + ssize_t written = write(fd, buffer, MCP_DEV_PREVIEW_API_KEY_LENGTH + 1); // +1 for newline
38 + if (written != (ssize_t)(MCP_DEV_PREVIEW_API_KEY_LENGTH + 1)) {
39 + netdata_log_error("MCP: Failed to write API key to file: %s", strerror(errno));
40 + close(fd);
41 + unlink(path);
42 + return false;
43 + }
44 +
45 + close(fd);
46 +
47 + // Ensure file permissions are correct (only owner can read/write)
48 + if (chmod(path, 0600) == -1) {
49 + netdata_log_error("MCP: Failed to set permissions on API key file: %s", strerror(errno));
50 + unlink(path);
51 + return false;
52 + }
53 +
54 + netdata_log_info("MCP: Generated new developer preview API key");
55 + return true;
56 +}
57 +
58 +static bool mcp_api_key_load(void) {
59 + // Construct full path
60 + char path[PATH_MAX];
61 + snprintf(path, sizeof(path), "%s/%s", netdata_configured_varlib_dir, MCP_DEV_PREVIEW_API_KEY_FILENAME);
62 +
63 + int fd = open(path, O_RDONLY);
64 + if (fd == -1) {
65 + if (errno == ENOENT) {
66 + // File doesn't exist, this is expected on first run
67 + return false;
68 + }
69 + netdata_log_error("MCP: Failed to open API key file %s: %s",
70 + path, strerror(errno));
71 + return false;
72 + }
73 +
74 + char buffer[MCP_DEV_PREVIEW_API_KEY_LENGTH + 2]; // +1 for potential newline, +1 for null
75 + ssize_t bytes_read = read(fd, buffer, sizeof(buffer) - 1);
76 + close(fd);
77 +
78 + if (bytes_read < MCP_DEV_PREVIEW_API_KEY_LENGTH || bytes_read > MCP_DEV_PREVIEW_API_KEY_LENGTH + 1) {
79 + netdata_log_error("MCP: Invalid API key file size: expected %d or %d bytes, got %zd",
80 + MCP_DEV_PREVIEW_API_KEY_LENGTH, MCP_DEV_PREVIEW_API_KEY_LENGTH + 1, bytes_read);
81 + return false;
82 + }
83 +
84 + buffer[bytes_read] = '\0';
85 +
86 + // Strip trailing newline if present
87 + if (bytes_read > 0 && buffer[bytes_read - 1] == '\n') {
88 + buffer[bytes_read - 1] = '\0';
89 + }
90 +
91 + // Basic validation - should be a valid UUID format
92 + nd_uuid_t uuid;
93 + if (uuid_parse(buffer, uuid) != 0) {
94 + netdata_log_error("MCP: Invalid UUID format in API key file");
95 + return false;
96 + }
97 +
98 + strncpy(mcp_dev_preview_api_key, buffer, MCP_DEV_PREVIEW_API_KEY_LENGTH);
99 + mcp_dev_preview_api_key[MCP_DEV_PREVIEW_API_KEY_LENGTH] = '\0';
100 +
101 + netdata_log_info("MCP: Loaded developer preview API key");
102 + return true;
103 +}
104 +
105 +void mcp_api_key_initialize(void) {
106 + // Try to load existing key first
107 + if (!mcp_api_key_load()) {
108 + // If loading fails, generate a new one
109 + if (!mcp_api_key_generate_and_save()) {
110 + netdata_log_error("MCP: Failed to initialize API key system");
111 + return;
112 + }
113 + }
114 +
115 + char path[PATH_MAX];
116 + snprintf(path, sizeof(path), "%s/%s", netdata_configured_varlib_dir, MCP_DEV_PREVIEW_API_KEY_FILENAME);
117 + netdata_log_info("MCP: Developer preview API key initialized. Location: %s", path);
118 +}
119 +
120 +bool mcp_api_key_verify(const char *api_key) {
121 + if (!api_key || !*api_key) {
122 + netdata_log_error("MCP: No API key provided");
123 + return false;
124 + }
125 +
126 + // Check if agent is claimed
127 + if (!is_agent_claimed()) {
128 + netdata_log_error("MCP: API key authentication rejected - agent is not claimed to Netdata Cloud");
129 + return false;
130 + }
131 +
132 + // Check if we have a loaded API key
133 + if (!mcp_dev_preview_api_key[0]) {
134 + netdata_log_error("MCP: No API key loaded");
135 + return false;
136 + }
137 +
138 + // Compare the keys
139 + bool valid = (strcmp(api_key, mcp_dev_preview_api_key) == 0);
140 +
141 + if (!valid) {
142 + netdata_log_error("MCP: Invalid API key provided");
143 + }
144 +
145 + return valid;
146 +}
147 +
148 +const char *mcp_api_key_get(void) {
149 + return mcp_dev_preview_api_key;
150 +}
151 +
152 +#endif // NETDATA_MCP_DEV_PREVIEW_API_KEY
\ No newline at end of file
src/web/mcp/mcp-api-key.h new
+28
@@ -0,0 +1,28 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_API_KEY_H
4 +#define NETDATA_MCP_API_KEY_H
5 +
6 +#include "daemon/common.h"
7 +
8 +#define NETDATA_MCP_DEV_PREVIEW_API_KEY 1
9 +
10 +#ifdef NETDATA_MCP_DEV_PREVIEW_API_KEY
11 +
12 +#define MCP_DEV_PREVIEW_API_KEY_FILENAME "mcp_dev_preview_api_key"
13 +#define MCP_DEV_PREVIEW_API_KEY_LENGTH 36 // UUID format: 8-4-4-4-12 = 32 hex chars + 4 hyphens
14 +
15 +// Initialize the MCP API key subsystem - creates key file if it doesn't exist
16 +void mcp_api_key_initialize(void);
17 +
18 +// Verify if the provided API key matches the stored one
19 +// Returns true if valid and agent is claimed, false otherwise
20 +bool mcp_api_key_verify(const char *api_key);
21 +
22 +// Get the current API key (for display purposes)
23 +// Returns a static buffer that should not be freed
24 +const char *mcp_api_key_get(void);
25 +
26 +#endif // NETDATA_MCP_DEV_PREVIEW_API_KEY
27 +
28 +#endif // NETDATA_MCP_API_KEY_H
\ No newline at end of file
src/web/mcp/mcp-completion.c new
+113
@@ -0,0 +1,113 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +/**
4 + * MCP Completion Namespace
5 + *
6 + * The MCP Completion namespace provides methods for handling input argument completions.
7 + * In the MCP protocol, completion methods allow clients to request suggestions
8 + * for input fields, providing a better user experience when interacting with tools,
9 + * resources, or prompts.
10 + *
11 + * Standard methods in the MCP specification:
12 + *
13 + * 1. completion/complete - Requests completion suggestions for an input argument
14 + * - Takes an argument name, current value, and reference context
15 + * - Reference can be to a tool, resource, or prompt
16 + * - Returns an array of possible completion values
17 + * - May include pagination information for large result sets
18 + *
19 + * The completion namespace enhances user interactions by providing
20 + * real-time suggestions for input fields, improving usability
21 + * and reducing errors when working with complex arguments.
22 + *
23 + * Completions are context-aware, meaning they take into account what the
24 + * completion is for (which tool, resource, or prompt) and provide
25 + * relevant suggestions based on that context.
26 + */
27 +
28 +#include "mcp-completion.h"
29 +
30 +// Implementation of completion/complete (transport-agnostic)
31 +static MCP_RETURN_CODE mcp_completion_method_complete(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
32 + if (!mcpc || id == 0) return MCP_RC_ERROR;
33 +
34 + // Extract argument and ref parameters
35 + struct json_object *argument_obj = NULL;
36 + struct json_object *ref_obj = NULL;
37 +
38 + if (!json_object_object_get_ex(params, "argument", &argument_obj)) {
39 + buffer_sprintf(mcpc->error, "Missing required parameter 'argument'");
40 + return MCP_RC_BAD_REQUEST;
41 + }
42 +
43 + if (!json_object_object_get_ex(params, "ref", &ref_obj)) {
44 + buffer_sprintf(mcpc->error, "Missing required parameter 'ref'");
45 + return MCP_RC_BAD_REQUEST;
46 + }
47 +
48 + // Extract argument name and value
49 + struct json_object *name_obj = NULL;
50 + struct json_object *value_obj = NULL;
51 +
52 + if (!json_object_object_get_ex(argument_obj, "name", &name_obj)) {
53 + buffer_sprintf(mcpc->error, "Missing required parameter 'argument.name'");
54 + return MCP_RC_BAD_REQUEST;
55 + }
56 +
57 + if (!json_object_object_get_ex(argument_obj, "value", &value_obj)) {
58 + buffer_sprintf(mcpc->error, "Missing required parameter 'argument.value'");
59 + return MCP_RC_BAD_REQUEST;
60 + }
61 +
62 + const char *name = json_object_get_string(name_obj);
63 + const char *value = json_object_get_string(value_obj);
64 +
65 + // Log that we received this request (actual implementation would generate completion options)
66 + netdata_log_info("MCP received completion/complete request for argument '%s' with value '%s'", name, value);
67 +
68 + // Initialize success response
69 + mcp_init_success_result(mcpc, id);
70 +
71 + // Add completion data (sample implementation with a few static options)
72 + buffer_json_member_add_object(mcpc->result, "completion");
73 +
74 + // Add values-array with sample completion options
75 + buffer_json_member_add_array(mcpc->result, "values");
76 + buffer_json_add_array_item_string(mcpc->result, "option1");
77 + buffer_json_add_array_item_string(mcpc->result, "option2");
78 + buffer_json_add_array_item_string(mcpc->result, "option3");
79 + buffer_json_array_close(mcpc->result); // Close values array
80 +
81 + // Add optional fields
82 + buffer_json_member_add_boolean(mcpc->result, "hasMore", false);
83 + buffer_json_member_add_int64(mcpc->result, "total", 3);
84 +
85 + buffer_json_object_close(mcpc->result); // Close completion object
86 + buffer_json_finalize(mcpc->result);
87 +
88 + return MCP_RC_OK;
89 +}
90 +
91 +// Completion namespace method dispatcher (transport-agnostic)
92 +MCP_RETURN_CODE mcp_completion_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id) {
93 + if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
94 +
95 + netdata_log_debug(D_MCP, "MCP completion method: %s", method);
96 +
97 + // Flush previous buffers
98 + buffer_flush(mcpc->result);
99 + buffer_flush(mcpc->error);
100 +
101 + MCP_RETURN_CODE rc;
102 +
103 + if (strcmp(method, "complete") == 0) {
104 + rc = mcp_completion_method_complete(mcpc, params, id);
105 + }
106 + else {
107 + // Method not found in completion namespace
108 + buffer_sprintf(mcpc->error, "Method 'completion/%s' not supported. The MCP specification only defines 'complete' method.", method);
109 + rc = MCP_RC_NOT_IMPLEMENTED;
110 + }
111 +
112 + return rc;
113 +}
\ No newline at end of file
src/web/mcp/mcp-completion.h new
+11
@@ -0,0 +1,11 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_COMPLETION_H
4 +#define NETDATA_MCP_COMPLETION_H
5 +
6 +#include "mcp.h"
7 +
8 +// Completion namespace method dispatcher (transport-agnostic)
9 +MCP_RETURN_CODE mcp_completion_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id);
10 +
11 +#endif // NETDATA_MCP_COMPLETION_H
\ No newline at end of file
src/web/mcp/mcp-context.c deleted
-102
@@ -1,102 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -/**
4 - * MCP Context Namespace
5 - *
6 - * The MCP Context namespace provides methods for managing contextual information
7 - * exchanged between clients and servers. Context represents stateful information
8 - * that enhances the interaction between components.
9 - *
10 - * Key features of the context namespace:
11 - *
12 - * 1. Context Management:
13 - * - Provide contextual information to the server (context/provide)
14 - * - Clear specific context data (context/clear)
15 - * - Check the status of current context (context/status)
16 - *
17 - * 2. Context Persistence:
18 - * - Save context for future use (context/save)
19 - * - Load previously saved context (context/load)
20 - *
21 - * Context in MCP can include:
22 - * - User preferences and settings
23 - * - Session-specific information
24 - * - Authentication and authorization details
25 - * - Client capabilities and limitations
26 - * - Conversation or interaction history
27 - *
28 - * In the Netdata environment, context might include:
29 - * - User display preferences (theme, date formats, etc.)
30 - * - View configurations (dashboard layouts, chart settings)
31 - * - Filtering and query preferences
32 - * - Historical interaction patterns
33 - * - Authentication tokens and permissions
34 - *
35 - * Context can be transient (per session) or persistent (saved across sessions),
36 - * and may be scoped to specific interactions or broadly applied.
37 - */
38 -
39 -#include "mcp-context.h"
40 -#include "mcp-initialize.h"
41 -
42 -// Stub implementations for all context namespace methods (transport-agnostic)
43 -static MCP_RETURN_CODE mcp_context_method_provide(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
44 - buffer_sprintf(mcpc->error, "Method 'context/provide' not implemented yet");
45 - return MCP_RC_NOT_IMPLEMENTED;
46 -}
47 -
48 -static MCP_RETURN_CODE mcp_context_method_clear(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
49 - buffer_sprintf(mcpc->error, "Method 'context/clear' not implemented yet");
50 - return MCP_RC_NOT_IMPLEMENTED;
51 -}
52 -
53 -static MCP_RETURN_CODE mcp_context_method_status(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
54 - buffer_sprintf(mcpc->error, "Method 'context/status' not implemented yet");
55 - return MCP_RC_NOT_IMPLEMENTED;
56 -}
57 -
58 -static MCP_RETURN_CODE mcp_context_method_save(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
59 - buffer_sprintf(mcpc->error, "Method 'context/save' not implemented yet");
60 - return MCP_RC_NOT_IMPLEMENTED;
61 -}
62 -
63 -static MCP_RETURN_CODE mcp_context_method_load(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
64 - buffer_sprintf(mcpc->error, "Method 'context/load' not implemented yet");
65 - return MCP_RC_NOT_IMPLEMENTED;
66 -}
67 -
68 -// Context namespace method dispatcher (transport-agnostic)
69 -MCP_RETURN_CODE mcp_context_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id) {
70 - if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
71 -
72 - netdata_log_debug(D_MCP, "MCP context method: %s", method);
73 -
74 - // Flush previous buffers
75 - buffer_flush(mcpc->result);
76 - buffer_flush(mcpc->error);
77 -
78 - MCP_RETURN_CODE rc;
79 -
80 - if (strcmp(method, "provide") == 0) {
81 - rc = mcp_context_method_provide(mcpc, params, id);
82 - }
83 - else if (strcmp(method, "clear") == 0) {
84 - rc = mcp_context_method_clear(mcpc, params, id);
85 - }
86 - else if (strcmp(method, "status") == 0) {
87 - rc = mcp_context_method_status(mcpc, params, id);
88 - }
89 - else if (strcmp(method, "save") == 0) {
90 - rc = mcp_context_method_save(mcpc, params, id);
91 - }
92 - else if (strcmp(method, "load") == 0) {
93 - rc = mcp_context_method_load(mcpc, params, id);
94 - }
95 - else {
96 - // Method not found in context namespace
97 - buffer_sprintf(mcpc->error, "Method 'context/%s' not implemented yet", method);
98 - rc = MCP_RC_NOT_IMPLEMENTED;
99 - }
100 -
101 - return rc;
102 -}
src/web/mcp/mcp-context.h deleted
-11
@@ -1,11 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -#ifndef NETDATA_MCP_CONTEXT_H
4 -#define NETDATA_MCP_CONTEXT_H
5 -
6 -#include "mcp.h"
7 -
8 -// Context namespace method dispatcher (transport-agnostic)
9 -MCP_RETURN_CODE mcp_context_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id);
10 -
11 -#endif // NETDATA_MCP_CONTEXT_H
src/web/mcp/mcp-initialize.c
+37 -137
@@ -1,16 +1,41 @@
1 // SPDX-License-Identifier: GPL-3.0-or-later
2
3 +/**
4 + * MCP Initialize Method
5 + *
6 + * The initialize method is a core part of the Model Context Protocol (MCP),
7 + * serving as the initial handshake between client and server.
8 + *
9 + * According to the MCP specification:
10 + *
11 + * 1. Purpose:
12 + * - Establishes the protocol version to use for communication
13 + * - Provides information about server capabilities
14 + * - Exchanges client and server metadata
15 + * - Sets up the foundation for subsequent interactions
16 + *
17 + * 2. Protocol flow:
18 + * - The client sends an initialize request with its supported protocol version
19 + * - The server responds with its capabilities and selected protocol version
20 + * - After successful initialization, other methods become available
21 + *
22 + * 3. Key components in the response:
23 + * - protocolVersion: The protocol version the server will use
24 + * - capabilities: A structured object describing supported features
25 + * - serverInfo: Information about the server implementation
26 + *
27 + * This method must be called before any other MCP method, and handles
28 + * protocol version negotiation and capability discovery.
29 + */
30 +
31 #include "mcp-initialize.h"
32 #include "database/rrd-metadata.h"
5 -#include "database/rrd-retention.h"
33 #include "daemon/common.h"
34
35 // Initialize handler - provides information about what's available (transport-agnostic)
9 -MCP_RETURN_CODE mcp_method_initialize(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
10 - if (!mcpc) {
11 - buffer_strcat(mcpc->error, "Invalid MCP client context");
36 +MCP_RETURN_CODE mcp_method_initialize(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
37 + if (!mcpc)
38 return MCP_RC_ERROR;
13 - }
39
40 // Extract client's requested protocol version
41 struct json_object *protocol_version_obj = NULL;
@@ -38,12 +63,7 @@ MCP_RETURN_CODE mcp_method_initialize(MCP_CLIENT *mcpc, struct json_object *para
63
64 // Use rrdstats_metadata_collect to get infrastructure statistics
65 RRDSTATS_METADATA metadata = rrdstats_metadata_collect();
41 -
42 - // Use rrdstats_retention_collect to get retention information
43 - RRDSTATS_RETENTION retention = rrdstats_retention_collect();
66
45 - buffer_json_member_add_object(mcpc->result, "result");
46 -
67 // Add protocol version based on what client requested
68 buffer_json_member_add_string(mcpc->result, "protocolVersion",
69 MCP_PROTOCOL_VERSION_2str(mcpc->protocol_version));
@@ -63,28 +83,28 @@ MCP_RETURN_CODE mcp_method_initialize(MCP_CLIENT *mcpc, struct json_object *para
83 buffer_json_member_add_boolean(mcpc->result, "asyncExecution", true);
84 buffer_json_member_add_boolean(mcpc->result, "batchExecution", true);
85 buffer_json_object_close(mcpc->result); // Close tools
66 -
86 +
87 // Resources capabilities
88 buffer_json_member_add_object(mcpc->result, "resources");
89 buffer_json_member_add_boolean(mcpc->result, "listChanged", true);
90 buffer_json_member_add_boolean(mcpc->result, "subscribe", true);
91 buffer_json_object_close(mcpc->result); // Close resources
72 -
92 +
93 // Prompts capabilities
94 buffer_json_member_add_object(mcpc->result, "prompts");
95 buffer_json_member_add_boolean(mcpc->result, "listChanged", false);
96 buffer_json_object_close(mcpc->result); // Close prompts
77 -
97 +
98 // Notification capabilities
99 buffer_json_member_add_object(mcpc->result, "notifications");
100 buffer_json_member_add_boolean(mcpc->result, "push", true);
101 buffer_json_member_add_boolean(mcpc->result, "subscription", true);
102 buffer_json_object_close(mcpc->result); // Close notifications
83 -
103 +
104 // Add logging capabilities
105 buffer_json_member_add_object(mcpc->result, "logging");
106 buffer_json_object_close(mcpc->result); // Close logging
87 -
107 +
108 // Add version-specific capabilities
109 if (mcpc->protocol_version >= MCP_PROTOCOL_VERSION_2025_03_26) {
110 // Add completions capability - new in 2025-03-26
@@ -119,131 +139,11 @@ MCP_RETURN_CODE mcp_method_initialize(MCP_CLIENT *mcpc, struct json_object *para
139 snprintfz(instructions, sizeof(instructions),
140 "This is Netdata on a Standalone Server.\n\n%s", common);
141 }
122 -
142 +
143 buffer_json_member_add_string(mcpc->result, "instructions", instructions);
144
125 - // Add _meta field (optional)
145 + // Add _meta field (optional) - empty as requested
146 buffer_json_member_add_object(mcpc->result, "_meta");
127 - buffer_json_member_add_string(mcpc->result, "generator", "netdata");
128 -
129 - // Get current time and calculate uptimes
130 - time_t now = now_realtime_sec();
131 - time_t system_uptime_seconds = now_boottime_sec();
132 - time_t netdata_uptime_seconds = now - netdata_start_time;
133 -
134 - buffer_json_member_add_int64(mcpc->result, "timestamp", (int64_t)now);
135 -
136 - // Add system uptime info - both raw seconds and human-readable format
137 - char human_readable[128];
138 - duration_snprintf_time_t(human_readable, sizeof(human_readable), system_uptime_seconds);
139 -
140 - buffer_json_member_add_object(mcpc->result, "system_uptime");
141 - buffer_json_member_add_int64(mcpc->result, "seconds", (int64_t)system_uptime_seconds);
142 - buffer_json_member_add_string(mcpc->result, "human", human_readable);
143 - buffer_json_object_close(mcpc->result); // Close system_uptime
144 -
145 - // Add netdata uptime info - both raw seconds and human-readable format
146 - duration_snprintf_time_t(human_readable, sizeof(human_readable), netdata_uptime_seconds);
147 -
148 - buffer_json_member_add_object(mcpc->result, "netdata_uptime");
149 - buffer_json_member_add_int64(mcpc->result, "seconds", (int64_t)netdata_uptime_seconds);
150 - buffer_json_member_add_string(mcpc->result, "human", human_readable);
151 - buffer_json_object_close(mcpc->result); // Close netdata_uptime
152 -
153 - // Add infrastructure statistics to metadata
154 - buffer_json_member_add_object(mcpc->result, "infrastructure");
155 -
156 - // Add nodes statistics
157 - buffer_json_member_add_object(mcpc->result, "nodes");
158 - buffer_json_member_add_int64(mcpc->result, "total", metadata.nodes.total);
159 - buffer_json_member_add_int64(mcpc->result, "receiving_from_children", metadata.nodes.receiving);
160 - buffer_json_member_add_int64(mcpc->result, "sending_to_next_parent", metadata.nodes.sending);
161 - buffer_json_member_add_int64(mcpc->result, "archived_but_available_for_queries", metadata.nodes.archived);
162 - buffer_json_member_add_string(mcpc->result, "info", "Nodes (or hosts, or servers, or devices) are Netdata Agent installations or virtual Netdata nodes or SNMP devices.");
163 - buffer_json_object_close(mcpc->result); // Close nodes
164 -
165 - // Add metrics statistics
166 - buffer_json_member_add_object(mcpc->result, "metrics");
167 - buffer_json_member_add_int64(mcpc->result, "currently_being_collected", metadata.metrics.collected);
168 - buffer_json_member_add_int64(mcpc->result, "total_available_for_queries", metadata.metrics.available);
169 - buffer_json_member_add_string(mcpc->result, "info", "Metrics are unique time-series in the Netdata time-series database.");
170 - buffer_json_object_close(mcpc->result); // Close metrics
171 -
172 - // Add instances statistics
173 - buffer_json_member_add_object(mcpc->result, "instances");
174 - buffer_json_member_add_int64(mcpc->result, "currently_being_collected", metadata.instances.collected);
175 - buffer_json_member_add_int64(mcpc->result, "total_available_for_queries", metadata.instances.available);
176 - buffer_json_member_add_string(mcpc->result, "info", "Instances are collections of metrics referring to a component (system, disk, network interface, application, process, container, etc).");
177 - buffer_json_object_close(mcpc->result); // Close instances
178 -
179 - // Add contexts statistics
180 - buffer_json_member_add_object(mcpc->result, "contexts");
181 - buffer_json_member_add_int64(mcpc->result, "unique_across_all_nodes", metadata.contexts.unique);
182 - buffer_json_member_add_string(mcpc->result, "info", "Contexts are distinct charts shown on the Netdata dashboards, like system.cpu (system CPU utilization), or net.net (network interfaces bandwidth). When monitoring applications, the context usually includes the application name.");
183 - buffer_json_object_close(mcpc->result); // Close contexts
184 -
185 - // Add retention information
186 - if (retention.storage_tiers > 0) {
187 - buffer_json_member_add_object(mcpc->result, "retention");
188 - buffer_json_member_add_array(mcpc->result, "tiers");
189 -
190 - for (size_t i = 0; i < retention.storage_tiers; i++) {
191 - RRD_STORAGE_TIER *tier_info = &retention.tiers[i];
192 -
193 - // Skip empty tiers
194 - if (tier_info->metrics == 0 && tier_info->samples == 0)
195 - continue;
196 -
197 - buffer_json_add_array_item_object(mcpc->result);
198 -
199 - // Add basic tier info
200 - buffer_json_member_add_int64(mcpc->result, "tier", tier_info->tier);
201 - buffer_json_member_add_string(mcpc->result, "backend",
202 - tier_info->backend == STORAGE_ENGINE_BACKEND_DBENGINE ? "dbengine" :
203 - tier_info->backend == STORAGE_ENGINE_BACKEND_RRDDIM ? "ram" : "unknown");
204 - buffer_json_member_add_int64(mcpc->result, "granularity", tier_info->group_seconds);
205 - buffer_json_member_add_string(mcpc->result, "granularity_human", tier_info->granularity_human);
206 -
207 - // Add metrics info
208 - buffer_json_member_add_int64(mcpc->result, "metrics", tier_info->metrics);
209 - buffer_json_member_add_int64(mcpc->result, "samples", tier_info->samples);
210 -
211 - // Add storage info when available
212 - if (tier_info->disk_max > 0) {
213 - buffer_json_member_add_int64(mcpc->result, "disk_used", tier_info->disk_used);
214 - buffer_json_member_add_int64(mcpc->result, "disk_max", tier_info->disk_max);
215 - // Format disk_percent to have only 2 decimal places
216 - double rounded_percent = floor(tier_info->disk_percent * 100.0 + 0.5) / 100.0;
217 - buffer_json_member_add_double(mcpc->result, "disk_percent", rounded_percent);
218 - }
219 -
220 - // Add retention info
221 - if (tier_info->retention > 0) {
222 - buffer_json_member_add_int64(mcpc->result, "first_time_s", tier_info->first_time_s);
223 - buffer_json_member_add_int64(mcpc->result, "last_time_s", tier_info->last_time_s);
224 - buffer_json_member_add_int64(mcpc->result, "retention", tier_info->retention);
225 - buffer_json_member_add_string(mcpc->result, "retention_human", tier_info->retention_human);
226 -
227 - if (tier_info->requested_retention > 0) {
228 - buffer_json_member_add_int64(mcpc->result, "requested_retention", tier_info->requested_retention);
229 - buffer_json_member_add_string(mcpc->result, "requested_retention_human", tier_info->requested_retention_human);
230 - }
231 -
232 - if (tier_info->expected_retention > 0) {
233 - buffer_json_member_add_int64(mcpc->result, "expected_retention", tier_info->expected_retention);
234 - buffer_json_member_add_string(mcpc->result, "expected_retention_human", tier_info->expected_retention_human);
235 - }
236 - }
237 -
238 - buffer_json_object_close(mcpc->result); // Close tier object
239 - }
240 -
241 - buffer_json_array_close(mcpc->result); // Close tiers array
242 - buffer_json_member_add_string(mcpc->result, "info", "Metrics retention information for each storage tier in the Netdata database.\nHigher tiers can provide min, max, average, sum and anomaly rate with the same accuracy as tier 0.\nTiers are automatically selected during query.");
243 - buffer_json_object_close(mcpc->result); // Close retention
244 - }
245 -
246 - buffer_json_object_close(mcpc->result); // Close infrastructure
147 buffer_json_object_close(mcpc->result); // Close _meta
148 buffer_json_object_close(mcpc->result); // Close result object
149 buffer_json_finalize(mcpc->result); // Finalize JSON
src/web/mcp/mcp-initialize.h
+1 -1
@@ -6,6 +6,6 @@
6 #include "mcp.h"
7
8 // Initialize method handler (transport-agnostic)
9 -MCP_RETURN_CODE mcp_method_initialize(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id);
9 +MCP_RETURN_CODE mcp_method_initialize(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
10
11 #endif // NETDATA_MCP_INITIALIZE_H
\ No newline at end of file
src/web/mcp/mcp-logging.c new
+97
@@ -0,0 +1,97 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +/**
4 + * MCP Logging Namespace
5 + *
6 + * The MCP Logging namespace provides methods for controlling server logging behavior.
7 + * In the MCP protocol, logging methods allow clients to control the verbosity
8 + * of logging information shared by the server through notifications.
9 + *
10 + * Standard methods in the MCP specification:
11 + *
12 + * 1. logging/setLevel - Sets the logging level for the server
13 + * - Takes a logging level parameter (e.g., "debug", "info", "warning", "error")
14 + * - Configures which severity of log messages the server should send
15 + * - Does not affect internal server logging, only what's sent to the client
16 + * - Server responds with an empty result on success
17 + *
18 + * The logging namespace helps clients control debugging information
19 + * and monitor server activity by adjusting the amount of logging
20 + * information provided by the server. After setting the level,
21 + * clients may receive logging notifications matching that level.
22 + *
23 + * The server logs are sent as notifications/message events, providing
24 + * real-time monitoring of server operations for debugging purposes.
25 + */
26 +
27 +#include "mcp.h"
28 +#include "mcp-logging.h"
29 +
30 +// Implementation of logging/setLevel (transport-agnostic)
31 +static MCP_RETURN_CODE mcp_logging_method_setLevel(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
32 + if (!mcpc || id == 0)
33 + return MCP_RC_ERROR;
34 +
35 + // Extract level parameter
36 + struct json_object *level_obj = NULL;
37 + if (!json_object_object_get_ex(params, "level", &level_obj)) {
38 + buffer_sprintf(mcpc->error, "Missing required parameter 'level'");
39 + return MCP_RC_BAD_REQUEST;
40 + }
41 +
42 + if (!json_object_is_type(level_obj, json_type_string)) {
43 + buffer_sprintf(mcpc->error, "Parameter 'level' must be a string");
44 + return MCP_RC_BAD_REQUEST;
45 + }
46 +
47 + const char *level = json_object_get_string(level_obj);
48 +
49 + // Validate log level
50 + if (!level || !*level) {
51 + buffer_sprintf(mcpc->error, "Log level cannot be empty");
52 + return MCP_RC_BAD_REQUEST;
53 + }
54 +
55 + // Parse the log level using the enum string mapping
56 + MCP_LOGGING_LEVEL parsed_level = MCP_LOGGING_LEVEL_2id(level);
57 + if (parsed_level == MCP_LOGGING_LEVEL_UNKNOWN) {
58 + buffer_sprintf(mcpc->error, "Invalid log level: '%s'. Valid values are: debug, info, notice, warning, error, critical, alert, emergency", level);
59 + return MCP_RC_BAD_REQUEST;
60 + }
61 +
62 + // Store the parsed log level in the client context
63 + mcpc->logging_level = parsed_level;
64 +
65 + // Log that we received this request and set the level
66 + netdata_log_info("MCP client %s logging level set to: %s", string2str(mcpc->client_name), MCP_LOGGING_LEVEL_2str(parsed_level));
67 +
68 + // Initialize success' response with an empty result object
69 + mcp_init_success_result(mcpc, id);
70 + buffer_json_finalize(mcpc->result);
71 +
72 + return MCP_RC_OK;
73 +}
74 +
75 +// Logging namespace method dispatcher (transport-agnostic)
76 +MCP_RETURN_CODE mcp_logging_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id) {
77 + if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
78 +
79 + netdata_log_debug(D_MCP, "MCP logging method: %s", method);
80 +
81 + // Flush previous buffers
82 + buffer_flush(mcpc->result);
83 + buffer_flush(mcpc->error);
84 +
85 + MCP_RETURN_CODE rc;
86 +
87 + if (strcmp(method, "setLevel") == 0) {
88 + rc = mcp_logging_method_setLevel(mcpc, params, id);
89 + }
90 + else {
91 + // Method not found in logging namespace
92 + buffer_sprintf(mcpc->error, "Method 'logging/%s' not supported. The MCP specification only defines 'setLevel' method.", method);
93 + rc = MCP_RC_NOT_IMPLEMENTED;
94 + }
95 +
96 + return rc;
97 +}
\ No newline at end of file
src/web/mcp/mcp-logging.h new
+11
@@ -0,0 +1,11 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_LOGGING_H
4 +#define NETDATA_MCP_LOGGING_H
5 +
6 +#include "mcp.h"
7 +
8 +// Logging namespace method dispatcher (transport-agnostic)
9 +MCP_RETURN_CODE mcp_logging_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id);
10 +
11 +#endif // NETDATA_MCP_LOGGING_H
\ No newline at end of file
src/web/mcp/mcp-notifications.c deleted
-131
@@ -1,131 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -/**
4 - * MCP Notifications Namespace
5 - *
6 - * The MCP Notifications namespace provides methods for managing and handling notifications.
7 - * In the MCP protocol, notifications enable real-time communication of events from server to client,
8 - * and from client to server.
9 - *
10 - * Key features of the notifications namespace:
11 - *
12 - * 1. Initialization:
13 - * - Clients notify the server when they're initialized (notifications/initialized)
14 - * - This initiates the notification subsystem
15 - *
16 - * 2. Subscription Management:
17 - * - Subscribe to specific notification types (notifications/subscribe)
18 - * - Unsubscribe from notifications (notifications/unsubscribe)
19 - * - Configure notification settings (notifications/getSettings)
20 - *
21 - * 3. Notification Handling:
22 - * - Acknowledge received notifications (notifications/acknowledge)
23 - * - View notification history (notifications/getHistory)
24 - * - Send notifications from client to server (notifications/send)
25 - *
26 - * Notifications in MCP are bidirectional:
27 - * - Server-to-client notifications inform about system events, alerts, changes
28 - * - Client-to-server notifications provide user actions and status updates
29 - *
30 - * In the Netdata context, notifications include:
31 - * - Health monitoring alerts
32 - * - System status changes
33 - * - Configuration changes
34 - * - Resource availability updates
35 - * - Client status updates
36 - *
37 - * Notifications can be transient or persistent, prioritized, and may require
38 - * acknowledgment depending on their type and importance.
39 - */
40 -
41 -#include "mcp-notifications.h"
42 -#include "mcp-initialize.h"
43 -
44 -// Implementation of notifications/initialized (transport-agnostic)
45 -static MCP_RETURN_CODE mcp_notifications_method_initialized(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
46 - // This is just a notification, just log it
47 - netdata_log_debug(D_MCP, "Client sent notifications/initialized notification");
48 -
49 - // No response needed if this is a notification (id == 0)
50 - if (id == 0) return MCP_RC_OK;
51 -
52 - // If it was a request (has id), send an empty success response
53 - mcp_init_success_result(mcpc, id);
54 - buffer_json_finalize(mcpc->result);
55 -
56 - return MCP_RC_OK;
57 -}
58 -
59 -// Stub implementations for other notifications methods (transport-agnostic)
60 -static MCP_RETURN_CODE mcp_notifications_method_subscribe(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
61 - buffer_sprintf(mcpc->error, "Method 'notifications/subscribe' not implemented yet");
62 - return MCP_RC_NOT_IMPLEMENTED;
63 -}
64 -
65 -static MCP_RETURN_CODE mcp_notifications_method_unsubscribe(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
66 - buffer_sprintf(mcpc->error, "Method 'notifications/unsubscribe' not implemented yet");
67 - return MCP_RC_NOT_IMPLEMENTED;
68 -}
69 -
70 -static MCP_RETURN_CODE mcp_notifications_method_acknowledge(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
71 - buffer_sprintf(mcpc->error, "Method 'notifications/acknowledge' not implemented yet");
72 - return MCP_RC_NOT_IMPLEMENTED;
73 -}
74 -
75 -static MCP_RETURN_CODE mcp_notifications_method_getHistory(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
76 - buffer_sprintf(mcpc->error, "Method 'notifications/getHistory' not implemented yet");
77 - return MCP_RC_NOT_IMPLEMENTED;
78 -}
79 -
80 -static MCP_RETURN_CODE mcp_notifications_method_send(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
81 - buffer_sprintf(mcpc->error, "Method 'notifications/send' not implemented yet");
82 - return MCP_RC_NOT_IMPLEMENTED;
83 -}
84 -
85 -static MCP_RETURN_CODE mcp_notifications_method_getSettings(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
86 - buffer_sprintf(mcpc->error, "Method 'notifications/getSettings' not implemented yet");
87 - return MCP_RC_NOT_IMPLEMENTED;
88 -}
89 -
90 -// Notifications namespace method dispatcher (transport-agnostic)
91 -MCP_RETURN_CODE mcp_notifications_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id) {
92 - if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
93 -
94 - netdata_log_debug(D_MCP, "MCP notifications method: %s", method);
95 -
96 - // Flush previous buffers
97 - buffer_flush(mcpc->result);
98 - buffer_flush(mcpc->error);
99 -
100 - MCP_RETURN_CODE rc;
101 -
102 - if (strcmp(method, "initialized") == 0) {
103 - rc = mcp_notifications_method_initialized(mcpc, params, id);
104 - }
105 - else if (strcmp(method, "subscribe") == 0) {
106 - rc = mcp_notifications_method_subscribe(mcpc, params, id);
107 - }
108 - else if (strcmp(method, "unsubscribe") == 0) {
109 - rc = mcp_notifications_method_unsubscribe(mcpc, params, id);
110 - }
111 - else if (strcmp(method, "acknowledge") == 0) {
112 - rc = mcp_notifications_method_acknowledge(mcpc, params, id);
113 - }
114 - else if (strcmp(method, "getHistory") == 0) {
115 - rc = mcp_notifications_method_getHistory(mcpc, params, id);
116 - }
117 - else if (strcmp(method, "send") == 0) {
118 - rc = mcp_notifications_method_send(mcpc, params, id);
119 - }
120 - else if (strcmp(method, "getSettings") == 0) {
121 - rc = mcp_notifications_method_getSettings(mcpc, params, id);
122 - }
123 - else {
124 - // Method not found in notifications namespace
125 - buffer_sprintf(mcpc->error, "Method 'notifications/%s' not implemented yet", method);
126 - rc = MCP_RC_NOT_IMPLEMENTED;
127 - }
128 -
129 - return rc;
130 -}
131 -
src/web/mcp/mcp-notifications.h deleted
-11
@@ -1,11 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -#ifndef NETDATA_MCP_NOTIFICATIONS_H
4 -#define NETDATA_MCP_NOTIFICATIONS_H
5 -
6 -#include "mcp.h"
7 -
8 -// Notifications namespace method dispatcher (transport-agnostic)
9 -MCP_RETURN_CODE mcp_notifications_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id);
10 -
11 -#endif // NETDATA_MCP_NOTIFICATIONS_H
\ No newline at end of file
src/web/mcp/mcp-params.c new
+604
@@ -0,0 +1,604 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-params.h"
4 +
5 +// Utility function to check if a string contains wildcards
6 +static bool contains_wildcards(const char *str) {
7 + if (!str || !*str) return false;
8 +
9 + if(*str == '!') {
10 + // simple patterns negative match (only if it starts with '!')
11 + return true;
12 + }
13 +
14 + // Check for common wildcard characters
15 + for (const char *p = str; *p; p++) {
16 + if (*p == '*') {
17 + // simple patterns wildcard match
18 + return true;
19 + }
20 +
21 + // let's check for simple pattern separators
22 + for (const char *c = SIMPLE_PATTERN_DEFAULT_WEB_SEPARATORS; *c ; c++) {
23 + if (*p == *c)
24 + return true;
25 + }
26 + }
27 + return false;
28 +}
29 +
30 +// Convert JSON array to pipe-separated string
31 +static int mcp_params_array_to_pipe_string(
32 + struct json_object *array_obj,
33 + BUFFER *output_buffer,
34 + bool allow_wildcards
35 +) {
36 + if (!array_obj || !output_buffer)
37 + return -1;
38 +
39 + if (!json_object_is_type(array_obj, json_type_array))
40 + return -1;
41 +
42 + size_t array_len = json_object_array_length(array_obj);
43 + if (array_len == 0)
44 + return 0; // Empty array is valid
45 +
46 + for (size_t i = 0; i < array_len; i++) {
47 + struct json_object *item = json_object_array_get_idx(array_obj, i);
48 + if (!json_object_is_type(item, json_type_string))
49 + return -1;
50 +
51 + const char *str = json_object_get_string(item);
52 + if (!str || !*str)
53 + return -1;
54 +
55 + // Check for wildcards if not allowed
56 + if (!allow_wildcards && contains_wildcards(str))
57 + return -2;
58 +
59 + if (i > 0)
60 + buffer_strcat(output_buffer, "|");
61 + buffer_strcat(output_buffer, str);
62 + }
63 +
64 + return 0;
65 +}
66 +
67 +// Parse array parameters (nodes, instances, dimensions)
68 +BUFFER *mcp_params_parse_array_to_pattern(
69 + struct json_object *params,
70 + const char *param_name,
71 + bool required,
72 + bool allow_wildcards,
73 + const char *list_tool,
74 + BUFFER *error
75 +) {
76 + struct json_object *array_obj = NULL;
77 +
78 + if (!json_object_object_get_ex(params, param_name, &array_obj) || !array_obj) {
79 + if (required && error) {
80 + buffer_flush(error);
81 + buffer_sprintf(error, "Missing required parameter '%s'", param_name);
82 + }
83 + return NULL; // Parameter not provided
84 + }
85 +
86 + if (!json_object_is_type(array_obj, json_type_array)) {
87 + if (error) {
88 + buffer_flush(error);
89 + buffer_sprintf(error, "%s must be an array of strings, not %s",
90 + param_name, json_type_to_name(json_object_get_type(array_obj)));
91 + }
92 + return NULL;
93 + }
94 +
95 + // Check for an empty array if required
96 + if (required && json_object_array_length(array_obj) == 0) {
97 + if (error) {
98 + buffer_flush(error);
99 + buffer_sprintf(error, "The '%s' parameter cannot be an empty array", param_name);
100 + }
101 + return NULL;
102 + }
103 +
104 + BUFFER *wb = buffer_create(256, NULL);
105 + int result = mcp_params_array_to_pipe_string(array_obj, wb, allow_wildcards);
106 +
107 + if (result == -2) {
108 + buffer_free(wb);
109 + if (error) {
110 + buffer_flush(error);
111 + if (allow_wildcards) {
112 + buffer_strcat(error, "invalid array format");
113 + } else {
114 + if (list_tool) {
115 + buffer_sprintf(error,
116 + "must contain exact values, not patterns. "
117 + "Wildcards are not supported. "
118 + "Use the '%s' tool to discover exact values.", list_tool);
119 + } else {
120 + buffer_strcat(error,
121 + "must contain exact values, not patterns. Wildcards are not supported.");
122 + }
123 + }
124 + }
125 + return NULL;
126 + } else if (result != 0) {
127 + buffer_free(wb);
128 + if (error) {
129 + buffer_flush(error);
130 + buffer_sprintf(error, "%s must be an array of strings", param_name);
131 + }
132 + return NULL;
133 + }
134 +
135 + return wb; // Success - return the buffer
136 +}
137 +
138 +// Parse labels object parameter
139 +BUFFER *mcp_params_parse_labels_object(
140 + struct json_object *params,
141 + const char *list_tool,
142 + BUFFER *error
143 +) {
144 + struct json_object *labels_obj = NULL;
145 +
146 + if (!json_object_object_get_ex(params, "labels", &labels_obj) || !labels_obj)
147 + return NULL; // Parameter not provided
148 +
149 + if (!json_object_is_type(labels_obj, json_type_object)) {
150 + if (error) {
151 + buffer_flush(error);
152 + buffer_strcat(error, "must be an object where each key maps to an array of string values");
153 + }
154 + return NULL;
155 + }
156 +
157 + BUFFER *wb = buffer_create(256, NULL);
158 + int first = 1;
159 +
160 + struct json_object_iterator it = json_object_iter_begin(labels_obj);
161 + struct json_object_iterator itEnd = json_object_iter_end(labels_obj);
162 +
163 + while (!json_object_iter_equal(&it, &itEnd)) {
164 + const char *key = json_object_iter_peek_name(&it);
165 + struct json_object *val = json_object_iter_peek_value(&it);
166 +
167 + if (!json_object_is_type(val, json_type_array)) {
168 + buffer_free(wb);
169 + if (error) {
170 + buffer_flush(error);
171 + buffer_strcat(error, "each label key must map to an array of string values");
172 + }
173 + return NULL;
174 + }
175 +
176 + size_t array_len = json_object_array_length(val);
177 + for (size_t i = 0; i < array_len; i++) {
178 + struct json_object *item = json_object_array_get_idx(val, i);
179 + if (!json_object_is_type(item, json_type_string)) {
180 + buffer_free(wb);
181 + if (error) {
182 + buffer_flush(error);
183 + buffer_strcat(error, "label values must be strings");
184 + }
185 + return NULL;
186 + }
187 +
188 + const char *value = json_object_get_string(item);
189 + if (!value || !*value) {
190 + buffer_free(wb);
191 + if (error) {
192 + buffer_flush(error);
193 + buffer_strcat(error, "label values cannot be empty");
194 + }
195 + return NULL;
196 + }
197 +
198 + // Check for wildcards in label values
199 + if (contains_wildcards(value)) {
200 + buffer_free(wb);
201 + if (error) {
202 + buffer_flush(error);
203 + if (list_tool) {
204 + buffer_sprintf(error,
205 + "label values must be exact values, not patterns. "
206 + "Wildcards are not supported. "
207 + "Use the %s tool to discover available label values.", list_tool);
208 + } else {
209 + buffer_strcat(error,
210 + "label values must be exact values, not patterns. Wildcards are not supported.");
211 + }
212 + }
213 + return NULL;
214 + }
215 +
216 + if (!first) buffer_strcat(wb, "|");
217 + buffer_sprintf(wb, "%s:%s", key, value);
218 + first = 0;
219 + }
220 +
221 + json_object_iter_next(&it);
222 + }
223 +
224 + return wb; // Success
225 +}
226 +
227 +
228 +// Schema generation functions
229 +
230 +// Add array parameter schema
231 +void mcp_schema_add_array_param(
232 + BUFFER *buffer,
233 + const char *param_name,
234 + const char *title,
235 + const char *description
236 +) {
237 + buffer_json_member_add_object(buffer, param_name);
238 + {
239 + buffer_json_member_add_string(buffer, "type", "array");
240 + buffer_json_member_add_string(buffer, "title", title);
241 + buffer_json_member_add_string(buffer, "description", description);
242 + buffer_json_member_add_object(buffer, "items");
243 + buffer_json_member_add_string(buffer, "type", "string");
244 + buffer_json_object_close(buffer); // items
245 + }
246 + buffer_json_object_close(buffer);
247 +}
248 +
249 +// Add labels object parameter schema
250 +void mcp_schema_add_labels_object(
251 + BUFFER *buffer,
252 + const char *title,
253 + const char *description
254 +) {
255 + buffer_json_member_add_object(buffer, "labels");
256 + {
257 + buffer_json_member_add_string(buffer, "type", "object");
258 + buffer_json_member_add_string(buffer, "title",
259 + title ? title : "Filter by labels");
260 + buffer_json_member_add_string(buffer, "description",
261 + description ? description :
262 + "Filter using labels where each key maps to an array of exact values. "
263 + "Values in the same array are ORed, different keys are ANDed. "
264 + "Example: {\"disk_type\": [\"ssd\", \"nvme\"], \"mount_point\": [\"/\"]}\n"
265 + "Note: Wildcards are not supported. Use exact label keys and values only.");
266 + buffer_json_member_add_object(buffer, "additionalProperties");
267 + {
268 + buffer_json_member_add_string(buffer, "type", "array");
269 + buffer_json_member_add_object(buffer, "items");
270 + buffer_json_member_add_string(buffer, "type", "string");
271 + buffer_json_object_close(buffer); // items
272 + }
273 + buffer_json_object_close(buffer); // additionalProperties
274 + }
275 + buffer_json_object_close(buffer); // labels
276 +}
277 +
278 +// Add time window parameters to the schema
279 +void mcp_schema_add_time_params(
280 + BUFFER *buffer,
281 + const char *time_description_prefix,
282 + bool required
283 +) {
284 + // Build descriptions for after and before
285 + BUFFER *after_desc = buffer_create(256, NULL);
286 + BUFFER *before_desc = buffer_create(256, NULL);
287 +
288 + if (time_description_prefix && *time_description_prefix) {
289 + buffer_sprintf(after_desc, "Start time for %s.", time_description_prefix);
290 + buffer_sprintf(before_desc, "End time for %s.", time_description_prefix);
291 + } else {
292 + buffer_strcat(after_desc, "Start time for the query.");
293 + buffer_strcat(before_desc, "End time for the query.");
294 + }
295 +
296 + // Use the new function for both parameters
297 + mcp_schema_add_time_param(buffer, "after", "Start time", buffer_tostring(after_desc),
298 + "'before'", required ? 0 : MCP_DEFAULT_AFTER_TIME, required);
299 +
300 + mcp_schema_add_time_param(buffer, "before", "End time", buffer_tostring(before_desc),
301 + "now", required ? 0 : MCP_DEFAULT_BEFORE_TIME, required);
302 +
303 + buffer_free(after_desc);
304 + buffer_free(before_desc);
305 +}
306 +
307 +// Add cardinality limit parameter to schema
308 +void mcp_schema_add_cardinality_limit(
309 + BUFFER *buffer,
310 + const char *description,
311 + size_t default_value,
312 + size_t min_value,
313 + size_t max_value
314 +) {
315 + buffer_json_member_add_object(buffer, "cardinality_limit");
316 + {
317 + buffer_json_member_add_string(buffer, "type", "number");
318 + buffer_json_member_add_string(buffer, "title", "Cardinality Limit");
319 + buffer_json_member_add_string(
320 + buffer, "description",
321 + description ? description : "When multiple nodes, instances, dimensions, labels are queried, "
322 + "limit their numbers to prevent response size explosion.");
323 +
324 + if (default_value > 0)
325 + buffer_json_member_add_uint64(buffer, "default", default_value);
326 +
327 + buffer_json_member_add_uint64(buffer, "minimum", min_value > 0 ? min_value : 1);
328 +
329 + if (max_value > 0)
330 + buffer_json_member_add_uint64(buffer, "maximum", max_value);
331 + }
332 + buffer_json_object_close(buffer);
333 +}
334 +
335 +// Extract string parameter with optional default
336 +const char *mcp_params_extract_string(
337 + struct json_object *params,
338 + const char *param_name,
339 + const char *default_value
340 +) {
341 + struct json_object *obj = NULL;
342 +
343 + if (json_object_object_get_ex(params, param_name, &obj) && obj) {
344 + if (json_object_is_type(obj, json_type_string)) {
345 + const char *value = json_object_get_string(obj);
346 + if (value && *value) {
347 + return value;
348 + }
349 + }
350 + }
351 +
352 + return default_value;
353 +}
354 +
355 +// Extract numeric size parameter with bounds checking
356 +size_t mcp_params_extract_size(
357 + struct json_object *params,
358 + const char *param_name,
359 + size_t default_value,
360 + size_t min_value,
361 + size_t max_value,
362 + BUFFER *error
363 +) {
364 + struct json_object *obj = NULL;
365 +
366 + if (json_object_object_get_ex(params, param_name, &obj) && obj) {
367 + if (json_object_is_type(obj, json_type_int)) {
368 + int64_t value = json_object_get_int64(obj);
369 + if (value < 0) {
370 + if (error) {
371 + buffer_flush(error);
372 + buffer_sprintf(error, "%s must be a positive number", param_name);
373 + }
374 + return default_value;
375 + }
376 + size_t size_value = (size_t)value;
377 + if (size_value < min_value || (max_value > 0 && size_value > max_value)) {
378 + if (error) {
379 + buffer_flush(error);
380 + if (max_value > 0) {
381 + buffer_sprintf(error,
382 + "%s must be between %zu and %zu",
383 + param_name, min_value, max_value);
384 + } else {
385 + buffer_sprintf(error,
386 + "%s must be at least %zu",
387 + param_name, min_value);
388 + }
389 + }
390 + return default_value;
391 + }
392 + return size_value;
393 + }
394 + }
395 +
396 + return default_value;
397 +}
398 +
399 +// Extract timeout parameter (in seconds)
400 +int mcp_params_extract_timeout(
401 + struct json_object *params,
402 + const char *param_name,
403 + int default_seconds,
404 + int min_seconds,
405 + int max_seconds,
406 + BUFFER *error
407 +) {
408 + struct json_object *obj = NULL;
409 +
410 + if (json_object_object_get_ex(params, param_name, &obj) && obj) {
411 + if (json_object_is_type(obj, json_type_int)) {
412 + int value = json_object_get_int(obj);
413 + if (value < min_seconds || (max_seconds > 0 && value > max_seconds)) {
414 + if (error) {
415 + buffer_flush(error);
416 + if (max_seconds > 0) {
417 + buffer_sprintf(error,
418 + "%s must be between %d and %d seconds",
419 + param_name, min_seconds, max_seconds);
420 + } else {
421 + buffer_sprintf(error,
422 + "%s must be at least %d seconds",
423 + param_name, min_seconds);
424 + }
425 + }
426 + return default_seconds;
427 + }
428 + return value;
429 + }
430 + }
431 +
432 + return default_seconds;
433 +}
434 +
435 +// Schema generation for timeout parameter
436 +void mcp_schema_add_timeout(
437 + BUFFER *buffer,
438 + const char *param_name,
439 + const char *title,
440 + const char *description,
441 + int default_seconds,
442 + int min_seconds,
443 + int max_seconds,
444 + bool required
445 +) {
446 + buffer_json_member_add_object(buffer, param_name);
447 + {
448 + buffer_json_member_add_string(buffer, "type", "number");
449 + buffer_json_member_add_string(buffer, "title", title);
450 + buffer_json_member_add_string(buffer, "description", description);
451 +
452 + if (!required && default_seconds >= 0)
453 + buffer_json_member_add_int64(buffer, "default", default_seconds);
454 +
455 + if (min_seconds >= 0)
456 + buffer_json_member_add_int64(buffer, "minimum", min_seconds);
457 +
458 + if (max_seconds > 0)
459 + buffer_json_member_add_int64(buffer, "maximum", max_seconds);
460 + }
461 + buffer_json_object_close(buffer);
462 +}
463 +
464 +// Schema generation for generic string parameter
465 +void mcp_schema_add_string_param(
466 + BUFFER *buffer,
467 + const char *param_name,
468 + const char *title,
469 + const char *description,
470 + const char *default_value,
471 + bool required
472 +) {
473 + buffer_json_member_add_object(buffer, param_name);
474 + {
475 + buffer_json_member_add_string(buffer, "type", "string");
476 + buffer_json_member_add_string(buffer, "title", title);
477 + buffer_json_member_add_string(buffer, "description", description);
478 +
479 + if (!required && default_value)
480 + buffer_json_member_add_string(buffer, "default", default_value);
481 + }
482 + buffer_json_object_close(buffer);
483 +}
484 +
485 +// Schema generation for size parameter
486 +void mcp_schema_add_size_param(
487 + BUFFER *buffer,
488 + const char *param_name,
489 + const char *title,
490 + const char *description,
491 + size_t default_value,
492 + size_t min_value,
493 + size_t max_value,
494 + bool required
495 +) {
496 + buffer_json_member_add_object(buffer, param_name);
497 + {
498 + buffer_json_member_add_string(buffer, "type", "number");
499 + buffer_json_member_add_string(buffer, "title", title);
500 + buffer_json_member_add_string(buffer, "description", description);
501 +
502 + if (!required)
503 + buffer_json_member_add_uint64(buffer, "default", default_value);
504 +
505 + if (min_value > 0)
506 + buffer_json_member_add_uint64(buffer, "minimum", min_value);
507 +
508 + if (max_value < SIZE_MAX)
509 + buffer_json_member_add_uint64(buffer, "maximum", max_value);
510 + }
511 + buffer_json_object_close(buffer);
512 +}
513 +
514 +time_t mcp_params_parse_time(struct json_object *params, const char *name, time_t default_value) {
515 + if (!params)
516 + return default_value;
517 +
518 + struct json_object *obj = NULL;
519 + if (!json_object_object_get_ex(params, name, &obj) || !obj)
520 + return default_value;
521 +
522 + // First try as integer
523 + if (json_object_is_type(obj, json_type_int))
524 + return json_object_get_int64(obj);
525 +
526 + // Then try as string
527 + if (json_object_is_type(obj, json_type_string)) {
528 + const char *val_str = json_object_get_string(obj);
529 + if (!val_str || !*val_str)
530 + return default_value;
531 +
532 + // Try to parse as RFC3339 first
533 + char *endptr = NULL;
534 + usec_t timestamp_ut = rfc3339_parse_ut(val_str, &endptr);
535 +
536 + // Check if RFC3339 parsing was successful
537 + // Note: We check endptr to see if the entire string was consumed
538 + // We don't check timestamp_ut > 0 because dates before 1970 are valid
539 + if (endptr && (*endptr == '\0' || isspace((unsigned char)*endptr))) {
540 + // Successfully parsed as RFC3339, convert to seconds
541 + return (time_t)(timestamp_ut / USEC_PER_SEC);
542 + }
543 +
544 + // RFC3339 parsing failed, fall back to parsing as integer
545 + // This handles:
546 + // - Unix timestamps as strings: "1705318200"
547 + // - Relative times as strings: "-3600", "-86400"
548 + // - Special values: "0", "now" (handled by str2l)
549 + return str2l(val_str);
550 + }
551 +
552 + return default_value;
553 +}
554 +
555 +// Schema generation for individual time parameter
556 +void mcp_schema_add_time_param(
557 + BUFFER *buffer,
558 + const char *param_name,
559 + const char *title,
560 + const char *description,
561 + const char *relative_to,
562 + time_t default_value,
563 + bool required
564 +) {
565 + buffer_json_member_add_object(buffer, param_name);
566 + {
567 + buffer_json_member_add_string(buffer, "title", title);
568 +
569 + if (description && *description)
570 + buffer_json_member_add_string(buffer, "description", description);
571 + else
572 + buffer_json_member_add_string(
573 + buffer, "description",
574 + "Unix epoch timestamp in seconds (e.g. 1705318200), "
575 + "or number of seconds, "
576 + "or RFC3339 datetime string");
577 +
578 + // Use anyOf for multiple types
579 + buffer_json_member_add_array(buffer, "anyOf");
580 + {
581 + buffer_json_add_array_item_object(buffer);
582 + {
583 + buffer_json_member_add_string(buffer, "type", "number");
584 + buffer_json_member_add_sprintf(buffer, "description",
585 + "Unix epoch timestamp in seconds (e.g. 1705318200), or number of seconds relative to %s (e.g. -3600 for an hour before %s)",
586 + relative_to ? relative_to : "now", relative_to ? relative_to : "now");
587 + }
588 + buffer_json_object_close(buffer);
589 +
590 + buffer_json_add_array_item_object(buffer);
591 + {
592 + buffer_json_member_add_string(buffer, "type", "string");
593 + buffer_json_member_add_string(buffer, "description", "RFC3339 datetime string (e.g., \"2024-01-15T10:30:00Z\", \"2024-01-15T10:30:00-05:00\")");
594 + }
595 + buffer_json_object_close(buffer);
596 + }
597 + buffer_json_array_close(buffer);
598 +
599 + if (!required && default_value != 0) {
600 + buffer_json_member_add_int64(buffer, "default", default_value);
601 + }
602 + }
603 + buffer_json_object_close(buffer);
604 +}
src/web/mcp/mcp-params.h new
+148
@@ -0,0 +1,148 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_PARAMS_H
4 +#define NETDATA_MCP_PARAMS_H
5 +
6 +#include "mcp.h"
7 +#include <json-c/json.h>
8 +
9 +// Common parameter parsing functions
10 +
11 +// Parse array parameters (nodes, instances, dimensions) and convert to pipe-separated string
12 +// Returns newly allocated BUFFER on success, or NULL if not provided/on error
13 +// Writes error message to error buffer on failure
14 +// required: if true, validates parameter exists, is array type, and is non-empty
15 +// list_tool: tool name to recommend for discovering exact values (e.g., MCP_TOOL_LIST_NODES)
16 +BUFFER *mcp_params_parse_array_to_pattern(
17 + struct json_object *params,
18 + const char *param_name,
19 + bool required,
20 + bool allow_wildcards,
21 + const char *list_tool,
22 + BUFFER *error
23 +);
24 +
25 +// Parse labels object parameter and convert to query string format
26 +// Returns newly allocated BUFFER on success, or NULL if not provided/on error
27 +// Writes error message to error buffer on failure
28 +// list_tool: tool name to recommend for discovering exact values (e.g., MCP_TOOL_GET_METRICS_DETAILS)
29 +BUFFER *mcp_params_parse_labels_object(
30 + struct json_object *params,
31 + const char *list_tool,
32 + BUFFER *error
33 +);
34 +
35 +// Add array parameter schema (for nodes, instances, dimensions)
36 +void mcp_schema_add_array_param(
37 + BUFFER *buffer,
38 + const char *param_name,
39 + const char *title,
40 + const char *description
41 +);
42 +
43 +// Add labels object parameter schema
44 +void mcp_schema_add_labels_object(
45 + BUFFER *buffer,
46 + const char *title,
47 + const char *description
48 +);
49 +
50 +// Add time window parameters (after, before) to schema
51 +void mcp_schema_add_time_params(
52 + BUFFER *buffer,
53 + const char *time_description_prefix,
54 + bool required
55 +);
56 +
57 +// Add cardinality limit parameter to schema
58 +void mcp_schema_add_cardinality_limit(
59 + BUFFER *buffer,
60 + const char *description,
61 + size_t default_value,
62 + size_t min_value,
63 + size_t max_value
64 +);
65 +
66 +// Extract string parameter with optional default
67 +// Returns the string value or default_value if not found
68 +const char *mcp_params_extract_string(
69 + struct json_object *params,
70 + const char *param_name,
71 + const char *default_value
72 +);
73 +
74 +// Extract numeric size parameter with bounds checking
75 +// Returns the size value or default_value if not found
76 +// Writes error message to error buffer if value is out of bounds
77 +size_t mcp_params_extract_size(
78 + struct json_object *params,
79 + const char *param_name,
80 + size_t default_value,
81 + size_t min_value,
82 + size_t max_value,
83 + BUFFER *error
84 +);
85 +
86 +// Extract timeout parameter (in seconds)
87 +// Returns the timeout value or default_value if not found
88 +// Writes error message to error buffer if value is out of bounds
89 +int mcp_params_extract_timeout(
90 + struct json_object *params,
91 + const char *param_name,
92 + int default_seconds,
93 + int min_seconds,
94 + int max_seconds,
95 + BUFFER *error
96 +);
97 +
98 +// Schema generation for timeout parameter
99 +void mcp_schema_add_timeout(
100 + BUFFER *buffer,
101 + const char *param_name,
102 + const char *title,
103 + const char *description,
104 + int default_seconds,
105 + int min_seconds,
106 + int max_seconds,
107 + bool required
108 +);
109 +
110 +// Schema generation for generic string parameter
111 +void mcp_schema_add_string_param(
112 + BUFFER *buffer,
113 + const char *param_name,
114 + const char *title,
115 + const char *description,
116 + const char *default_value,
117 + bool required
118 +);
119 +
120 +// Schema generation for size parameter
121 +void mcp_schema_add_size_param(
122 + BUFFER *buffer,
123 + const char *param_name,
124 + const char *title,
125 + const char *description,
126 + size_t default_value,
127 + size_t min_value,
128 + size_t max_value,
129 + bool required
130 +);
131 +
132 +time_t mcp_params_parse_time(
133 + struct json_object *params,
134 + const char *name,
135 + time_t default_value);
136 +
137 +// Schema generation for individual time parameter
138 +void mcp_schema_add_time_param(
139 + BUFFER *buffer,
140 + const char *param_name,
141 + const char *title,
142 + const char *description,
143 + const char *relative_to,
144 + time_t default_value,
145 + bool required
146 +);
147 +
148 +#endif // NETDATA_MCP_PARAMS_H
\ No newline at end of file
src/web/mcp/mcp-ping.c new
+49
@@ -0,0 +1,49 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +/**
4 + * MCP Ping Method
5 + *
6 + * The ping method is a core part of the Model Context Protocol (MCP),
7 + * allowing connection health checks between client and server.
8 + *
9 + * Standard method in the MCP specification:
10 + *
11 + * 1. ping - Simple connection health check
12 + * - Takes no parameters (empty params object)
13 + * - The receiver must respond promptly with an empty result object
14 + * - Either client or server can initiate a ping
15 + * - If no response is received within a reasonable timeout, the connection may be considered stale
16 + *
17 + * According to the MCP specification:
18 + * - The ping method is mandatory for all MCP implementations
19 + * - It serves as a basic mechanism to verify the connection is still active
20 + * - Implementations should handle ping requests promptly to ensure accurate health checks
21 + *
22 + * This implementation provides a simple handler for ping requests that responds with an
23 + * empty result object, as required by the specification.
24 + */
25 +
26 +#include "mcp-ping.h"
27 +
28 +/**
29 + * Handle a ping request from a client or server
30 + *
31 + * @param mcpc The MCP client context
32 + * @param params The JSON params object (should be empty for ping)
33 + * @param id The request ID
34 + * @return MCP_RETURN_CODE - MCP_RC_OK if successful
35 + */
36 +MCP_RETURN_CODE mcp_method_ping(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, MCP_REQUEST_ID id) {
37 + if (!mcpc) {
38 + return MCP_RC_ERROR;
39 + }
40 +
41 + // Initialize success response with empty result object
42 + mcp_init_success_result(mcpc, id);
43 + buffer_json_finalize(mcpc->result);
44 +
45 + // Log the ping for debugging
46 + netdata_log_debug(D_MCP, "Received ping request (ID: %zu), responded", id);
47 +
48 + return MCP_RC_OK;
49 +}
\ No newline at end of file
src/web/mcp/mcp-ping.h new
+20
@@ -0,0 +1,20 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_PING_H
4 +#define NETDATA_MCP_PING_H
5 +
6 +#include "libnetdata/libnetdata.h"
7 +#include <json-c/json.h>
8 +#include "mcp.h"
9 +
10 +/**
11 + * Handle ping requests from a client
12 + *
13 + * @param mcpc The MCP client context
14 + * @param params The JSON params object (should be empty for ping)
15 + * @param id The request ID
16 + * @return MCP_RETURN_CODE - MCP_RC_OK if successful
17 + */
18 +MCP_RETURN_CODE mcp_method_ping(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
19 +
20 +#endif // NETDATA_MCP_PING_H
\ No newline at end of file
src/web/mcp/mcp-prompts.c
+16 -52
@@ -3,22 +3,21 @@
3 /**
4 * MCP Prompts Namespace
5 *
6 - * The MCP Prompts namespace provides methods for managing and executing prompts.
6 + * The MCP Prompts namespace provides methods for managing prompts.
7 * In the MCP protocol, prompts are text templates that guide AI generation for specific tasks.
8 * Prompts are user-controlled interactions that leverage AI capabilities in predefined ways.
9 *
10 - * Key features of the prompts namespace:
10 + * Standard methods in the MCP specification:
11 *
12 - * 1. Prompt Management:
13 - * - List available prompts (prompts/list)
14 - * - Get details about specific prompts (prompts/get)
15 - * - Save custom prompts (prompts/save)
16 - * - Delete prompts (prompts/delete)
17 - * - Organize prompts into categories (prompts/getCategories)
12 + * 1. prompts/list - Lists available prompts
13 + * - Returns a collection of prompts the server offers
14 + * - Includes prompt names, descriptions, and potentially argument information
15 + * - Can be paginated for large prompt collections
16 *
19 - * 2. Prompt Execution:
20 - * - Execute prompts with input parameters (prompts/execute)
21 - * - View execution history (prompts/getHistory)
17 + * 2. prompts/get - Gets details about a specific prompt
18 + * - Takes a prompt name and returns its full definition
19 + * - Returns the prompt template, messages, and metadata
20 + * - May include information about required arguments
21 *
22 * Prompts differ from tools in that they are:
23 * - More flexible and text-oriented
@@ -38,16 +37,16 @@
37 */
38
39 #include "mcp-prompts.h"
41 -#include "mcp-initialize.h"
40
41 // Implementation of prompts/list (transport-agnostic)
44 -static MCP_RETURN_CODE mcp_prompts_method_list(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
45 - if (!mcpc || id == 0) return MCP_RC_ERROR;
42 +static MCP_RETURN_CODE mcp_prompts_method_list(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, MCP_REQUEST_ID id) {
43 + if (!mcpc || id == 0)
44 + return MCP_RC_ERROR;
45
46 // Initialize success response
47 mcp_init_success_result(mcpc, id);
48
50 - // Add empty prompts array
49 + // Add an empty prompts array
50 buffer_json_member_add_array(mcpc->result, "prompts");
51 buffer_json_array_close(mcpc->result); // Close prompts array
52
@@ -58,38 +57,18 @@ static MCP_RETURN_CODE mcp_prompts_method_list(MCP_CLIENT *mcpc, struct json_obj
57 }
58
59 // Stub implementations for other prompts methods (transport-agnostic)
61 -static MCP_RETURN_CODE mcp_prompts_method_execute(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
62 - buffer_sprintf(mcpc->error, "Method 'prompts/execute' not implemented yet");
63 - return MCP_RC_NOT_IMPLEMENTED;
64 -}
60
66 -static MCP_RETURN_CODE mcp_prompts_method_get(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
61 +static MCP_RETURN_CODE mcp_prompts_method_get(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, MCP_REQUEST_ID id __maybe_unused) {
62 buffer_sprintf(mcpc->error, "Method 'prompts/get' not implemented yet");
63 return MCP_RC_NOT_IMPLEMENTED;
64 }
65
71 -static MCP_RETURN_CODE mcp_prompts_method_save(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
72 - buffer_sprintf(mcpc->error, "Method 'prompts/save' not implemented yet");
73 - return MCP_RC_NOT_IMPLEMENTED;
74 -}
66
76 -static MCP_RETURN_CODE mcp_prompts_method_delete(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
77 - buffer_sprintf(mcpc->error, "Method 'prompts/delete' not implemented yet");
78 - return MCP_RC_NOT_IMPLEMENTED;
79 -}
67
81 -static MCP_RETURN_CODE mcp_prompts_method_getCategories(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
82 - buffer_sprintf(mcpc->error, "Method 'prompts/getCategories' not implemented yet");
83 - return MCP_RC_NOT_IMPLEMENTED;
84 -}
68
86 -static MCP_RETURN_CODE mcp_prompts_method_getHistory(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
87 - buffer_sprintf(mcpc->error, "Method 'prompts/getHistory' not implemented yet");
88 - return MCP_RC_NOT_IMPLEMENTED;
89 -}
69
70 // Prompts namespace method dispatcher (transport-agnostic)
92 -MCP_RETURN_CODE mcp_prompts_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id) {
71 +MCP_RETURN_CODE mcp_prompts_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id) {
72 if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
73
74 netdata_log_debug(D_MCP, "MCP prompts method: %s", method);
@@ -103,24 +82,9 @@ MCP_RETURN_CODE mcp_prompts_route(MCP_CLIENT *mcpc, const char *method, struct j
82 if (strcmp(method, "list") == 0) {
83 rc = mcp_prompts_method_list(mcpc, params, id);
84 }
106 - else if (strcmp(method, "execute") == 0) {
107 - rc = mcp_prompts_method_execute(mcpc, params, id);
108 - }
85 else if (strcmp(method, "get") == 0) {
86 rc = mcp_prompts_method_get(mcpc, params, id);
87 }
112 - else if (strcmp(method, "save") == 0) {
113 - rc = mcp_prompts_method_save(mcpc, params, id);
114 - }
115 - else if (strcmp(method, "delete") == 0) {
116 - rc = mcp_prompts_method_delete(mcpc, params, id);
117 - }
118 - else if (strcmp(method, "getCategories") == 0) {
119 - rc = mcp_prompts_method_getCategories(mcpc, params, id);
120 - }
121 - else if (strcmp(method, "getHistory") == 0) {
122 - rc = mcp_prompts_method_getHistory(mcpc, params, id);
123 - }
88 else {
89 // Method not found in prompts namespace
90 buffer_sprintf(mcpc->error, "Method 'prompts/%s' not implemented yet", method);
src/web/mcp/mcp-prompts.h
+1 -1
@@ -6,6 +6,6 @@
6 #include "mcp.h"
7
8 // Prompts namespace method dispatcher (transport-agnostic)
9 -MCP_RETURN_CODE mcp_prompts_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id);
9 +MCP_RETURN_CODE mcp_prompts_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id);
10
11 #endif // NETDATA_MCP_PROMPTS_H
\ No newline at end of file
src/web/mcp/mcp-request-id.c new
+174
@@ -0,0 +1,174 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-request-id.h"
4 +#include "mcp.h"
5 +
6 +// Request ID structure - stored in JudyL array
7 +typedef struct mcp_request_id_entry {
8 + enum {
9 + MCP_REQUEST_ID_TYPE_INT,
10 + MCP_REQUEST_ID_TYPE_STRING
11 + } type;
12 +
13 + union {
14 + int64_t int_value;
15 + STRING *str_value;
16 + };
17 +} MCP_REQUEST_ID_ENTRY;
18 +
19 +/**
20 + * Extract and register a request ID from a JSON object
21 + *
22 + * @param mcpc The MCP client context
23 + * @param request The JSON request object that may contain an ID
24 + * @return MCP_REQUEST_ID - the assigned ID (0 if no ID was present)
25 + */
26 +MCP_REQUEST_ID mcp_request_id_add(MCP_CLIENT *mcpc, struct json_object *request) {
27 + if (!mcpc || !request)
28 + return 0;
29 +
30 + // Extract ID (optional, for notifications)
31 + struct json_object *id_obj = NULL;
32 + bool has_id = json_object_object_get_ex(request, "id", &id_obj);
33 +
34 + if (!has_id)
35 + return 0;
36 +
37 + // Allocate a new entry
38 + MCP_REQUEST_ID_ENTRY *entry = callocz(1, sizeof(MCP_REQUEST_ID_ENTRY));
39 +
40 + // Generate a new sequential ID
41 + MCP_REQUEST_ID id = ++mcpc->request_id_counter;
42 +
43 + // Store the entry in the JudyL array
44 + Word_t Index = (Word_t)id;
45 + Pvoid_t *PValue = JudyLIns(&mcpc->request_ids, Index, NULL);
46 + if (unlikely(PValue == PJERR)) {
47 + netdata_log_error("MCP: JudyLIns failed for request ID %zu", id);
48 + freez(entry);
49 + return 0;
50 + }
51 +
52 + // Parse the ID value
53 + if (json_object_get_type(id_obj) == json_type_int) {
54 + entry->type = MCP_REQUEST_ID_TYPE_INT;
55 + entry->int_value = json_object_get_int64(id_obj);
56 + }
57 + else if (json_object_get_type(id_obj) == json_type_string) {
58 + entry->type = MCP_REQUEST_ID_TYPE_STRING;
59 + entry->str_value = string_strdupz(json_object_get_string(id_obj));
60 + }
61 + else {
62 + // Unsupported ID type, treat as no ID
63 + freez(entry);
64 + return 0;
65 + }
66 +
67 + // Store the entry in the JudyL
68 + *PValue = entry;
69 +
70 + return id;
71 +}
72 +
73 +/**
74 + * Delete a request ID from the registry
75 + *
76 + * @param mcpc The MCP client context
77 + * @param id The request ID to delete
78 + */
79 +void mcp_request_id_del(MCP_CLIENT *mcpc, MCP_REQUEST_ID id) {
80 + if (!mcpc || id == 0)
81 + return;
82 +
83 + // Get the entry from JudyL
84 + Word_t Index = (Word_t)id;
85 + Pvoid_t *PValue = JudyLGet(mcpc->request_ids, Index, NULL);
86 + if (!PValue)
87 + return;
88 +
89 + MCP_REQUEST_ID_ENTRY *entry = *PValue;
90 +
91 + // Free string value if present
92 + if (entry->type == MCP_REQUEST_ID_TYPE_STRING)
93 + string_freez(entry->str_value);
94 +
95 + // Free the entry
96 + freez(entry);
97 +
98 + // Remove the entry from JudyL
99 + int rc = JudyLDel(&mcpc->request_ids, Index, NULL);
100 + if (unlikely(!rc)) {
101 + netdata_log_error("MCP: JudyLDel failed for request ID %zu", id);
102 + }
103 +}
104 +
105 +/**
106 + * Clean up all request IDs for a client
107 + *
108 + * @param mcpc The MCP client context
109 + */
110 +void mcp_request_id_cleanup_all(MCP_CLIENT *mcpc) {
111 + if (!mcpc || !mcpc->request_ids)
112 + return;
113 +
114 + Word_t Index = 0;
115 + Pvoid_t *PValue;
116 +
117 + // Get the first index
118 + PValue = JudyLFirst(mcpc->request_ids, &Index, NULL);
119 +
120 + // Iterate through all entries
121 + while (PValue != NULL) {
122 + // Free the request ID entry
123 + MCP_REQUEST_ID_ENTRY *entry = *PValue;
124 + if (entry->type == MCP_REQUEST_ID_TYPE_STRING)
125 + string_freez(entry->str_value);
126 + freez(entry);
127 +
128 + // Move to next entry
129 + PValue = JudyLNext(mcpc->request_ids, &Index, NULL);
130 + }
131 +
132 + // Free the JudyL array
133 + JudyLFreeArray(&mcpc->request_ids, NULL);
134 + mcpc->request_ids = NULL;
135 +}
136 +
137 +/**
138 + * Add a request ID to a buffer as a JSON member
139 + *
140 + * @param mcpc The MCP client context
141 + * @param wb The buffer to add the ID to
142 + * @param key The JSON key name to use
143 + * @param id The request ID to add
144 + */
145 +void mcp_request_id_to_buffer(MCP_CLIENT *mcpc, BUFFER *wb, const char *key, MCP_REQUEST_ID id) {
146 + if (!wb || !key) {
147 + return;
148 + }
149 +
150 + if (!mcpc || id == 0) {
151 + // For ID 0 or no client context, add it as a numeric 0
152 + buffer_json_member_add_uint64(wb, key, 0);
153 + return;
154 + }
155 +
156 + // Get the entry from JudyL
157 + Word_t Index = (Word_t)id;
158 + Pvoid_t *PValue = JudyLGet(mcpc->request_ids, Index, NULL);
159 + if (!PValue) {
160 + // If entry not found, add 0 as the ID
161 + buffer_json_member_add_uint64(wb, key, 0);
162 + return;
163 + }
164 +
165 + MCP_REQUEST_ID_ENTRY *entry = *PValue;
166 +
167 + // Add the ID based on its type
168 + if (entry->type == MCP_REQUEST_ID_TYPE_INT) {
169 + buffer_json_member_add_uint64(wb, key, entry->int_value);
170 + }
171 + else if (entry->type == MCP_REQUEST_ID_TYPE_STRING) {
172 + buffer_json_member_add_string(wb, key, string2str(entry->str_value));
173 + }
174 +}
\ No newline at end of file
src/web/mcp/mcp-request-id.h new
+48
@@ -0,0 +1,48 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_REQUEST_ID_H
4 +#define NETDATA_MCP_REQUEST_ID_H
5 +
6 +#include "libnetdata/libnetdata.h"
7 +
8 +// Request ID type - 0 is reserved for "no ID given"
9 +typedef size_t MCP_REQUEST_ID;
10 +
11 +// Forward declaration
12 +struct mcp_client;
13 +
14 +/**
15 + * Extract and register a request ID from a JSON object
16 + *
17 + * @param mcpc The MCP client context
18 + * @param request The JSON request object that may contain an ID
19 + * @return MCP_REQUEST_ID - the assigned ID (0 if no ID was present)
20 + */
21 +MCP_REQUEST_ID mcp_request_id_add(struct mcp_client *mcpc, struct json_object *request);
22 +
23 +/**
24 + * Delete a request ID from the registry
25 + *
26 + * @param mcpc The MCP client context
27 + * @param id The request ID to delete
28 + */
29 +void mcp_request_id_del(struct mcp_client *mcpc, MCP_REQUEST_ID id);
30 +
31 +/**
32 + * Clean up all request IDs for a client
33 + *
34 + * @param mcpc The MCP client context
35 + */
36 +void mcp_request_id_cleanup_all(struct mcp_client *mcpc);
37 +
38 +/**
39 + * Add a request ID to a buffer as a JSON member
40 + *
41 + * @param mcpc The MCP client context
42 + * @param wb The buffer to add the ID to
43 + * @param key The JSON key name to use
44 + * @param id The request ID to add
45 + */
46 +void mcp_request_id_to_buffer(struct mcp_client *mcpc, BUFFER *wb, const char *key, MCP_REQUEST_ID id);
47 +
48 +#endif // NETDATA_MCP_REQUEST_ID_H
src/web/mcp/mcp-resources.c
+46 -357
@@ -7,36 +7,44 @@
7 * In the MCP protocol, resources are application-controlled data stores that provide context to the model.
8 * Resources are passive, meaning they provide data but don't perform actions on their own.
9 *
10 - * Key features of the resources namespace:
10 + * Standard methods in the MCP specification:
11 *
12 - * 1. Resource Discovery:
13 - * - Clients can list available resources (resources/list)
14 - * - Get detailed descriptions and schemas (resources/describe, resources/getSchema)
15 - * - Search for resources matching specific criteria (resources/search)
16 - *
17 - * 2. Resource Access:
18 - * - Retrieve specific resources or portions of resources (resources/get)
19 - * - Access resources by ID or path
20 - * - Resources can be structured or unstructured
21 - *
22 - * 3. Resource Subscriptions:
23 - * - Subscribe to updates for specific resources (resources/subscribe)
24 - * - Unsubscribe from resources (resources/unsubscribe)
25 - * - Get real-time updates when subscribed resources change
12 + * 1. resources/list - Lists available resources
13 + * - Returns a collection of resources the server can provide
14 + * - May include resource metadata such as name, description, and URI
15 + * - Can be paginated for large resource collections
16 + *
17 + * 2. resources/read - Reads a specific resource by URI
18 + * - Takes a resource URI and returns its contents
19 + * - Contents can be text, binary data, or structured information
20 + * - URIs follow a standard format, typically with a scheme prefix
21 + *
22 + * 3. resources/templates/list - Lists available resource templates
23 + * - Returns a collection of URI templates for constructing resource URIs
24 + * - Templates describe how to construct valid resource URIs
25 + * - May include template descriptions and parameter information
26 + *
27 + * 4. resources/subscribe - Subscribes to changes in a resource
28 + * - Takes a resource URI and registers for change notifications
29 + * - When the resource changes, the server sends notifications
30 + * - Allows clients to maintain up-to-date views of resources
31 + *
32 + * 5. resources/unsubscribe - Unsubscribes from a resource
33 + * - Takes a resource URI and removes the subscription
34 + * - Stops receiving notifications for that resource
35 *
36 * In the Netdata context, resources include:
37 * - metrics: Time-series data collected from various sources
38 * - logs: Log entries from system and application logs
39 * - alerts: Health monitoring alerts and notifications
31 - * - functions: Live infrastructure snapshots providing real-time views
40 + * - contexts: Hierarchical organization of metrics and their metadata
41 * - nodes: Monitored infrastructure nodes with their metadata
42 *
34 - * Resources can be hierarchical or flat, and may support different access patterns
35 - * (e.g., time-based querying for metrics, full-text search for logs).
43 + * Resources are identified by URIs (e.g., "nd://contexts") and can be hierarchical or flat,
44 + * supporting different access patterns like time-based querying for metrics.
45 */
46
47 #include "mcp-resources.h"
39 -#include "mcp-initialize.h"
48 #include "database/contexts/rrdcontext.h"
49
50 // Audience enum - bitmask for the intended audience of a resource
@@ -46,8 +54,8 @@ typedef enum {
54 RESOURCE_AUDIENCE_BOTH = RESOURCE_AUDIENCE_USER | RESOURCE_AUDIENCE_ASSISTANT
55 } RESOURCE_AUDIENCE;
56
49 -// Function pointer type for resource read callbacks
50 -typedef MCP_RETURN_CODE (*resource_read_fn)(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id);
57 +// Function pointer type for resource-read callbacks
58 +typedef MCP_RETURN_CODE (*resource_read_fn)(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
59
60 // Function pointer type for resource size callbacks
61 typedef size_t (*resource_size_fn)(void);
@@ -74,263 +82,23 @@ typedef struct {
82 double priority; // Priority (0.0-1.0)
83 } MCP_RESOURCE_TEMPLATE;
84
77 -// Basic implementation of the contexts resource read function
78 -static MCP_RETURN_CODE mcp_resource_read_contexts(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
79 - if (!mcpc || !params || id == 0) return MCP_RC_INTERNAL_ERROR;
80 -
81 - // Extract URI from params to check for query parameters
82 - struct json_object *uri_obj = NULL;
83 - json_object_object_get_ex(params, "uri", &uri_obj);
84 - const char *uri = json_object_get_string(uri_obj);
85 -
86 - SIMPLE_PATTERN *pattern = NULL;
87 -
88 - // Check if we have a query parameter
89 - if (uri && strstr(uri, "?like=")) {
90 - const char *like_param = strstr(uri, "?like=") + 6; // Skip past "?like="
91 -
92 - // Decode the query parameter
93 - const char *decoded_query = mcp_uri_decode(mcpc, like_param);
94 -
95 - // Create a simple pattern
96 - if (decoded_query && *decoded_query)
97 - pattern = simple_pattern_create(decoded_query, "|", SIMPLE_PATTERN_EXACT, false);
98 - }
99 -
100 - mcp_init_success_result(mcpc, id);
101 - buffer_json_member_add_object(mcpc->result, "result");
102 -
103 - // Add the filtered contexts
104 - rrdcontext_context_registry_json_mcp_array(mcpc->result, pattern);
105 -
106 - // Add instructions
107 - buffer_json_member_add_string(mcpc->result, "instructions",
108 - "Additional information per context (like title, dimensions, unit, label\n"
109 - "keys and possible values, the list of nodes collecting it, and its retention)\n"
110 - "can be obtained by reading URIs in the format 'nd://contexts/{context}'\n"
111 - "(like nd://context/system.cpu.user).\n\n"
112 - "You can search contexts using glob-like patterns using the 'like' parameter:\n"
113 - "nd://contexts?like=*sql*|*db*|*redis*|*mongo*\n"
114 - "to find postgresql, mysql, mariadb and mongodb related contexts.\n\n"
115 - "For a high-level overview of monitoring categories, use nd://context-categories");
116 -
117 - buffer_json_finalize(mcpc->result);
118 -
119 - if (pattern)
120 - simple_pattern_free(pattern);
121 -
122 - return MCP_RC_OK;
123 -}
124 -
125 -// Implementation of the context categories resource read function
126 -static MCP_RETURN_CODE mcp_resource_read_context_categories(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
127 - if (!mcpc || !params || id == 0) return MCP_RC_INTERNAL_ERROR;
128 -
129 - // Extract URI from params to check for query parameters
130 - struct json_object *uri_obj = NULL;
131 - json_object_object_get_ex(params, "uri", &uri_obj);
132 - const char *uri = json_object_get_string(uri_obj);
133 -
134 - SIMPLE_PATTERN *pattern = NULL;
135 -
136 - // Check if we have a query parameter
137 - if (uri && strstr(uri, "?like=")) {
138 - const char *like_param = strstr(uri, "?like=") + 6; // Skip past "?like="
139 -
140 - // Decode the query parameter
141 - const char *decoded_query = mcp_uri_decode(mcpc, like_param);
142 -
143 - // Create a simple pattern
144 - if (decoded_query && *decoded_query)
145 - pattern = simple_pattern_create(decoded_query, "|", SIMPLE_PATTERN_EXACT, false);
146 - }
147 -
148 - mcp_init_success_result(mcpc, id);
149 - buffer_json_member_add_object(mcpc->result, "result");
150 -
151 - // Add the filtered context categories
152 - rrdcontext_context_registry_json_mcp_categories_array(mcpc->result, pattern);
153 -
154 - // Add instructions
155 - buffer_json_member_add_string(mcpc->result, "instructions",
156 - "Context categories provide a high-level overview of what's being monitored.\n"
157 - "Each category represents a group of related contexts (e.g., 'system.cpu' for CPU metrics).\n\n"
158 - "To explore all contexts within a specific category, use the pattern:\n"
159 - "nd://contexts?like={category}.*\n\n"
160 - "For example, if the cateogy is 'redis' to see all Redis-related contexts:\n"
161 - "nd://contexts?like=redis.*\n\n"
162 - "You can search categories using glob-like patterns with the 'like' parameter:\n"
163 - "nd://context-categories?like=*sql*|*db*|*mongo*\n"
164 - "to find postgresql, mysql, mariadb and mongodb related categories.");
165 -
166 - buffer_json_finalize(mcpc->result);
167 -
168 - if (pattern)
169 - simple_pattern_free(pattern);
170 -
171 - return MCP_RC_OK;
172 -}
173 -
174 -// Size estimation functions for resources
175 -static size_t mcp_resource_contexts_size(void) {
176 - CLEAN_BUFFER *wb = buffer_create(0, NULL);
177 - buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
178 - rrdcontext_context_registry_json_mcp_array(wb, NULL);
179 - buffer_json_finalize(wb);
180 - return buffer_strlen(wb);
181 -}
182 -
183 -static size_t mcp_resource_context_categories_size(void) {
184 - CLEAN_BUFFER *wb = buffer_create(0, NULL);
185 - buffer_json_initialize(wb, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
186 - rrdcontext_context_registry_json_mcp_categories_array(wb, NULL);
187 - buffer_json_finalize(wb);
188 - return buffer_strlen(wb);
189 -}
190 -
191 -// Static array of all available resources
192 -static const MCP_RESOURCE mcp_resources[] = {
193 - {
194 - .name = "contexts",
195 - .uri = "nd://contexts",
196 - .description =
197 - "Primary discovery mechanism for what's being monitored.\n"
198 - "Contexts are the equivalent of charts in Netdata dashboards and they are multi-node and multi-instance.\n"
199 - "Usually contexts have the same set of label keys and common or similar dimensions.\n"
200 - "Supports searches for contexts using glob-like patterns with the 'like=' parameter.\n",
201 - .content_type = CT_APPLICATION_JSON,
202 - .audience = RESOURCE_AUDIENCE_BOTH,
203 - .priority = 1.0,
204 - .read_fn = mcp_resource_read_contexts,
205 - .size_fn = mcp_resource_contexts_size
206 - },
207 - {
208 - .name = "context-categories",
209 - .uri = "nd://context-categories",
210 - .description =
211 - "High-level categories of contexts being monitored.\n"
212 - "Provides a summarized view of monitoring domains by grouping contexts by their prefix.\n"
213 - "Useful for getting a quick overview of what's being monitored without detailed breakdown.\n",
214 - .content_type = CT_APPLICATION_JSON,
215 - .audience = RESOURCE_AUDIENCE_BOTH,
216 - .priority = 0.9,
217 - .read_fn = mcp_resource_read_context_categories,
218 - .size_fn = mcp_resource_context_categories_size
219 - },
220 - // Add more resources here as they are implemented
221 - // Example:
222 - // {
223 - // .name = "nodes",
224 - // .uri = "nd://nodes",
225 - // .description = "Infrastructure discovery...",
226 - // ...
227 - // },
228 -};
229 -
230 -// Static array of all available resource templates
231 -static const MCP_RESOURCE_TEMPLATE mcp_resource_templates[] = {
232 - {
233 - .name = "Contexts Search",
234 - .uri_template = "nd://contexts{?like}",
235 - .description =
236 - "Search for monitoring contexts by matching their names against glob-like patterns.\n"
237 - "The 'like' parameter accepts pipe-separated patterns with wildcards\n"
238 - "(e.g., '?like=*sql*|*db*|*redis*|*mongo*|*{db-name}*' for common database-related contexts).",
239 - .content_type = CT_APPLICATION_JSON,
240 - .audience = RESOURCE_AUDIENCE_BOTH,
241 - .priority = 1.0
242 - },
243 - {
244 - .name = "Context Categories Search",
245 - .uri_template = "nd://context-categories{?like}",
246 - .description =
247 - "Search for high-level context categories by matching their names against glob-like patterns.\n"
248 - "The 'like' parameter accepts pipe-separated patterns with wildcards\n"
249 - "(e.g., '?like=*sql*|*db*|*redis*|*mongo*|*{db-name}*' for common database-related categories).",
250 - .content_type = CT_APPLICATION_JSON,
251 - .audience = RESOURCE_AUDIENCE_BOTH,
252 - .priority = 0.9
253 - },
254 - // Add more templates here as they are implemented
255 -};
256 -
257 -// Implementation of resources/list (transport-agnostic)
258 -static MCP_RETURN_CODE mcp_resources_method_list(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
85 +// Implementation of resources/list
86 +static MCP_RETURN_CODE mcp_resources_method_list(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
87 if (!mcpc || !params || !id) return MCP_RC_INTERNAL_ERROR;
88
89 // Initialize success response
90 mcp_init_success_result(mcpc, id);
91
264 - // Create a resources array object
92 + // Create an empty resource array object
93 buffer_json_member_add_array(mcpc->result, "resources");
266 -
267 - // Iterate through our resources array and add each one
268 - for (size_t i = 0; i < _countof(mcp_resources); i++) {
269 - const MCP_RESOURCE *resource = &mcp_resources[i];
270 -
271 - buffer_json_add_array_item_object(mcpc->result);
272 -
273 - // Add required fields
274 - buffer_json_member_add_string(mcpc->result, "name", resource->name);
275 - buffer_json_member_add_string(mcpc->result, "uri", resource->uri);
276 -
277 - // Add optional fields
278 - if (resource->description) {
279 - buffer_json_member_add_string(mcpc->result, "description", resource->description);
280 - }
281 -
282 - // Convert the content_type enum to string
283 - const char *mime_type = content_type_id2string(resource->content_type);
284 - if (mime_type) {
285 - buffer_json_member_add_string(mcpc->result, "mimeType", mime_type);
286 - }
287 -
288 - // Add size information if available
289 - if (resource->size_fn) {
290 - size_t size = resource->size_fn();
291 - if (size > 0) {
292 - buffer_json_member_add_uint64(mcpc->result, "size", size);
293 - }
294 - }
295 -
296 - // Add audience annotations if specified
297 - if (resource->audience != 0) {
298 - buffer_json_member_add_object(mcpc->result, "annotations");
299 -
300 - buffer_json_member_add_array(mcpc->result, "audience");
301 -
302 - if (resource->audience & RESOURCE_AUDIENCE_USER) {
303 - buffer_json_add_array_item_string(mcpc->result, "user");
304 - }
305 -
306 - if (resource->audience & RESOURCE_AUDIENCE_ASSISTANT) {
307 - buffer_json_add_array_item_string(mcpc->result, "assistant");
308 - }
309 -
310 - buffer_json_array_close(mcpc->result); // Close audience array
311 -
312 - // Add priority if it's non-zero
313 - if (resource->priority > 0) {
314 - buffer_json_member_add_double(mcpc->result, "priority", resource->priority);
315 - }
316 -
317 - buffer_json_object_close(mcpc->result); // Close annotations object
318 - }
319 -
320 - buffer_json_object_close(mcpc->result); // Close resource object
321 - }
322 -
94 buffer_json_array_close(mcpc->result); // Close resources array
324 - buffer_json_object_close(mcpc->result); // Close result object
325 -
326 - // For now, no need for pagination since we have a small number of resources
327 - // If we add many resources later, implement cursor-based pagination here
328 -
95 +
96 + buffer_json_finalize(mcpc->result);
97 return MCP_RC_OK;
98 }
99
332 -// Implementation of resources/read (transport-agnostic)
333 -static MCP_RETURN_CODE mcp_resources_method_read(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
100 +// Implementation of resources/read
101 +static MCP_RETURN_CODE mcp_resources_method_read(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
102 if (!mcpc || id == 0 || !params) return MCP_RC_INTERNAL_ERROR;
103
104 // Extract URI from params
@@ -348,119 +116,40 @@ static MCP_RETURN_CODE mcp_resources_method_read(MCP_CLIENT *mcpc, struct json_o
116
117 netdata_log_debug(D_MCP, "MCP resources/read for URI: %s", uri);
118
351 - // Find the matching resource in our array
352 - for (size_t i = 0; i < _countof(mcp_resources); i++) {
353 - const MCP_RESOURCE *resource = &mcp_resources[i];
354 -
355 - // Get the URI without query parameters for matching
356 - const char *query_start = strchr(uri, '?');
357 - size_t base_uri_length = query_start ? (size_t)(query_start - uri) : strlen(uri);
358 -
359 - // Check if the base URI matches the resource URI
360 - if (strlen(resource->uri) == base_uri_length &&
361 - strncmp(resource->uri, uri, base_uri_length) == 0) {
362 -
363 - // Found matching resource, check if read function exists
364 - if (resource->read_fn) {
365 - // Call the resource-specific read function
366 - return resource->read_fn(mcpc, params, id);
367 - }
368 - else {
369 - // No read function implemented
370 - buffer_strcat(mcpc->error, "Resource reading not implemented");
371 - return MCP_RC_NOT_IMPLEMENTED;
372 - }
373 - }
374 - }
375 -
376 - // No matching resource found
119 + // Since we have no resources, always return not found
120 buffer_sprintf(mcpc->error, "Unknown resource URI: %s", uri);
121 return MCP_RC_NOT_FOUND;
122 }
123
381 -// Implementation of resources/templates/list (transport-agnostic)
382 -static MCP_RETURN_CODE mcp_resources_method_templates_list(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
124 +// Implementation of resources/templates/list
125 +static MCP_RETURN_CODE mcp_resources_method_templates_list(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
126 if (!mcpc || !params || !id) return MCP_RC_INTERNAL_ERROR;
127
128 // Initialize success response
129 mcp_init_success_result(mcpc, id);
130
388 - // Create a resourceTemplates array object
389 - buffer_json_member_add_object(mcpc->result, "result");
131 + // Create an empty resourceTemplates array object
132 buffer_json_member_add_array(mcpc->result, "resourceTemplates");
391 -
392 - // Iterate through our templates array and add each one
393 - for (size_t i = 0; i < _countof(mcp_resource_templates); i++) {
394 - const MCP_RESOURCE_TEMPLATE *template = &mcp_resource_templates[i];
395 -
396 - buffer_json_add_array_item_object(mcpc->result);
397 -
398 - // Add required fields
399 - buffer_json_member_add_string(mcpc->result, "name", template->name);
400 - buffer_json_member_add_string(mcpc->result, "uriTemplate", template->uri_template);
401 -
402 - // Add optional fields
403 - if (template->description) {
404 - buffer_json_member_add_string(mcpc->result, "description", template->description);
405 - }
406 -
407 - // Convert the content_type enum to string
408 - const char *mime_type = content_type_id2string(template->content_type);
409 - if (mime_type) {
410 - buffer_json_member_add_string(mcpc->result, "mimeType", mime_type);
411 - }
412 -
413 - // Add audience annotations if specified
414 - if (template->audience != 0) {
415 - buffer_json_member_add_object(mcpc->result, "annotations");
416 -
417 - buffer_json_member_add_array(mcpc->result, "audience");
418 -
419 - if (template->audience & RESOURCE_AUDIENCE_USER) {
420 - buffer_json_add_array_item_string(mcpc->result, "user");
421 - }
422 -
423 - if (template->audience & RESOURCE_AUDIENCE_ASSISTANT) {
424 - buffer_json_add_array_item_string(mcpc->result, "assistant");
425 - }
426 -
427 - buffer_json_array_close(mcpc->result); // Close audience array
428 -
429 - // Add priority if it's non-zero
430 - if (template->priority > 0) {
431 - buffer_json_member_add_double(mcpc->result, "priority", template->priority);
432 - }
433 -
434 - buffer_json_object_close(mcpc->result); // Close annotations object
435 - }
436 -
437 - buffer_json_object_close(mcpc->result); // Close template object
438 - }
439 -
133 buffer_json_array_close(mcpc->result); // Close resourceTemplates array
441 - buffer_json_object_close(mcpc->result); // Close result object
442 - buffer_json_finalize(mcpc->result);
134
444 - // For now, no need for pagination since we have a small number of templates
445 - // If we add many templates later, implement cursor-based pagination here
446 -
135 + buffer_json_finalize(mcpc->result);
136 return MCP_RC_OK;
137 }
138
139 // Implementation of resources/subscribe (transport-agnostic)
451 -static MCP_RETURN_CODE mcp_resources_method_subscribe(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
140 +static MCP_RETURN_CODE mcp_resources_method_subscribe(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
141 if (!mcpc || !id || !params) return MCP_RC_INTERNAL_ERROR;
142 return MCP_RC_NOT_IMPLEMENTED;
143 }
144
145 // Implementation of resources/unsubscribe (transport-agnostic)
457 -static MCP_RETURN_CODE mcp_resources_method_unsubscribe(MCP_CLIENT *mcpc, struct json_object *params, uint64_t id) {
146 +static MCP_RETURN_CODE mcp_resources_method_unsubscribe(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
147 if (!mcpc || id == 0 || !params) return MCP_RC_INTERNAL_ERROR;
148 return MCP_RC_NOT_IMPLEMENTED;
149 }
150
462 -// Resources namespace method dispatcher (transport-agnostic)
463 -MCP_RETURN_CODE mcp_resources_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id) {
151 +// Resource namespace method dispatcher (transport-agnostic)
152 +MCP_RETURN_CODE mcp_resources_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id) {
153 if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
154
155 netdata_log_debug(D_MCP, "MCP resources method: %s", method);
@@ -487,7 +176,7 @@ MCP_RETURN_CODE mcp_resources_route(MCP_CLIENT *mcpc, const char *method, struct
176 rc = mcp_resources_method_unsubscribe(mcpc, params, id);
177 }
178 else {
490 - // Method not found in resources namespace
179 + // Method not found in resource namespace
180 buffer_sprintf(mcpc->error, "Method 'resources/%s' not implemented yet", method);
181 rc = MCP_RC_NOT_IMPLEMENTED;
182 }
src/web/mcp/mcp-resources.h
+1 -1
@@ -6,6 +6,6 @@
6 #include "mcp.h"
7
8 // Resources namespace method dispatcher (transport-agnostic)
9 -MCP_RETURN_CODE mcp_resources_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id);
9 +MCP_RETURN_CODE mcp_resources_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id);
10
11 #endif // NETDATA_MCP_RESOURCES_H
\ No newline at end of file
src/web/mcp/mcp-system.c deleted
-114
@@ -1,114 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -/**
4 - * MCP System Namespace
5 - *
6 - * The MCP System namespace provides methods for querying and managing the server system.
7 - * These methods provide information about the server's state, health, and performance,
8 - * and allow for basic administrative operations.
9 - *
10 - * Key features of the system namespace:
11 - *
12 - * 1. System Information:
13 - * - Get server health status (system/health)
14 - * - Get detailed version information (system/version)
15 - * - Get server performance metrics (system/metrics)
16 - * - Get current system status (system/status)
17 - *
18 - * 2. System Management:
19 - * - Request server restart (system/restart)
20 - *
21 - * System methods typically require elevated permissions, as they can affect
22 - * the operation of the server and may provide sensitive information.
23 - *
24 - * In the Netdata context, system methods provide:
25 - * - Netdata Agent version and build information
26 - * - Server metrics (CPU, memory usage, uptime, etc.)
27 - * - Runtime configuration status
28 - * - Agent health and operational status
29 - * - Administrative operations for authorized users
30 - *
31 - * These methods are particularly useful for:
32 - * - System administrators monitoring Netdata servers
33 - * - Tools that need to check for version compatibility
34 - * - Health monitoring systems tracking Netdata itself
35 - * - Administrative interfaces
36 - */
37 -
38 -#include "mcp-system.h"
39 -#include "mcp-initialize.h"
40 -#include "config.h" // Include config.h for NETDATA_VERSION
41 -
42 -// Stub implementations for system methods (transport-agnostic)
43 -static MCP_RETURN_CODE mcp_system_method_health(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
44 - buffer_sprintf(mcpc->error, "Method 'system/health' not implemented yet");
45 - return MCP_RC_NOT_IMPLEMENTED;
46 -}
47 -
48 -static MCP_RETURN_CODE mcp_system_method_version(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
49 - if (!mcpc || id == 0) return MCP_RC_ERROR;
50 -
51 - // Initialize success response
52 - mcp_init_success_result(mcpc, id);
53 -
54 - // Add version information
55 - buffer_json_member_add_string(mcpc->result, "name", "Netdata");
56 - buffer_json_member_add_string(mcpc->result, "version", NETDATA_VERSION);
57 - buffer_json_member_add_string(mcpc->result, "mcpVersion", MCP_PROTOCOL_VERSION_2str(MCP_PROTOCOL_VERSION_LATEST));
58 -
59 - // Close the result object
60 - buffer_json_finalize(mcpc->result);
61 -
62 - return MCP_RC_OK;
63 -}
64 -
65 -static MCP_RETURN_CODE mcp_system_method_metrics(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
66 - buffer_sprintf(mcpc->error, "Method 'system/metrics' not implemented yet");
67 - return MCP_RC_NOT_IMPLEMENTED;
68 -}
69 -
70 -static MCP_RETURN_CODE mcp_system_method_restart(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
71 - buffer_sprintf(mcpc->error, "Method 'system/restart' not implemented yet");
72 - return MCP_RC_NOT_IMPLEMENTED;
73 -}
74 -
75 -static MCP_RETURN_CODE mcp_system_method_status(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
76 - buffer_sprintf(mcpc->error, "Method 'system/status' not implemented yet");
77 - return MCP_RC_NOT_IMPLEMENTED;
78 -}
79 -
80 -// System namespace method dispatcher (transport-agnostic)
81 -MCP_RETURN_CODE mcp_system_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id) {
82 - if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
83 -
84 - netdata_log_debug(D_MCP, "MCP system method: %s", method);
85 -
86 - // Flush previous buffers
87 - buffer_flush(mcpc->result);
88 - buffer_flush(mcpc->error);
89 -
90 - MCP_RETURN_CODE rc;
91 -
92 - if (strcmp(method, "health") == 0) {
93 - rc = mcp_system_method_health(mcpc, params, id);
94 - }
95 - else if (strcmp(method, "version") == 0) {
96 - rc = mcp_system_method_version(mcpc, params, id);
97 - }
98 - else if (strcmp(method, "metrics") == 0) {
99 - rc = mcp_system_method_metrics(mcpc, params, id);
100 - }
101 - else if (strcmp(method, "restart") == 0) {
102 - rc = mcp_system_method_restart(mcpc, params, id);
103 - }
104 - else if (strcmp(method, "status") == 0) {
105 - rc = mcp_system_method_status(mcpc, params, id);
106 - }
107 - else {
108 - // Method not found in system namespace
109 - buffer_sprintf(mcpc->error, "Method 'system/%s' not implemented yet", method);
110 - rc = MCP_RC_NOT_IMPLEMENTED;
111 - }
112 -
113 - return rc;
114 -}
src/web/mcp/mcp-system.h deleted
-11
@@ -1,11 +0,0 @@
1 -// SPDX-License-Identifier: GPL-3.0-or-later
2 -
3 -#ifndef NETDATA_MCP_SYSTEM_H
4 -#define NETDATA_MCP_SYSTEM_H
5 -
6 -#include "mcp.h"
7 -
8 -// System namespace method dispatcher (transport-agnostic)
9 -MCP_RETURN_CODE mcp_system_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id);
10 -
11 -#endif // NETDATA_MCP_SYSTEM_H
\ No newline at end of file
src/web/mcp/mcp-test-client/README.md new
+63
@@ -0,0 +1,63 @@
1 +# Netdata MCP Web Client
2 +
3 +A web-based client for testing and interacting with Netdata's Model Context Protocol (MCP) server via WebSocket.
4 +
5 +## Features
6 +
7 +- **WebSocket Connection**: Connect to any MCP server via WebSocket
8 +- **Schema Validation**: Validates tool schemas against MCP specification
9 +- **Custom UI Generator**: Lightweight form generator for tool parameters
10 +- **JSON Pretty Printing**: Advanced formatting with syntax highlighting
11 +- **Request/Response Logging**: Full JSON-RPC message logging with colors
12 +- **Tool Discovery**: Automatically discovers and lists available tools
13 +
14 +## Files
15 +
16 +- `index.html` - Main web client interface
17 +- `mcp-schema-ui-generator.js` - Custom JSON Schema form generator
18 +- `json-pretty-printer.js` - JSON formatter with syntax highlighting
19 +
20 +## Usage
21 +
22 +1. Open `index.html` in a web browser
23 +2. Enter your MCP WebSocket URL (default: `ws://localhost:19999/mcp`)
24 +3. Click "Connect"
25 +4. Use the interface to:
26 + - Initialize the connection
27 + - List available tools
28 + - Call tools with parameters
29 + - View formatted responses
30 +
31 +## Features Details
32 +
33 +### Schema UI Generator
34 +- Validates schemas against MCP specification
35 +- Generates forms for all JSON Schema types
36 +- Supports arrays, objects, enums, and constraints
37 +- Shows validation errors clearly
38 +
39 +### JSON Pretty Printer
40 +- Syntax highlighting with colors
41 +- Visualizes newlines in strings (`\n` shown as badges)
42 +- Detects and formats nested JSON in string values
43 +- Handles large, complex JSON structures
44 +
45 +### WebSocket Client
46 +- Full JSON-RPC 2.0 support
47 +- Request history tracking
48 +- Dynamic tool discovery
49 +- Tab-based interface for raw JSON and form editing
50 +
51 +## Development
52 +
53 +To extend or modify the client:
54 +
55 +1. **Add new flows**: Edit the `flows` object in `index.html`
56 +2. **Customize colors**: Modify the color scheme in `json-pretty-printer.js`
57 +3. **Enhance validation**: Update schema validation in `mcp-schema-ui-generator.js`
58 +
59 +## Browser Requirements
60 +
61 +- Modern browser with WebSocket support
62 +- JavaScript enabled
63 +- No external dependencies required
\ No newline at end of file
src/web/mcp/mcp-test-client/index.html new
+2434
@@ -0,0 +1,2434 @@
1 +<!DOCTYPE html>
2 +<html lang="en">
3 +<head>
4 + <meta charset="UTF-8">
5 + <meta name="viewport" content="width=device-width, initial-scale=1.0">
6 + <title>Netdata MCP Web Client</title>
7 + <!-- Custom MCP Schema UI Generator -->
8 + <script src="mcp-schema-ui-generator.js"></script>
9 + <!-- JSON Pretty Printer -->
10 + <script src="json-pretty-printer.js"></script>
11 + <style>
12 + * {
13 + box-sizing: border-box;
14 + }
15 + body {
16 + font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
17 + margin: 0;
18 + padding: 0;
19 + line-height: 1.6;
20 + color: #333;
21 + overflow: hidden;
22 + height: 100vh;
23 + display: flex;
24 + flex-direction: column;
25 + }
26 + h1 {
27 + color: #0088cc;
28 + margin: 10px;
29 + }
30 + .connection-panel {
31 + background-color: #e2f4ff;
32 + padding: 8px;
33 + border-radius: 5px;
34 + margin: 0 10px 10px 10px;
35 + display: flex;
36 + align-items: center;
37 + flex-wrap: wrap;
38 + gap: 8px;
39 + }
40 + .four-column-layout {
41 + display: grid;
42 + grid-template-columns: 112px 216px 1fr 2fr;
43 + gap: 10px;
44 + flex: 1;
45 + padding: 0 10px 10px 10px;
46 + overflow: hidden;
47 + }
48 + .column {
49 + background-color: #f5f5f5;
50 + border-radius: 5px;
51 + padding: 10px;
52 + box-shadow: 0 2px 5px rgba(0, 0, 0, 0.1);
53 + display: flex;
54 + flex-direction: column;
55 + overflow: hidden;
56 + position: relative;
57 + }
58 + /* Keep methods column scrollable */
59 + .column:nth-child(2) {
60 + overflow: hidden !important;
61 + }
62 + .column:nth-child(2) .column-content {
63 + overflow-y: auto !important;
64 + overflow-x: hidden !important;
65 + }
66 + .column-header {
67 + margin-top: 0;
68 + margin-bottom: 10px;
69 + padding-bottom: 8px;
70 + border-bottom: 1px solid #ddd;
71 + font-size: 1.2em;
72 + color: #0088cc;
73 + display: flex;
74 + justify-content: space-between;
75 + align-items: center;
76 + }
77 + .column-content {
78 + overflow-y: auto;
79 + overflow-x: hidden;
80 + flex: 1;
81 + }
82 + .flow-item, .method-item {
83 + padding: 8px;
84 + margin-bottom: 6px;
85 + border-radius: 5px;
86 + cursor: pointer;
87 + transition: background-color 0.2s;
88 + }
89 + .flow-item {
90 + font-size: 0.9em;
91 + }
92 + .method-item {
93 + font-size: 0.85em;
94 + padding: 10px 8px;
95 + }
96 + .method-item .tool-name {
97 + font-weight: bold;
98 + font-size: 0.95em;
99 + color: #333;
100 + display: block;
101 + margin-bottom: 2px;
102 + }
103 + .method-item .tool-title {
104 + font-size: 0.85em;
105 + color: #666;
106 + display: block;
107 + margin-bottom: 2px;
108 + line-height: 1.3;
109 + }
110 + .method-item .tool-method {
111 + font-size: 0.75em;
112 + color: #999;
113 + display: block;
114 + }
115 + .flow-item:hover, .method-item:hover {
116 + background-color: #e0e0e0;
117 + }
118 + .flow-item.active, .method-item.active {
119 + background-color: #d0e8f2;
120 + font-weight: bold;
121 + }
122 + .method-item.dynamic {
123 + border-left: 3px solid #0088cc;
124 + background-color: #f0f8ff;
125 + }
126 + .method-item.dynamic:hover {
127 + background-color: #e0f0ff;
128 + }
129 + /* Smaller font for method descriptions in tooltips */
130 + .method-item .tooltip {
131 + font-size: 0.8em;
132 + margin-left: 4px;
133 + }
134 + .method-item.history {
135 + border-left: 3px solid #666;
136 + background-color: #f8f8f8;
137 + }
138 + .method-item.history:hover {
139 + background-color: #eeeeee;
140 + }
141 + .method-item.history.error {
142 + border-left-color: #d00;
143 + background-color: #fff5f5;
144 + }
145 + .method-item.history.error:hover {
146 + background-color: #ffeeee;
147 + }
148 + #jsonEditor {
149 + width: 100%;
150 + height: 100%;
151 + font-family: monospace;
152 + font-size: 14px;
153 + padding: 8px;
154 + border: 1px solid #ccc;
155 + border-radius: 5px;
156 + resize: none;
157 + }
158 + /* JSON syntax highlighting */
159 + .json-string {
160 + color: #008800;
161 + font-weight: bold;
162 + display: inline-block;
163 + }
164 + .json-number { color: #0055aa; font-weight: bold; }
165 + .json-boolean-true { color: #008800; font-weight: bold; }
166 + .json-boolean-false { color: #dd0000; font-weight: bold; }
167 + .json-null { color: #666666; font-weight: bold; }
168 + .json-key { color: #555555; }
169 + .json-bracket { color: #aaaaaa; }
170 + .json-comma { color: #aaaaaa; }
171 + .json-colon { color: #aaaaaa; }
172 + button {
173 + background-color: #0088cc;
174 + color: white;
175 + border: none;
176 + padding: 8px 12px;
177 + border-radius: 5px;
178 + cursor: pointer;
179 + font-size: 14px;
180 + }
181 + button:hover {
182 + background-color: #006699;
183 + }
184 + button:disabled {
185 + background-color: #cccccc;
186 + cursor: not-allowed;
187 + }
188 + /* Send without validation button - orange color to indicate caution */
189 + #sendFromFormBtn {
190 + background-color: #ff8c00;
191 + }
192 + #sendFromFormBtn:hover {
193 + background-color: #e67e00;
194 + }
195 + /* Validate & send button - green for safe operation */
196 + #validateAndSendBtn {
197 + background-color: #28a745;
198 + }
199 + #validateAndSendBtn:hover {
200 + background-color: #218838;
201 + }
202 + #validateAndSendBtn:disabled {
203 + background-color: #cccccc;
204 + cursor: not-allowed;
205 + }
206 +
207 + /* Send without validation button - disabled state */
208 + #sendFromFormBtn:disabled {
209 + background-color: #cccccc;
210 + cursor: not-allowed;
211 + }
212 +
213 + /* Validate button - disabled state */
214 + #validateFormBtn:disabled {
215 + background-color: #cccccc;
216 + cursor: not-allowed;
217 + }
218 + .button-row {
219 + display: flex;
220 + gap: 8px;
221 + margin-top: 8px;
222 + }
223 + .status {
224 + font-weight: bold;
225 + }
226 + .connected {
227 + color: green;
228 + }
229 + .disconnected {
230 + color: red;
231 + }
232 + .tooltip {
233 + position: relative;
234 + display: inline-block;
235 + cursor: help;
236 + }
237 + .tooltip .tooltiptext {
238 + visibility: hidden;
239 + width: 300px;
240 + background-color: #555;
241 + color: #fff;
242 + text-align: left;
243 + border-radius: 6px;
244 + padding: 10px;
245 + position: absolute;
246 + z-index: 9999;
247 + bottom: 125%;
248 + left: 50%;
249 + transform: translateX(-50%);
250 + opacity: 0;
251 + transition: opacity 0.3s;
252 + font-size: 12px;
253 + line-height: 1.4;
254 + }
255 + /* Specific positioning for method tooltips */
256 + .method-item.tooltip .tooltiptext {
257 + position: fixed;
258 + z-index: 9999;
259 + width: 300px;
260 + max-width: 400px;
261 + height: auto !important;
262 + min-height: fit-content;
263 + white-space: normal;
264 + word-wrap: break-word;
265 + }
266 + .tooltip:hover .tooltiptext {
267 + visibility: visible;
268 + opacity: 1;
269 + }
270 +
271 + /* Ensure JSON content wraps properly */
272 + pre, #responseViewer {
273 + white-space: pre-wrap !important;
274 + word-break: break-word !important;
275 + overflow-x: auto !important;
276 + }
277 +
278 + /* Tab styles */
279 + .tabs {
280 + display: flex;
281 + border-bottom: 2px solid #ddd;
282 + margin-bottom: 10px;
283 + }
284 + .tab {
285 + padding: 8px 16px;
286 + cursor: pointer;
287 + background-color: #f5f5f5;
288 + border: 1px solid #ddd;
289 + border-bottom: none;
290 + margin-right: 4px;
291 + transition: background-color 0.2s;
292 + font-size: 0.9em;
293 + pointer-events: auto;
294 + position: relative;
295 + z-index: 10;
296 + }
297 + .tab:hover {
298 + background-color: #e0e0e0;
299 + }
300 + .tab.active {
301 + background-color: white;
302 + font-weight: bold;
303 + border-bottom: 2px solid white;
304 + margin-bottom: -2px;
305 + }
306 + .tab-content {
307 + display: none;
308 + height: 100%;
309 + position: relative;
310 + background-color: white;
311 + opacity: 0;
312 + visibility: hidden;
313 + }
314 + .tab-content.active {
315 + display: block;
316 + opacity: 1;
317 + visibility: visible;
318 + }
319 + #editRequestContent.active {
320 + display: flex !important;
321 + }
322 + /* Custom Schema Form Editor */
323 + #editRequestContent {
324 + height: 100%;
325 + overflow: hidden;
326 + flex-direction: column;
327 + }
328 + #schemaFormEditor {
329 + flex: 1;
330 + overflow-y: auto;
331 + padding: 10px;
332 + background-color: #f8f8f8;
333 + border: 1px solid #ddd;
334 + border-radius: 4px;
335 + min-height: 0;
336 + }
337 + #noSchemaMessage {
338 + padding: 20px;
339 + text-align: center;
340 + color: #666;
341 + font-style: italic;
342 + }
343 + /* Response viewer styles */
344 + #responseViewer {
345 + font-family: 'Courier New', monospace;
346 + font-size: 13px;
347 + line-height: 1.5;
348 + background-color: #f8f8f8;
349 + padding: 10px !important;
350 + border-radius: 4px;
351 + overflow-y: auto;
352 + overflow-x: hidden;
353 + }
354 + /* JSON Pretty Printer styles */
355 + .json-entry {
356 + margin-bottom: 15px;
357 + padding-bottom: 15px;
358 + border-bottom: 1px solid #e0e0e0;
359 + }
360 + .json-entry:last-child {
361 + border-bottom: none;
362 + }
363 + .json-timestamp {
364 + color: #666;
365 + font-size: 12px;
366 + margin-bottom: 5px;
367 + }
368 + .json-direction {
369 + font-weight: bold;
370 + margin-right: 10px;
371 + }
372 + .json-sent {
373 + color: #0088cc;
374 + }
375 + .json-received {
376 + color: #00aa00;
377 + }
378 + #clearLogBtn {
379 + position: absolute;
380 + top: 5px;
381 + right: 5px;
382 + padding: 4px 8px;
383 + font-size: 12px;
384 + background-color: #dc3545;
385 + z-index: 10;
386 + }
387 + #clearLogBtn:hover {
388 + background-color: #c82333;
389 + }
390 + .response-nav-buttons {
391 + position: absolute;
392 + top: 5px;
393 + right: 80px;
394 + z-index: 10;
395 + display: flex;
396 + gap: 5px;
397 + }
398 + .nav-button {
399 + padding: 4px 8px;
400 + font-size: 12px;
401 + background-color: #6c757d;
402 + color: white;
403 + border: none;
404 + border-radius: 3px;
405 + cursor: pointer;
406 + }
407 + .nav-button:hover {
408 + background-color: #5a6268;
409 + }
410 + .nav-button:disabled {
411 + background-color: #cccccc;
412 + cursor: not-allowed;
413 + }
414 + /* Method count badge */
415 + .method-count {
416 + background-color: #0088cc;
417 + color: white;
418 + padding: 2px 6px;
419 + border-radius: 10px;
420 + font-size: 0.8em;
421 + margin-left: 5px;
422 + }
423 + /* History count */
424 + .history-count {
425 + background-color: #666;
426 + color: white;
427 + padding: 2px 6px;
428 + border-radius: 10px;
429 + font-size: 0.7em;
430 + margin-left: auto;
431 + }
432 + .method-header {
433 + display: flex;
434 + align-items: center;
435 + width: 100%;
436 + }
437 + .method-text {
438 + flex: 1;
439 + }
440 + /* Validation message */
441 + .validation-message {
442 + background-color: #f8d7da;
443 + color: #721c24;
444 + padding: 8px;
445 + border-radius: 4px;
446 + margin: 10px;
447 + font-size: 0.85em;
448 + border: 1px solid #f5c6cb;
449 + }
450 + .validation-message.success {
451 + background-color: #d4edda;
452 + color: #155724;
453 + border-color: #c3e6cb;
454 + }
455 + /* Saved indicator */
456 + .saved-indicator {
457 + position: absolute;
458 + top: 8px;
459 + right: 120px;
460 + background-color: #28a745;
461 + color: white;
462 + padding: 2px 6px;
463 + border-radius: 3px;
464 + font-size: 11px;
465 + opacity: 0;
466 + transition: opacity 0.3s;
467 + z-index: 10;
468 + }
469 + .saved-indicator.show {
470 + opacity: 1;
471 + }
472 + /* Storage controls */
473 + .storage-controls {
474 + display: flex;
475 + gap: 5px;
476 + margin-left: auto;
477 + font-size: 12px;
478 + }
479 + .storage-btn {
480 + padding: 2px 6px;
481 + font-size: 11px;
482 + background-color: #6c757d;
483 + color: white;
484 + border: none;
485 + border-radius: 3px;
486 + cursor: pointer;
487 + }
488 + .storage-btn:hover {
489 + background-color: #5a6268;
490 + }
491 + .storage-btn.clear {
492 + background-color: #dc3545;
493 + }
494 + .storage-btn.clear:hover {
495 + background-color: #c82333;
496 + }
497 + /* JSON header line */
498 + .json-header-line {
499 + display: flex;
500 + justify-content: space-between;
501 + align-items: center;
502 + width: 100%;
503 + }
504 +
505 + /* Entry-level controls */
506 + .entry-controls {
507 + display: flex;
508 + gap: 3px;
509 + }
510 + .entry-btn {
511 + background: #6c757d;
512 + color: white;
513 + border: none;
514 + padding: 2px 6px;
515 + border-radius: 3px;
516 + font-size: 11px;
517 + cursor: pointer;
518 + }
519 + .entry-btn:hover {
520 + background: #5a6268;
521 + }
522 +
523 + /* Modal styles */
524 + .modal {
525 + display: none;
526 + position: fixed;
527 + z-index: 10000;
528 + left: 0;
529 + top: 0;
530 + width: 100%;
531 + height: 100%;
532 + background-color: rgba(0,0,0,0.5);
533 + }
534 + .modal-content {
535 + background-color: #fefefe;
536 + margin: 5% auto;
537 + padding: 20px;
538 + border: 1px solid #888;
539 + border-radius: 8px;
540 + width: 80%;
541 + max-width: 800px;
542 + max-height: 80vh;
543 + display: flex;
544 + flex-direction: column;
545 + }
546 + .modal-header {
547 + display: flex;
548 + justify-content: space-between;
549 + align-items: center;
550 + margin-bottom: 15px;
551 + padding-bottom: 10px;
552 + border-bottom: 1px solid #ddd;
553 + }
554 + .modal-close {
555 + color: #aaa;
556 + font-size: 28px;
557 + font-weight: bold;
558 + cursor: pointer;
559 + line-height: 20px;
560 + }
561 + .modal-close:hover,
562 + .modal-close:focus {
563 + color: #000;
564 + }
565 + .modal-body {
566 + flex: 1;
567 + overflow: hidden;
568 + display: flex;
569 + flex-direction: column;
570 + }
571 + .modal-textarea {
572 + flex: 1;
573 + width: 100%;
574 + font-family: monospace;
575 + font-size: 13px;
576 + padding: 10px;
577 + border: 1px solid #ccc;
578 + border-radius: 4px;
579 + resize: none;
580 + min-height: 300px;
581 + }
582 + .modal-footer {
583 + margin-top: 15px;
584 + display: flex;
585 + justify-content: flex-end;
586 + gap: 10px;
587 + }
588 + .import-example {
589 + background-color: #f5f5f5;
590 + padding: 8px;
591 + border-radius: 4px;
592 + font-size: 12px;
593 + margin-bottom: 10px;
594 + color: #666;
595 + }
596 + </style>
597 +</head>
598 +<body>
599 + <div class="connection-panel">
600 + <label for="serverUrl">Netdata MCP Test Client:</label>
601 + <input type="text" id="serverUrl" value="ws://localhost:19999/mcp" style="width: 300px;">
602 + <button id="connectBtn">Connect</button>
603 + <button id="connectAndInitBtn">Connect and Handshake</button>
604 + <button id="disconnectBtn" disabled>Disconnect</button>
605 + <span id="status" class="status disconnected">Disconnected</span>
606 + </div>
607 +
608 + <div class="four-column-layout">
609 + <!-- First Column - Flows -->
610 + <div class="column">
611 + <h3 class="column-header">Flows</h3>
612 + <div class="column-content" id="flowsList">
613 + <div class="flow-item active" data-flow="initialization">Initialize</div>
614 + <div class="flow-item" data-flow="tools">Tools</div>
615 + <div class="flow-item" data-flow="prompts">Prompts</div>
616 + <div class="flow-item" data-flow="resources">Resources</div>
617 + <div class="flow-item" data-flow="logging">Logging</div>
618 + <div class="flow-item" data-flow="sampling">Sampling</div>
619 + <div class="flow-item" data-flow="roots">Roots</div>
620 + <div class="flow-item" data-flow="custom">History</div>
621 + </div>
622 + </div>
623 +
624 + <!-- Second Column - Methods -->
625 + <div class="column">
626 + <h3 class="column-header">
627 + <span>Methods</span>
628 + <span id="methodCount" class="method-count">0</span>
629 + </h3>
630 + <div class="column-content" id="methodsList"></div>
631 + </div>
632 +
633 + <!-- Third Column - Request -->
634 + <div class="column">
635 + <h3 class="column-header">
636 + <span>Request</span>
637 + <span id="currentToolName" style="font-weight: normal; font-size: 0.9em; color: #666;"></span>
638 + <div class="storage-controls">
639 + <button id="clearParamsBtn" class="storage-btn clear" title="Clear saved parameters for current tool">Clear Params</button>
640 + <button id="clearHistoryBtn" class="storage-btn clear" title="Clear request history">Clear History</button>
641 + </div>
642 + <div id="savedIndicator" class="saved-indicator">Saved!</div>
643 + </h3>
644 + <div class="tabs">
645 + <div class="tab" onclick="switchTab('rawRequest')">Raw JSON</div>
646 + <div class="tab active" onclick="switchTab('editRequest')">Edit Form</div>
647 + <div class="tab" onclick="switchTab('viewSchema')">Schema</div>
648 + </div>
649 + <div class="column-content" style="display: flex; flex-direction: column; padding: 0; position: relative;">
650 + <div id="rawRequestContent" class="tab-content" style="position: absolute; inset: 0; display: flex; flex-direction: column;">
651 + <textarea id="jsonEditor" style="flex: 1;"></textarea>
652 + <div class="button-row" style="padding: 8px;">
653 + <button id="sendBtn" disabled>Send</button>
654 + <button id="formatBtn">Format JSON</button>
655 + <button id="copyBtn">Copy</button>
656 + <button id="importLLMBtn">Import from LLM</button>
657 + </div>
658 + </div>
659 + <div id="editRequestContent" class="tab-content active" style="position: absolute; inset: 0; overflow: hidden;">
660 + <div id="noSchemaMessage">
661 + Select a method to see its parameters
662 + </div>
663 + <div id="schemaFormEditor" style="display: none;">
664 + <!-- Form will be rendered here -->
665 + </div>
666 + <div class="button-row" style="padding: 8px; background: white; border-top: 1px solid #ddd;">
667 + <button id="sendFromFormBtn" disabled title="Send request without validation (for testing server-side validation)">Send w/o Validation</button>
668 + <button id="validateFormBtn" title="Validate form data against schema">Validate</button>
669 + <button id="validateAndSendBtn" disabled title="Validate first, then send request">Validate & Send</button>
670 + </div>
671 + </div>
672 + <div id="viewSchemaContent" class="tab-content" style="position: absolute; inset: 0; overflow: hidden;">
673 + <div id="schemaViewer" style="height: 100%; overflow-y: auto; padding: 10px; background-color: #f8f8f8; font-family: 'Courier New', monospace; font-size: 13px;">
674 + <div id="noSchemaAvailable" style="text-align: center; color: #666; padding: 20px;">
675 + No schema available. Select a tool to view its schema.
676 + </div>
677 + <pre id="schemaDisplay" style="display: none; margin: 0;"></pre>
678 + </div>
679 + </div>
680 + </div>
681 + </div>
682 +
683 + <!-- Fourth Column - Response -->
684 + <div class="column">
685 + <h3 class="column-header">
686 + Response
687 + <div class="response-nav-buttons">
688 + <button id="prevResponseBtn" class="nav-button" title="Previous response (↑)">↑</button>
689 + <button id="nextResponseBtn" class="nav-button" title="Next response (↓)">↓</button>
690 + </div>
691 + <button id="clearLogBtn">Clear</button>
692 + </h3>
693 + <pre id="responseViewer" class="column-content" style="padding: 0; margin: 0;"></pre>
694 + </div>
695 + </div>
696 +
697 + <!-- Import from LLM Modal -->
698 + <div id="importLLMModal" class="modal">
699 + <div class="modal-content">
700 + <div class="modal-header">
701 + <h2>Import Request from LLM</h2>
702 + <span class="modal-close" onclick="closeImportModal()">&times;</span>
703 + </div>
704 + <div class="modal-body">
705 + <div class="import-example">
706 + Example: Paste LLM requests that use backticks (`) instead of quotes ("). The JSON-RPC wrapper will be added automatically.
707 + </div>
708 + <textarea id="llmImportTextarea" class="modal-textarea" placeholder='Paste your LLM request here, e.g.:
709 +{
710 + `after`: `-604800`,
711 + `before`: `0`,
712 + `cardinality_limit`: 20,
713 + `instances`: [
714 + `*lab-child-host1*`
715 + ],
716 + `nodes`: [
717 + `beast`
718 + ]
719 +}'></textarea>
720 + </div>
721 + <div class="modal-footer">
722 + <button onclick="closeImportModal()">Cancel</button>
723 + <button onclick="importFromLLM()" style="background-color: #28a745;">Import</button>
724 + </div>
725 + </div>
726 + </div>
727 +
728 + <script>
729 + // Global variables
730 + let ws = null;
731 + let currentRequestId = 1;
732 + let serverCapabilities = {};
733 + let isUpdatingFromCode = false; // Flag to prevent recursive updates between JSON editor and form
734 + let availableTools = {};
735 + let availablePrompts = {};
736 + let availableResources = {};
737 + let requestHistory = [];
738 + let currentActiveTab = 'editRequest';
739 + let schemaFormGenerator = null;
740 + let currentMethodSchema = null;
741 + let jsonPrinter = null;
742 + let pendingRequests = new Map(); // Track request timestamps by ID
743 + let responseEntries = []; // Track response entries for navigation
744 + let currentResponseIndex = -1;
745 +
746 + // Local storage keys
747 + const STORAGE_KEYS = {
748 + TOOL_PARAMS: 'mcp_tool_params',
749 + REQUEST_HISTORY: 'mcp_request_history',
750 + SERVER_URL: 'mcp_server_url'
751 + };
752 +
753 + // Local Storage utility functions
754 + function saveToLocalStorage(key, data) {
755 + try {
756 + localStorage.setItem(key, JSON.stringify(data));
757 + } catch (e) {
758 + console.warn('Failed to save to localStorage:', e);
759 + }
760 + }
761 +
762 + function loadFromLocalStorage(key, defaultValue = null) {
763 + try {
764 + const item = localStorage.getItem(key);
765 + return item ? JSON.parse(item) : defaultValue;
766 + } catch (e) {
767 + console.warn('Failed to load from localStorage:', e);
768 + return defaultValue;
769 + }
770 + }
771 +
772 + // Save tool parameters
773 + function saveToolParams(toolName, params) {
774 + const savedParams = loadFromLocalStorage(STORAGE_KEYS.TOOL_PARAMS, {});
775 + savedParams[toolName] = {
776 + params: params,
777 + timestamp: Date.now()
778 + };
779 + saveToLocalStorage(STORAGE_KEYS.TOOL_PARAMS, savedParams);
780 + }
781 +
782 + // Load tool parameters
783 + function loadToolParams(toolName) {
784 + const savedParams = loadFromLocalStorage(STORAGE_KEYS.TOOL_PARAMS, {});
785 + return savedParams[toolName]?.params || {};
786 + }
787 +
788 + // Save request to history
789 + function saveRequestToHistory(request, method, toolName = null, error = false) {
790 + const historyItem = {
791 + id: Date.now() + '_' + Math.random().toString(36).substr(2, 9),
792 + timestamp: new Date().toISOString(),
793 + displayTime: new Date().toLocaleString(),
794 + method: method,
795 + toolName: toolName,
796 + request: JSON.parse(JSON.stringify(request)), // Deep clone to avoid reference issues
797 + error: error
798 + };
799 +
800 + let history = loadFromLocalStorage(STORAGE_KEYS.REQUEST_HISTORY, []);
801 + history.unshift(historyItem);
802 +
803 + // Keep only last 20 requests
804 + if (history.length > 20) {
805 + history = history.slice(0, 20);
806 + }
807 +
808 + saveToLocalStorage(STORAGE_KEYS.REQUEST_HISTORY, history);
809 + return historyItem;
810 + }
811 +
812 + // Load request history
813 + function loadRequestHistory() {
814 + return loadFromLocalStorage(STORAGE_KEYS.REQUEST_HISTORY, []);
815 + }
816 +
817 + // Save server URL
818 + function saveServerUrl(url) {
819 + saveToLocalStorage(STORAGE_KEYS.SERVER_URL, url);
820 + }
821 +
822 + // Load server URL
823 + function loadServerUrl() {
824 + return loadFromLocalStorage(STORAGE_KEYS.SERVER_URL, 'ws://localhost:19999/mcp');
825 + }
826 +
827 + // Clear saved parameters for current tool
828 + function clearCurrentToolParams() {
829 + if (!currentMethodSchema || !currentMethodSchema.toolName) {
830 + alert('No tool selected or tool has no parameters to clear');
831 + return;
832 + }
833 +
834 + if (confirm(`Clear saved parameters for ${currentMethodSchema.toolName}?`)) {
835 + const savedParams = loadFromLocalStorage(STORAGE_KEYS.TOOL_PARAMS, {});
836 + delete savedParams[currentMethodSchema.toolName];
837 + saveToLocalStorage(STORAGE_KEYS.TOOL_PARAMS, savedParams);
838 +
839 + // Reset form to defaults
840 + if (schemaFormGenerator && currentMethodSchema.schema) {
841 + schemaFormGenerator.setSchema(currentMethodSchema.schema.inputSchema, {});
842 + }
843 +
844 + alert('Parameters cleared!');
845 + }
846 + }
847 +
848 + // Clear request history
849 + function clearRequestHistory() {
850 + if (confirm('Clear all request history? This cannot be undone.')) {
851 + saveToLocalStorage(STORAGE_KEYS.REQUEST_HISTORY, []);
852 + requestHistory = [];
853 +
854 + // Refresh custom flow if currently displayed
855 + const activeFlow = document.querySelector('.flow-item.active');
856 + if (activeFlow && activeFlow.dataset.flow === 'custom') {
857 + displayMethods('custom');
858 + }
859 +
860 + // Update history flow indicator
861 + updateHistoryFlowIndicator();
862 +
863 + alert('Request history cleared!');
864 + }
865 + }
866 +
867 + // Show save indicator
868 + function showSaveIndicator() {
869 + savedIndicator.classList.add('show');
870 + setTimeout(() => {
871 + savedIndicator.classList.remove('show');
872 + }, 2000);
873 + }
874 +
875 + // Flow definitions
876 + const flows = {
877 + initialization: {
878 + name: 'Initialization',
879 + methods: [
880 + {
881 + name: 'initialize',
882 + type: 'request',
883 + description: 'Initialize the connection with the server and exchange capabilities',
884 + template: {
885 + jsonrpc: "2.0",
886 + id: 1,
887 + method: "initialize",
888 + params: {
889 + protocolVersion: "2024-11-05",
890 + capabilities: {
891 + roots: { listChanged: true },
892 + sampling: {}
893 + },
894 + clientInfo: {
895 + name: "Netdata MCP Test Client",
896 + version: "1.0.0"
897 + }
898 + }
899 + }
900 + },
901 + {
902 + name: 'initialized',
903 + type: 'notification',
904 + description: 'Notify the server that initialization is complete',
905 + template: {
906 + jsonrpc: "2.0",
907 + method: "notifications/initialized"
908 + }
909 + },
910 + {
911 + name: 'ping',
912 + type: 'request',
913 + description: 'Ping the server to check if it\'s alive',
914 + template: {
915 + jsonrpc: "2.0",
916 + id: 1,
917 + method: "ping"
918 + }
919 + }
920 + ]
921 + },
922 + tools: {
923 + name: 'Tools',
924 + methods: [
925 + {
926 + name: 'tools/list',
927 + type: 'request',
928 + description: 'List all available tools from the server',
929 + template: {
930 + jsonrpc: "2.0",
931 + id: 1,
932 + method: "tools/list"
933 + }
934 + },
935 + {
936 + name: 'tools/call',
937 + type: 'request',
938 + description: 'Call a specific tool with arguments',
939 + template: {
940 + jsonrpc: "2.0",
941 + id: 1,
942 + method: "tools/call",
943 + params: {
944 + name: "example_tool",
945 + arguments: {}
946 + }
947 + }
948 + }
949 + ]
950 + },
951 + prompts: {
952 + name: 'Prompts',
953 + methods: [
954 + {
955 + name: 'prompts/list',
956 + type: 'request',
957 + description: 'List all available prompts from the server',
958 + template: {
959 + jsonrpc: "2.0",
960 + id: 1,
961 + method: "prompts/list"
962 + }
963 + },
964 + {
965 + name: 'prompts/get',
966 + type: 'request',
967 + description: 'Get a specific prompt by name',
968 + template: {
969 + jsonrpc: "2.0",
970 + id: 1,
971 + method: "prompts/get",
972 + params: {
973 + name: "example_prompt",
974 + arguments: {}
975 + }
976 + }
977 + }
978 + ]
979 + },
980 + resources: {
981 + name: 'Resources',
982 + methods: [
983 + {
984 + name: 'resources/list',
985 + type: 'request',
986 + description: 'List all available resources from the server',
987 + template: {
988 + jsonrpc: "2.0",
989 + id: 1,
990 + method: "resources/list"
991 + }
992 + },
993 + {
994 + name: 'resources/read',
995 + type: 'request',
996 + description: 'Read a specific resource by URI',
997 + template: {
998 + jsonrpc: "2.0",
999 + id: 1,
1000 + method: "resources/read",
1001 + params: {
1002 + uri: "example://resource"
1003 + }
1004 + }
1005 + },
1006 + {
1007 + name: 'resources/subscribe',
1008 + type: 'request',
1009 + description: 'Subscribe to updates for a specific resource',
1010 + template: {
1011 + jsonrpc: "2.0",
1012 + id: 1,
1013 + method: "resources/subscribe",
1014 + params: {
1015 + uri: "example://resource"
1016 + }
1017 + }
1018 + },
1019 + {
1020 + name: 'resources/unsubscribe',
1021 + type: 'request',
1022 + description: 'Unsubscribe from updates for a specific resource',
1023 + template: {
1024 + jsonrpc: "2.0",
1025 + id: 1,
1026 + method: "resources/unsubscribe",
1027 + params: {
1028 + uri: "example://resource"
1029 + }
1030 + }
1031 + }
1032 + ]
1033 + },
1034 + logging: {
1035 + name: 'Logging',
1036 + methods: [
1037 + {
1038 + name: 'logging/setLevel',
1039 + type: 'request',
1040 + description: 'Set the logging level for server messages',
1041 + template: {
1042 + jsonrpc: "2.0",
1043 + id: 1,
1044 + method: "logging/setLevel",
1045 + params: {
1046 + level: "info"
1047 + }
1048 + }
1049 + }
1050 + ]
1051 + },
1052 + sampling: {
1053 + name: 'Sampling',
1054 + methods: [
1055 + {
1056 + name: 'sampling/createMessage',
1057 + type: 'request',
1058 + description: 'Request the server to create a message using LLM',
1059 + template: {
1060 + jsonrpc: "2.0",
1061 + id: 1,
1062 + method: "sampling/createMessage",
1063 + params: {
1064 + messages: [
1065 + {
1066 + role: "user",
1067 + content: {
1068 + type: "text",
1069 + text: "Hello"
1070 + }
1071 + }
1072 + ],
1073 + maxTokens: 100
1074 + }
1075 + }
1076 + }
1077 + ]
1078 + },
1079 + roots: {
1080 + name: 'Roots',
1081 + methods: [
1082 + {
1083 + name: 'roots/list',
1084 + type: 'request',
1085 + description: 'List root directories the client has access to',
1086 + template: {
1087 + jsonrpc: "2.0",
1088 + id: 1,
1089 + method: "roots/list",
1090 + params: {}
1091 + }
1092 + }
1093 + ]
1094 + },
1095 + custom: {
1096 + name: 'Request History',
1097 + methods: []
1098 + }
1099 + };
1100 +
1101 + // DOM elements
1102 + const connectBtn = document.getElementById('connectBtn');
1103 + const connectAndInitBtn = document.getElementById('connectAndInitBtn');
1104 + const disconnectBtn = document.getElementById('disconnectBtn');
1105 + const sendBtn = document.getElementById('sendBtn');
1106 + const sendFromFormBtn = document.getElementById('sendFromFormBtn');
1107 + const validateFormBtn = document.getElementById('validateFormBtn');
1108 + const validateAndSendBtn = document.getElementById('validateAndSendBtn');
1109 + const formatBtn = document.getElementById('formatBtn');
1110 + const copyBtn = document.getElementById('copyBtn');
1111 + const importLLMBtn = document.getElementById('importLLMBtn');
1112 + const clearLogBtn = document.getElementById('clearLogBtn');
1113 + const prevResponseBtn = document.getElementById('prevResponseBtn');
1114 + const nextResponseBtn = document.getElementById('nextResponseBtn');
1115 + const clearParamsBtn = document.getElementById('clearParamsBtn');
1116 + const clearHistoryBtn = document.getElementById('clearHistoryBtn');
1117 + const savedIndicator = document.getElementById('savedIndicator');
1118 + const jsonEditor = document.getElementById('jsonEditor');
1119 + const responseViewer = document.getElementById('responseViewer');
1120 + const statusElement = document.getElementById('status');
1121 + const serverUrlInput = document.getElementById('serverUrl');
1122 + const flowsList = document.getElementById('flowsList');
1123 + const methodsList = document.getElementById('methodsList');
1124 + const noSchemaMessage = document.getElementById('noSchemaMessage');
1125 + const schemaFormEditor = document.getElementById('schemaFormEditor');
1126 +
1127 + // Event listeners
1128 + connectBtn.addEventListener('click', connect);
1129 + connectAndInitBtn.addEventListener('click', connectAndInitialize);
1130 + disconnectBtn.addEventListener('click', disconnect);
1131 + sendBtn.addEventListener('click', sendRequest);
1132 + sendFromFormBtn.addEventListener('click', sendFromForm);
1133 + validateFormBtn.addEventListener('click', validateForm);
1134 + validateAndSendBtn.addEventListener('click', validateAndSendFromForm);
1135 + formatBtn.addEventListener('click', formatJSON);
1136 + copyBtn.addEventListener('click', copyJSON);
1137 + importLLMBtn.addEventListener('click', openImportModal);
1138 + clearLogBtn.addEventListener('click', clearLog);
1139 + prevResponseBtn.addEventListener('click', navigateToPrevResponse);
1140 + nextResponseBtn.addEventListener('click', navigateToNextResponse);
1141 + clearParamsBtn.addEventListener('click', clearCurrentToolParams);
1142 + clearHistoryBtn.addEventListener('click', clearRequestHistory);
1143 +
1144 + // JSON editor change detection
1145 + let jsonEditorChangeTimer = null;
1146 +
1147 + jsonEditor.addEventListener('input', () => {
1148 + // Skip if this change was triggered by code (not user)
1149 + if (isUpdatingFromCode) return;
1150 +
1151 + // Debounce the update to avoid excessive updates while typing
1152 + clearTimeout(jsonEditorChangeTimer);
1153 + jsonEditorChangeTimer = setTimeout(() => {
1154 + // Only update form if we're on the edit tab and have a schema
1155 + if (currentActiveTab === 'editRequest' && schemaFormGenerator) {
1156 + updateFormFromRaw();
1157 + }
1158 + }, 500); // 500ms delay
1159 + });
1160 +
1161 + // Flow selection
1162 + flowsList.addEventListener('click', (e) => {
1163 + const flowItem = e.target.closest('.flow-item');
1164 + if (flowItem) {
1165 + document.querySelectorAll('.flow-item').forEach(item => item.classList.remove('active'));
1166 + flowItem.classList.add('active');
1167 + const flowName = flowItem.dataset.flow;
1168 + displayMethods(flowName);
1169 + }
1170 + });
1171 +
1172 + // Initialize from storage
1173 + function initializeFromStorage() {
1174 + // Load server URL
1175 + const savedServerUrl = loadServerUrl();
1176 + if (savedServerUrl) {
1177 + serverUrlInput.value = savedServerUrl;
1178 + }
1179 +
1180 + // Load request history
1181 + requestHistory = loadRequestHistory();
1182 +
1183 + // Update history flow indicator
1184 + updateHistoryFlowIndicator();
1185 + }
1186 +
1187 + // Update history flow indicator
1188 + function updateHistoryFlowIndicator() {
1189 + const historyCount = loadRequestHistory().length;
1190 + const historyFlow = document.querySelector('[data-flow="custom"]');
1191 +
1192 + if (historyCount > 0) {
1193 + historyFlow.textContent = `History (${historyCount})`;
1194 + historyFlow.style.fontWeight = 'bold';
1195 + historyFlow.style.color = '#0088cc';
1196 + } else {
1197 + historyFlow.textContent = 'History';
1198 + historyFlow.style.fontWeight = '';
1199 + historyFlow.style.color = '';
1200 + }
1201 + }
1202 +
1203 + // Initialize JSON pretty printer
1204 + jsonPrinter = new JSONPrettyPrinter({
1205 + indent: 2,
1206 + visualizeNewlines: true,
1207 + detectNestedJSON: true,
1208 + syntaxHighlight: true
1209 + });
1210 +
1211 + // Initialize with saved data
1212 + initializeFromStorage();
1213 +
1214 + // Initialize with first flow
1215 + displayMethods('initialization');
1216 + updateNavButtons();
1217 +
1218 + // Keyboard shortcuts for response navigation
1219 + document.addEventListener('keydown', (e) => {
1220 + // Only work when not focused on an input field
1221 + if (e.target.tagName === 'INPUT' || e.target.tagName === 'TEXTAREA' || e.target.tagName === 'SELECT') {
1222 + return;
1223 + }
1224 +
1225 + if (e.key === 'ArrowUp' && e.ctrlKey) {
1226 + e.preventDefault();
1227 + navigateToPrevResponse();
1228 + } else if (e.key === 'ArrowDown' && e.ctrlKey) {
1229 + e.preventDefault();
1230 + navigateToNextResponse();
1231 + }
1232 + });
1233 +
1234 + // Tab switching
1235 + function switchTab(tabName) {
1236 + currentActiveTab = tabName;
1237 +
1238 + // Update tab active states
1239 + document.querySelectorAll('.tab').forEach(tab => {
1240 + tab.classList.remove('active');
1241 + const tabText = tab.textContent;
1242 + if ((tabName === 'rawRequest' && tabText.includes('Raw JSON')) ||
1243 + (tabName === 'editRequest' && tabText.includes('Edit Form')) ||
1244 + (tabName === 'viewSchema' && tabText.includes('Schema'))) {
1245 + tab.classList.add('active');
1246 + }
1247 + });
1248 +
1249 + // Show/hide content
1250 + document.getElementById('rawRequestContent').classList.toggle('active', tabName === 'rawRequest');
1251 + document.getElementById('editRequestContent').classList.toggle('active', tabName === 'editRequest');
1252 + document.getElementById('viewSchemaContent').classList.toggle('active', tabName === 'viewSchema');
1253 +
1254 + // Update form from raw JSON when switching to edit tab
1255 + if (tabName === 'editRequest' && currentMethodSchema) {
1256 + updateFormFromRaw();
1257 + }
1258 + }
1259 +
1260 + // Display methods for a flow
1261 + function displayMethods(flowName) {
1262 + const flow = flows[flowName];
1263 + if (!flow) return;
1264 +
1265 + methodsList.innerHTML = '';
1266 + let allMethods = [...flow.methods];
1267 +
1268 + // Add dynamic methods for tools
1269 + if (flowName === 'tools' && Object.keys(availableTools).length > 0) {
1270 + for (const [toolName, tool] of Object.entries(availableTools)) {
1271 + allMethods.push({
1272 + name: toolName,
1273 + title: tool.inputSchema?.title || tool.title || '',
1274 + type: 'request',
1275 + description: tool.description || 'No description available',
1276 + dynamic: true,
1277 + toolName: toolName,
1278 + schema: tool,
1279 + template: {
1280 + jsonrpc: "2.0",
1281 + id: 1,
1282 + method: "tools/call",
1283 + params: {
1284 + name: toolName,
1285 + arguments: {}
1286 + }
1287 + }
1288 + });
1289 + }
1290 + }
1291 +
1292 + // Add history for custom flow
1293 + if (flowName === 'custom') {
1294 + const savedHistory = loadRequestHistory();
1295 + savedHistory.forEach((item, index) => {
1296 + let displayName = `${item.displayTime} - ${item.method}`;
1297 + if (item.toolName) {
1298 + displayName += ` (${item.toolName})`;
1299 + }
1300 +
1301 + // For history items that are tool calls, try to find the schema
1302 + let toolSchema = null;
1303 + if (item.toolName && availableTools[item.toolName]) {
1304 + toolSchema = availableTools[item.toolName];
1305 + }
1306 +
1307 + allMethods.push({
1308 + name: displayName,
1309 + type: 'history',
1310 + description: `Historical request #${index + 1} - ${item.toolName ? `Tool: ${item.toolName}` : item.method}`,
1311 + template: JSON.parse(JSON.stringify(item.request)), // Clone the request to avoid editing original
1312 + error: item.error,
1313 + toolName: item.toolName,
1314 + savedItem: item,
1315 + dynamic: !!toolSchema, // Mark as dynamic if we have a schema
1316 + schema: toolSchema // Include the schema for form generation
1317 + });
1318 + });
1319 + }
1320 +
1321 + // Update method count
1322 + document.getElementById('methodCount').textContent = allMethods.length;
1323 +
1324 + // Create method items
1325 + allMethods.forEach(method => {
1326 + const methodDiv = document.createElement('div');
1327 + methodDiv.className = `method-item tooltip ${method.dynamic ? 'dynamic' : ''} ${method.type === 'history' ? 'history' : ''} ${method.error ? 'error' : ''}`;
1328 +
1329 + if (method.dynamic && method.toolName) {
1330 + // Special layout for tools
1331 + const toolName = document.createElement('div');
1332 + toolName.className = 'tool-name';
1333 + toolName.textContent = method.toolName;
1334 + methodDiv.appendChild(toolName);
1335 +
1336 + if (method.title) {
1337 + const toolTitle = document.createElement('div');
1338 + toolTitle.className = 'tool-title';
1339 + toolTitle.textContent = method.title;
1340 + methodDiv.appendChild(toolTitle);
1341 + }
1342 +
1343 + const toolMethod = document.createElement('div');
1344 + toolMethod.className = 'tool-method';
1345 + toolMethod.textContent = 'tools/call';
1346 + methodDiv.appendChild(toolMethod);
1347 + } else {
1348 + // Regular layout for other methods
1349 + const methodHeader = document.createElement('div');
1350 + methodHeader.className = 'method-header';
1351 +
1352 + const methodText = document.createElement('span');
1353 + methodText.className = 'method-text';
1354 + methodText.textContent = method.name;
1355 + methodHeader.appendChild(methodText);
1356 +
1357 + if (method.type === 'history') {
1358 + const count = document.createElement('span');
1359 + count.className = 'history-count';
1360 + count.textContent = `#${requestHistory.length - requestHistory.indexOf(requestHistory.find(h => h.request === method.template))}`;
1361 + methodHeader.appendChild(count);
1362 + }
1363 +
1364 + methodDiv.appendChild(methodHeader);
1365 + }
1366 +
1367 + // Add tooltip
1368 + const tooltipText = document.createElement('span');
1369 + tooltipText.className = 'tooltiptext';
1370 + tooltipText.textContent = method.description;
1371 + methodDiv.appendChild(tooltipText);
1372 +
1373 + // Position tooltip on hover
1374 + methodDiv.addEventListener('mouseenter', (e) => {
1375 + const rect = methodDiv.getBoundingClientRect();
1376 + tooltipText.style.left = `${rect.left + rect.width + 10}px`;
1377 + tooltipText.style.top = `${rect.top}px`;
1378 + tooltipText.style.bottom = 'auto';
1379 + tooltipText.style.transform = 'none';
1380 + });
1381 +
1382 + methodDiv.addEventListener('click', () => selectMethod(method));
1383 + methodsList.appendChild(methodDiv);
1384 + });
1385 + }
1386 +
1387 + // Select a method
1388 + function selectMethod(method) {
1389 + // Update active state
1390 + document.querySelectorAll('.method-item').forEach(item => item.classList.remove('active'));
1391 + event.target.closest('.method-item').classList.add('active');
1392 +
1393 + // Update current tool name display
1394 + const currentToolNameElement = document.getElementById('currentToolName');
1395 + if (method.toolName) {
1396 + currentToolNameElement.textContent = `: ${method.toolName}`;
1397 + } else if (method.name) {
1398 + currentToolNameElement.textContent = `: ${method.name}`;
1399 + } else {
1400 + currentToolNameElement.textContent = '';
1401 + }
1402 +
1403 + // Set the template in the editor (deep clone to avoid modifying original)
1404 + const template = JSON.parse(JSON.stringify(method.template));
1405 + // Always use the current next ID
1406 + if (template.id !== undefined) {
1407 + template.id = currentRequestId;
1408 + }
1409 + isUpdatingFromCode = true;
1410 + jsonEditor.value = JSON.stringify(template, null, 2);
1411 + setTimeout(() => { isUpdatingFromCode = false; }, 10);
1412 +
1413 + // Update schema form if we have a schema for this method
1414 + if (method.dynamic && method.schema) {
1415 + currentMethodSchema = method;
1416 + updateSchemaForm(method.schema);
1417 + } else {
1418 + currentMethodSchema = null;
1419 + updateSchemaForm(null);
1420 + // Update message for methods without parameters
1421 + if (method.name && !method.dynamic) {
1422 + noSchemaMessage.innerHTML = 'This method has no parameters.<br><br>Click "Send" to execute it.';
1423 + }
1424 + }
1425 + }
1426 +
1427 + // Update schema form
1428 + function updateSchemaForm(schema) {
1429 + // Update schema viewer
1430 + const noSchemaAvailable = document.getElementById('noSchemaAvailable');
1431 + const schemaDisplay = document.getElementById('schemaDisplay');
1432 +
1433 + if (!schema || !schema.inputSchema) {
1434 + noSchemaMessage.style.display = 'block';
1435 + schemaFormEditor.style.display = 'none';
1436 + noSchemaAvailable.style.display = 'block';
1437 + schemaDisplay.style.display = 'none';
1438 + return;
1439 + }
1440 +
1441 + noSchemaMessage.style.display = 'none';
1442 + schemaFormEditor.style.display = 'block';
1443 +
1444 + // Update schema viewer with pretty-printed schema
1445 + noSchemaAvailable.style.display = 'none';
1446 + schemaDisplay.style.display = 'block';
1447 + schemaDisplay.innerHTML = jsonPrinter.toHTMLContent(schema.inputSchema);
1448 +
1449 + // Clear and create new form
1450 + schemaFormEditor.innerHTML = '';
1451 +
1452 + try {
1453 + // Create form container
1454 + const formContainer = document.createElement('div');
1455 + schemaFormEditor.appendChild(formContainer);
1456 +
1457 + // Initialize generator with the container
1458 + schemaFormGenerator = new MCPSchemaUIGenerator(formContainer, {
1459 + onChange: (value) => {
1460 + // Skip if this change was triggered by code (not user)
1461 + if (isUpdatingFromCode) return;
1462 +
1463 + // Clear any validation messages when user edits
1464 + const existingMsg = schemaFormEditor.querySelector('.validation-message');
1465 + if (existingMsg && !existingMsg.classList.contains('success')) {
1466 + existingMsg.remove();
1467 + }
1468 +
1469 + // Save parameters to local storage for tools
1470 + if (currentMethodSchema && currentMethodSchema.toolName) {
1471 + saveToolParams(currentMethodSchema.toolName, value);
1472 + showSaveIndicator();
1473 + }
1474 +
1475 + if (currentActiveTab === 'editRequest') {
1476 + updateRawFromForm();
1477 + }
1478 + },
1479 + showDescriptions: false,
1480 + showTooltips: true
1481 + });
1482 +
1483 + // Get start value - prioritize JSON editor for history items, then saved parameters, then defaults
1484 + let startValue = {};
1485 +
1486 + // First try to get from JSON editor (this contains the historical data when loading history items)
1487 + try {
1488 + const currentRequest = JSON.parse(jsonEditor.value);
1489 + if (currentRequest.params && currentRequest.params.arguments) {
1490 + startValue = currentRequest.params.arguments;
1491 + }
1492 + } catch (e) {
1493 + // JSON parse failed, continue to saved parameters
1494 + }
1495 +
1496 + // If no JSON editor data, try to load saved parameters for this tool
1497 + if (Object.keys(startValue).length === 0 && currentMethodSchema && currentMethodSchema.toolName) {
1498 + const savedParams = loadToolParams(currentMethodSchema.toolName);
1499 + if (Object.keys(savedParams).length > 0) {
1500 + startValue = savedParams;
1501 + }
1502 + }
1503 +
1504 + // Set schema with validation
1505 + try {
1506 + schemaFormGenerator.setSchema(schema.inputSchema, startValue);
1507 +
1508 + // Update raw JSON if we loaded saved parameters
1509 + if (Object.keys(startValue).length > 0) {
1510 + updateRawFromForm();
1511 + }
1512 + } catch (validationError) {
1513 + // Show validation error
1514 + const errorDiv = document.createElement('div');
1515 + errorDiv.className = 'validation-message';
1516 + errorDiv.innerHTML = `<strong>Schema Validation Error:</strong><br><pre>${validationError.message}</pre>`;
1517 + schemaFormEditor.appendChild(errorDiv);
1518 + schemaFormGenerator = null;
1519 + }
1520 +
1521 + } catch (e) {
1522 + console.error('Error creating form:', e);
1523 + noSchemaMessage.style.display = 'block';
1524 + schemaFormEditor.style.display = 'none';
1525 + }
1526 + }
1527 +
1528 + // Update raw JSON from form
1529 + function updateRawFromForm() {
1530 + if (!schemaFormGenerator || !currentMethodSchema) return;
1531 +
1532 + try {
1533 + const formData = schemaFormGenerator.getValue();
1534 + const currentRequest = JSON.parse(jsonEditor.value);
1535 +
1536 + if (currentMethodSchema.toolName) {
1537 + // For tools/call, update the arguments
1538 + currentRequest.params.arguments = formData;
1539 + } else {
1540 + // For other methods, update params directly
1541 + currentRequest.params = formData;
1542 + }
1543 +
1544 + // Set flag to prevent recursive updates
1545 + isUpdatingFromCode = true;
1546 + jsonEditor.value = JSON.stringify(currentRequest, null, 2);
1547 + // Reset flag after a small delay to ensure the event has been processed
1548 + setTimeout(() => { isUpdatingFromCode = false; }, 10);
1549 + } catch (e) {
1550 + console.error('Error updating raw JSON:', e);
1551 + }
1552 + }
1553 +
1554 + // Update form from raw JSON
1555 + function updateFormFromRaw() {
1556 + if (!schemaFormGenerator || !currentMethodSchema) return;
1557 +
1558 + try {
1559 + const currentRequest = JSON.parse(jsonEditor.value);
1560 + let value = {};
1561 +
1562 + if (currentRequest.params) {
1563 + if (currentMethodSchema.toolName && currentRequest.params.arguments) {
1564 + value = currentRequest.params.arguments;
1565 + } else {
1566 + value = currentRequest.params;
1567 + }
1568 + }
1569 +
1570 + // Set flag to prevent onChange from updating JSON back
1571 + isUpdatingFromCode = true;
1572 + schemaFormGenerator.setValue(value);
1573 + // Reset flag after a delay to ensure all events have processed
1574 + setTimeout(() => {
1575 + isUpdatingFromCode = false;
1576 +
1577 + // Save the updated parameters to local storage after form is updated
1578 + if (currentMethodSchema && currentMethodSchema.toolName && value) {
1579 + saveToolParams(currentMethodSchema.toolName, value);
1580 + }
1581 + }, 50);
1582 + } catch (e) {
1583 + console.error('Error updating form from JSON:', e);
1584 + }
1585 + }
1586 +
1587 + // Validate form
1588 + function validateForm() {
1589 + if (!schemaFormGenerator) {
1590 + alert('No form to validate');
1591 + return;
1592 + }
1593 +
1594 + // Clear any previous errors first
1595 + schemaFormGenerator.clearErrors();
1596 +
1597 + const isValid = schemaFormGenerator.validate();
1598 +
1599 + // Show validation result
1600 + const existingMsg = schemaFormEditor.querySelector('.validation-message');
1601 + if (existingMsg) existingMsg.remove();
1602 +
1603 + const msgDiv = document.createElement('div');
1604 + msgDiv.className = `validation-message ${isValid ? 'success' : ''}`;
1605 +
1606 + if (isValid) {
1607 + msgDiv.textContent = '✓ Form is valid!';
1608 + } else {
1609 + const errors = Object.entries(schemaFormGenerator.errors).map(([field, error]) => `${field}: ${error}`);
1610 + msgDiv.innerHTML = '<strong>Validation errors:</strong><br>' + errors.join('<br>');
1611 + }
1612 +
1613 + schemaFormEditor.insertBefore(msgDiv, schemaFormEditor.firstChild);
1614 +
1615 + // Auto-remove success message after 3 seconds
1616 + if (isValid) {
1617 + setTimeout(() => msgDiv.remove(), 3000);
1618 + }
1619 + }
1620 +
1621 + // Send from form (without validation)
1622 + function sendFromForm() {
1623 + if (!schemaFormGenerator) {
1624 + // No form, but we can still send the request if there's no schema required
1625 + sendRequest();
1626 + return;
1627 + }
1628 +
1629 + // Update raw JSON and send WITHOUT validation
1630 + updateRawFromForm();
1631 + sendRequest();
1632 + }
1633 +
1634 + // Validate and send from form
1635 + function validateAndSendFromForm() {
1636 + if (!schemaFormGenerator) {
1637 + // No form, but we can still send the request if there's no schema required
1638 + sendRequest();
1639 + return;
1640 + }
1641 +
1642 + // Validate first
1643 + if (!schemaFormGenerator.validate()) {
1644 + validateForm();
1645 + return;
1646 + }
1647 +
1648 + // Update raw JSON and send
1649 + updateRawFromForm();
1650 + sendRequest();
1651 + }
1652 +
1653 + // Connect and Handshake automation
1654 + async function connectAndInitialize() {
1655 + const url = serverUrlInput.value;
1656 + if (!url) {
1657 + alert('Please enter a WebSocket URL');
1658 + return;
1659 + }
1660 +
1661 + try {
1662 + log('Starting automated connection and handshake...');
1663 +
1664 + // Step 1: Connect
1665 + await new Promise((resolve, reject) => {
1666 + log('1. Connecting to ' + url + '...');
1667 +
1668 + // Clear cached data from previous connection
1669 + availableTools = {};
1670 + availablePrompts = {};
1671 + availableResources = {};
1672 +
1673 + ws = new WebSocket(url);
1674 +
1675 + ws.onopen = () => {
1676 + log('✓ Connected to ' + url);
1677 + updateStatus(true);
1678 + saveServerUrl(url);
1679 + resolve();
1680 + };
1681 +
1682 + ws.onerror = (error) => {
1683 + log('✗ WebSocket connection error: ' + error);
1684 + reject(new Error('Connection failed'));
1685 + };
1686 +
1687 + ws.onclose = () => {
1688 + if (ws.readyState !== WebSocket.OPEN) {
1689 + log('✗ Connection closed before opening');
1690 + reject(new Error('Connection closed'));
1691 + }
1692 + };
1693 +
1694 + // Set up message handler for responses
1695 + ws.onmessage = (event) => {
1696 + try {
1697 + const data = JSON.parse(event.data);
1698 +
1699 + // Calculate metrics
1700 + let responseTime = null;
1701 + if (data.id !== undefined && pendingRequests.has(data.id)) {
1702 + responseTime = Date.now() - pendingRequests.get(data.id);
1703 + pendingRequests.delete(data.id);
1704 + }
1705 +
1706 + const responseSize = new Blob([event.data]).size;
1707 + const estimatedTokens = estimateTokens(event.data);
1708 +
1709 + logMessage(data, 'received', {
1710 + responseTime,
1711 + responseSize,
1712 + estimatedTokens
1713 + });
1714 +
1715 + // Handle specific responses
1716 + if (data.id && data.result) {
1717 + handleResponse(data);
1718 + }
1719 + } catch (e) {
1720 + log('← Received (raw): ' + event.data);
1721 + }
1722 + };
1723 + });
1724 +
1725 + // Step 2: Send initialize
1726 + await new Promise((resolve, reject) => {
1727 + log('2. Sending initialize request...');
1728 +
1729 + const initRequest = {
1730 + jsonrpc: "2.0",
1731 + id: currentRequestId++,
1732 + method: "initialize",
1733 + params: {
1734 + protocolVersion: "2024-11-05",
1735 + capabilities: {
1736 + roots: { listChanged: true },
1737 + sampling: {}
1738 + },
1739 + clientInfo: {
1740 + name: "Netdata MCP Test Client",
1741 + version: "1.0.0"
1742 + }
1743 + }
1744 + };
1745 +
1746 + // Track this request
1747 + pendingRequests.set(initRequest.id, Date.now());
1748 +
1749 + // Set up one-time response handler
1750 + const originalHandler = ws.onmessage;
1751 + let responseReceived = false;
1752 +
1753 + const timeout = setTimeout(() => {
1754 + if (!responseReceived) {
1755 + ws.onmessage = originalHandler;
1756 + reject(new Error('Initialize request timed out'));
1757 + }
1758 + }, 10000); // 10 second timeout
1759 +
1760 + ws.onmessage = (event) => {
1761 + // Call original handler first
1762 + originalHandler(event);
1763 +
1764 + if (responseReceived) return;
1765 +
1766 + try {
1767 + const data = JSON.parse(event.data);
1768 + if (data.id === initRequest.id) {
1769 + responseReceived = true;
1770 + clearTimeout(timeout);
1771 + ws.onmessage = originalHandler;
1772 +
1773 + if (data.error) {
1774 + log('✗ Initialize failed: ' + JSON.stringify(data.error));
1775 + reject(new Error('Initialize failed: ' + data.error.message));
1776 + } else {
1777 + log('✓ Initialize successful');
1778 + resolve();
1779 + }
1780 + }
1781 + } catch (e) {
1782 + // Ignore parsing errors for this handler
1783 + }
1784 + };
1785 +
1786 + ws.send(JSON.stringify(initRequest));
1787 + logMessage(initRequest, 'sent');
1788 + });
1789 +
1790 + // Step 3: Send initialized notification
1791 + log('3. Sending initialized notification...');
1792 + const initializedNotification = {
1793 + jsonrpc: "2.0",
1794 + method: "notifications/initialized"
1795 + };
1796 +
1797 + ws.send(JSON.stringify(initializedNotification));
1798 + logMessage(initializedNotification, 'sent');
1799 + log('✓ Initialized notification sent');
1800 +
1801 + // Small delay to ensure notification is processed
1802 + await new Promise(resolve => setTimeout(resolve, 100));
1803 +
1804 + // Step 4: Send tools/list
1805 + await new Promise((resolve, reject) => {
1806 + log('4. Requesting tools list...');
1807 +
1808 + const toolsRequest = {
1809 + jsonrpc: "2.0",
1810 + id: currentRequestId++,
1811 + method: "tools/list"
1812 + };
1813 +
1814 + // Track this request
1815 + pendingRequests.set(toolsRequest.id, Date.now());
1816 +
1817 + // Set up one-time response handler
1818 + const originalHandler = ws.onmessage;
1819 + let responseReceived = false;
1820 +
1821 + const timeout = setTimeout(() => {
1822 + if (!responseReceived) {
1823 + ws.onmessage = originalHandler;
1824 + reject(new Error('Tools list request timed out'));
1825 + }
1826 + }, 10000); // 10 second timeout
1827 +
1828 + ws.onmessage = (event) => {
1829 + // Call original handler first
1830 + originalHandler(event);
1831 +
1832 + if (responseReceived) return;
1833 +
1834 + try {
1835 + const data = JSON.parse(event.data);
1836 + if (data.id === toolsRequest.id) {
1837 + responseReceived = true;
1838 + clearTimeout(timeout);
1839 + ws.onmessage = originalHandler;
1840 +
1841 + if (data.error) {
1842 + log('✗ Tools list failed: ' + JSON.stringify(data.error));
1843 + reject(new Error('Tools list failed: ' + data.error.message));
1844 + } else {
1845 + log('✓ Tools list received (' + (data.result.tools ? data.result.tools.length : 0) + ' tools)');
1846 + resolve();
1847 + }
1848 + }
1849 + } catch (e) {
1850 + // Ignore parsing errors for this handler
1851 + }
1852 + };
1853 +
1854 + ws.send(JSON.stringify(toolsRequest));
1855 + logMessage(toolsRequest, 'sent');
1856 + });
1857 +
1858 + // Step 5: Set up proper persistent event handlers
1859 + log('5. Setting up persistent connection handlers...');
1860 + ws.onerror = (error) => {
1861 + log('WebSocket error: ' + error);
1862 + };
1863 +
1864 + ws.onclose = () => {
1865 + log('Disconnected');
1866 + updateStatus(false);
1867 + ws = null;
1868 + };
1869 + log('✓ Persistent handlers configured');
1870 +
1871 + // Step 6: Switch to Tools flow
1872 + log('6. Switching to Tools flow...');
1873 + document.querySelectorAll('.flow-item').forEach(item => item.classList.remove('active'));
1874 + const toolsFlow = document.querySelector('[data-flow="tools"]');
1875 + if (toolsFlow) {
1876 + toolsFlow.classList.add('active');
1877 + displayMethods('tools');
1878 + log('✓ Switched to Tools flow');
1879 + } else {
1880 + log('⚠ Tools flow not found');
1881 + }
1882 +
1883 + log('🎉 Automated connection and handshake completed successfully!');
1884 +
1885 + } catch (error) {
1886 + log('❌ Automated connection and handshake failed: ' + error.message);
1887 + if (ws) {
1888 + ws.close();
1889 + ws = null;
1890 + updateStatus(false);
1891 + }
1892 + }
1893 + }
1894 +
1895 + // WebSocket functions
1896 + function connect() {
1897 + const url = serverUrlInput.value;
1898 + if (!url) {
1899 + alert('Please enter a WebSocket URL');
1900 + return;
1901 + }
1902 +
1903 + log('Connecting to ' + url + '...');
1904 +
1905 + // Clear cached data from previous connection
1906 + availableTools = {};
1907 + availablePrompts = {};
1908 + availableResources = {};
1909 +
1910 + ws = new WebSocket(url);
1911 +
1912 + ws.onopen = () => {
1913 + log('Connected to ' + url);
1914 + updateStatus(true);
1915 + // Save successful connection URL
1916 + saveServerUrl(url);
1917 + };
1918 +
1919 + ws.onmessage = (event) => {
1920 + try {
1921 + const data = JSON.parse(event.data);
1922 +
1923 + // Calculate metrics
1924 + let responseTime = null;
1925 + if (data.id !== undefined && pendingRequests.has(data.id)) {
1926 + responseTime = Date.now() - pendingRequests.get(data.id);
1927 + pendingRequests.delete(data.id);
1928 + }
1929 +
1930 + const responseSize = new Blob([event.data]).size;
1931 + const estimatedTokens = estimateTokens(event.data);
1932 +
1933 + logMessage(data, 'received', {
1934 + responseTime,
1935 + responseSize,
1936 + estimatedTokens
1937 + });
1938 +
1939 + // Handle specific responses
1940 + if (data.id && data.result) {
1941 + handleResponse(data);
1942 + }
1943 + } catch (e) {
1944 + log('← Received (raw): ' + event.data);
1945 + }
1946 + };
1947 +
1948 + ws.onerror = (error) => {
1949 + log('WebSocket error: ' + error);
1950 + };
1951 +
1952 + ws.onclose = () => {
1953 + log('Disconnected');
1954 + updateStatus(false);
1955 + ws = null;
1956 + };
1957 + }
1958 +
1959 + function disconnect() {
1960 + if (ws) {
1961 + ws.close();
1962 + }
1963 + }
1964 +
1965 + function sendRequest() {
1966 + if (!ws || ws.readyState !== WebSocket.OPEN) {
1967 + alert('Not connected to server');
1968 + return;
1969 + }
1970 +
1971 + try {
1972 + const request = JSON.parse(jsonEditor.value);
1973 +
1974 + // Track request timestamp if it has an ID
1975 + if (request.id !== undefined) {
1976 + pendingRequests.set(request.id, Date.now());
1977 + }
1978 +
1979 + // Determine tool name if it's a tools/call request
1980 + let toolName = null;
1981 + if (request.method === 'tools/call' && request.params && request.params.name) {
1982 + toolName = request.params.name;
1983 + }
1984 +
1985 + // Save to history
1986 + const historyItem = saveRequestToHistory(request, request.method, toolName, false);
1987 +
1988 + // Update in-memory history for immediate UI updates
1989 + const oldHistoryItem = {
1990 + timestamp: new Date().toLocaleTimeString(),
1991 + method: request.method,
1992 + request: JSON.parse(JSON.stringify(request)), // Deep clone here too
1993 + error: false
1994 + };
1995 + requestHistory.unshift(oldHistoryItem);
1996 + if (requestHistory.length > 50) {
1997 + requestHistory = requestHistory.slice(0, 50);
1998 + }
1999 +
2000 + // Update history flow indicator
2001 + updateHistoryFlowIndicator();
2002 +
2003 + ws.send(JSON.stringify(request));
2004 + logMessage(request, 'sent');
2005 +
2006 + // Increment ID after sending for next request
2007 + if (request.id !== undefined && request.id === currentRequestId) {
2008 + currentRequestId++;
2009 + }
2010 + } catch (e) {
2011 + alert('Invalid JSON: ' + e.message);
2012 +
2013 + // Save error to history
2014 + saveRequestToHistory({ error: e.message }, 'Invalid JSON', null, true);
2015 +
2016 + // Add error to in-memory history
2017 + const historyItem = {
2018 + timestamp: new Date().toLocaleTimeString(),
2019 + method: 'Invalid JSON',
2020 + request: { error: e.message },
2021 + error: true
2022 + };
2023 + requestHistory.unshift(historyItem);
2024 +
2025 + // Update history flow indicator
2026 + updateHistoryFlowIndicator();
2027 + }
2028 + }
2029 +
2030 + function handleResponse(response) {
2031 + // Handle initialize response
2032 + if (response.result && response.result.capabilities) {
2033 + serverCapabilities = response.result.capabilities;
2034 + log('Server capabilities: ' + JSON.stringify(serverCapabilities, null, 2));
2035 + }
2036 +
2037 + // Handle tools/list response
2038 + if (response.result && response.result.tools) {
2039 + availableTools = {};
2040 + response.result.tools.forEach(tool => {
2041 + availableTools[tool.name] = tool;
2042 + });
2043 + log(`Loaded ${Object.keys(availableTools).length} tools`);
2044 + // Refresh tools methods if currently viewing
2045 + const activeFlow = document.querySelector('.flow-item.active');
2046 + if (activeFlow && activeFlow.dataset.flow === 'tools') {
2047 + displayMethods('tools');
2048 + }
2049 + }
2050 +
2051 + // Handle prompts/list response
2052 + if (response.result && response.result.prompts) {
2053 + availablePrompts = {};
2054 + response.result.prompts.forEach(prompt => {
2055 + availablePrompts[prompt.name] = prompt;
2056 + });
2057 + log(`Loaded ${Object.keys(availablePrompts).length} prompts`);
2058 + }
2059 +
2060 + // Handle resources/list response
2061 + if (response.result && response.result.resources) {
2062 + availableResources = {};
2063 + response.result.resources.forEach(resource => {
2064 + availableResources[resource.uri] = resource;
2065 + });
2066 + log(`Loaded ${Object.keys(availableResources).length} resources`);
2067 + }
2068 + }
2069 +
2070 + function updateStatus(connected) {
2071 + statusElement.textContent = connected ? 'Connected' : 'Disconnected';
2072 + statusElement.className = 'status ' + (connected ? 'connected' : 'disconnected');
2073 + connectBtn.disabled = connected;
2074 + connectAndInitBtn.disabled = connected;
2075 + disconnectBtn.disabled = !connected;
2076 + sendBtn.disabled = !connected;
2077 + sendFromFormBtn.disabled = !connected;
2078 + validateFormBtn.disabled = !connected;
2079 + validateAndSendBtn.disabled = !connected;
2080 + }
2081 +
2082 + function formatJSON() {
2083 + try {
2084 + const json = JSON.parse(jsonEditor.value);
2085 + isUpdatingFromCode = true;
2086 + jsonEditor.value = JSON.stringify(json, null, 2);
2087 + setTimeout(() => { isUpdatingFromCode = false; }, 10);
2088 + } catch (e) {
2089 + alert('Invalid JSON: ' + e.message);
2090 + }
2091 + }
2092 +
2093 + function copyJSON() {
2094 + jsonEditor.select();
2095 + document.execCommand('copy');
2096 +
2097 + // Visual feedback
2098 + const originalText = copyBtn.textContent;
2099 + copyBtn.textContent = 'Copied!';
2100 + setTimeout(() => {
2101 + copyBtn.textContent = originalText;
2102 + }, 1000);
2103 + }
2104 +
2105 + function clearLog() {
2106 + responseViewer.innerHTML = '';
2107 + responseEntries = [];
2108 + currentResponseIndex = -1;
2109 + updateNavButtons();
2110 + }
2111 +
2112 + // Navigate to previous response
2113 + function navigateToPrevResponse() {
2114 + if (currentResponseIndex > 0) {
2115 + currentResponseIndex--;
2116 + responseEntries[currentResponseIndex].scrollIntoView({ behavior: 'auto', block: 'start' });
2117 + updateNavButtons();
2118 + }
2119 + }
2120 +
2121 + // Navigate to next response
2122 + function navigateToNextResponse() {
2123 + if (currentResponseIndex < responseEntries.length - 1) {
2124 + currentResponseIndex++;
2125 + responseEntries[currentResponseIndex].scrollIntoView({ behavior: 'auto', block: 'start' });
2126 + updateNavButtons();
2127 + }
2128 + }
2129 +
2130 + // Update navigation button states
2131 + function updateNavButtons() {
2132 + prevResponseBtn.disabled = currentResponseIndex <= 0 || responseEntries.length === 0;
2133 + nextResponseBtn.disabled = currentResponseIndex >= responseEntries.length - 1 || responseEntries.length === 0;
2134 +
2135 + // Add keyboard shortcuts hint in title
2136 + if (responseEntries.length > 0) {
2137 + prevResponseBtn.title = `Previous response (${currentResponseIndex}/${responseEntries.length}) [↑]`;
2138 + nextResponseBtn.title = `Next response (${currentResponseIndex + 1}/${responseEntries.length}) [↓]`;
2139 + }
2140 + }
2141 +
2142 + function log(message) {
2143 + const timestamp = new Date().toLocaleTimeString();
2144 + const entry = document.createElement('div');
2145 + entry.className = 'json-entry';
2146 + entry.innerHTML = `<div class="json-timestamp">[${timestamp}]</div><div>${message}</div>`;
2147 + responseViewer.appendChild(entry);
2148 + responseViewer.scrollTop = responseViewer.scrollHeight;
2149 + }
2150 +
2151 + function logMessage(obj, direction, metrics = {}) {
2152 + const timestamp = new Date().toLocaleTimeString();
2153 + const entry = document.createElement('div');
2154 + entry.className = 'json-entry';
2155 +
2156 + const directionClass = direction === 'sent' ? 'json-sent' : 'json-received';
2157 + const arrow = direction === 'sent' ? '→' : '←';
2158 +
2159 + // Build metrics string
2160 + let metricsHtml = '';
2161 + if (direction === 'received' && (metrics.responseTime || metrics.responseSize || metrics.estimatedTokens)) {
2162 + const parts = [];
2163 + if (metrics.responseTime !== null) {
2164 + parts.push(`${metrics.responseTime}ms`);
2165 + }
2166 + if (metrics.responseSize) {
2167 + parts.push(`${formatBytes(metrics.responseSize)}`);
2168 + }
2169 + if (metrics.estimatedTokens) {
2170 + parts.push(`~${metrics.estimatedTokens} tokens`);
2171 + }
2172 + if (parts.length > 0) {
2173 + metricsHtml = ` <span style="color: #666; font-size: 0.85em;">(${parts.join(', ')})</span>`;
2174 + }
2175 + }
2176 +
2177 + // Add entry-level Raw and Copy buttons for responses
2178 + let entryButtons = '';
2179 + if (direction === 'received') {
2180 + const entryId = 'entry_' + Date.now() + '_' + Math.random().toString(36).substr(2, 9);
2181 + entryButtons = `<span class="entry-controls">
2182 + <button class="entry-btn" onclick="showRawEntry('${entryId}')" title="Show raw JSON">Raw</button>
2183 + <button class="entry-btn" onclick="copyEntryJSON('${entryId}')" title="Copy JSON">Copy</button>
2184 + </span>`;
2185 +
2186 + // Store the JSON data for later retrieval
2187 + entry.dataset.entryId = entryId;
2188 + entry.dataset.rawJson = JSON.stringify(obj);
2189 + }
2190 +
2191 + entry.innerHTML = `<div class="json-timestamp">[${timestamp}]</div><div class="json-header-line"><span class="json-direction ${directionClass}">${arrow} ${direction === 'sent' ? 'Sent' : 'Received'}:</span>${metricsHtml}${entryButtons}</div><div id="raw-container-${direction === 'received' ? entry.dataset.entryId : ''}" style="margin-top: 5px;"></div><div style="margin-top: 5px;">${jsonPrinter.toHTMLContent(obj)}</div>`;
2192 +
2193 + responseViewer.appendChild(entry);
2194 +
2195 + // Track response entries for navigation
2196 + if (direction === 'received') {
2197 + responseEntries.push(entry);
2198 + currentResponseIndex = responseEntries.length - 1;
2199 + updateNavButtons();
2200 +
2201 + // Small delay to ensure DOM is updated
2202 + setTimeout(() => {
2203 + entry.scrollIntoView({ behavior: 'auto', block: 'start' });
2204 + }, 5);
2205 + } else {
2206 + // For sent messages, continue scrolling to bottom
2207 + responseViewer.scrollTop = responseViewer.scrollHeight;
2208 + }
2209 + }
2210 +
2211 + // Estimate tokens (rough approximation)
2212 + function estimateTokens(text) {
2213 + // Simple estimation: ~4 characters per token on average
2214 + // This is a rough approximation - actual tokenization varies by model
2215 + return Math.ceil(text.length / 4);
2216 + }
2217 +
2218 + // Format bytes to human readable
2219 + function formatBytes(bytes) {
2220 + if (bytes < 1024) return bytes + ' B';
2221 + if (bytes < 1024 * 1024) return (bytes / 1024).toFixed(1) + ' KB';
2222 + return (bytes / (1024 * 1024)).toFixed(1) + ' MB';
2223 + }
2224 +
2225 + // Functions for entry-level Raw and Copy
2226 + function showRawEntry(entryId) {
2227 + const entry = document.querySelector(`[data-entry-id="${entryId}"]`);
2228 + const btn = document.querySelector(`[onclick="showRawEntry('${entryId}')"]`);
2229 + const rawContainer = document.getElementById(`raw-container-${entryId}`);
2230 +
2231 + if (entry && rawContainer) {
2232 + // Check if raw view already exists
2233 + let rawView = rawContainer.querySelector('.raw-json-view');
2234 +
2235 + if (rawView) {
2236 + // Toggle existing raw view
2237 + if (rawView.style.display === 'none') {
2238 + rawView.style.display = 'block';
2239 + btn.textContent = 'Hide Raw';
2240 + btn.style.background = '#28a745';
2241 + } else {
2242 + rawView.style.display = 'none';
2243 + btn.textContent = 'Raw';
2244 + btn.style.background = '';
2245 + }
2246 + } else {
2247 + // Create new raw view at the top
2248 + rawView = document.createElement('div');
2249 + rawView.className = 'raw-json-view';
2250 + rawView.style.cssText = `
2251 + margin-bottom: 10px;
2252 + padding: 10px;
2253 + background: #f8f8f8;
2254 + border: 1px solid #ddd;
2255 + border-radius: 4px;
2256 + font-family: 'Courier New', monospace;
2257 + font-size: 12px;
2258 + white-space: pre-wrap;
2259 + overflow-x: auto;
2260 + max-height: 400px;
2261 + overflow-y: auto;
2262 + `;
2263 +
2264 + const rawJson = entry.dataset.rawJson;
2265 + const formattedJson = JSON.stringify(JSON.parse(rawJson), null, 2);
2266 + rawView.textContent = formattedJson;
2267 +
2268 + rawContainer.appendChild(rawView);
2269 + btn.textContent = 'Hide Raw';
2270 + btn.style.background = '#28a745';
2271 + }
2272 + }
2273 + }
2274 +
2275 + function copyEntryJSON(entryId) {
2276 + const entry = document.querySelector(`[data-entry-id="${entryId}"]`);
2277 + if (entry) {
2278 + const rawJson = entry.dataset.rawJson;
2279 + navigator.clipboard.writeText(JSON.stringify(JSON.parse(rawJson), null, 2)).then(() => {
2280 + // Visual feedback
2281 + const btn = document.querySelector(`[onclick="copyEntryJSON('${entryId}')"]`);
2282 + const originalText = btn.textContent;
2283 + btn.textContent = 'Copied!';
2284 + btn.style.background = '#28a745';
2285 + setTimeout(() => {
2286 + btn.textContent = originalText;
2287 + btn.style.background = '';
2288 + }, 1000);
2289 + });
2290 + }
2291 + }
2292 +
2293 + // Set initial JSON
2294 + isUpdatingFromCode = true;
2295 + jsonEditor.value = JSON.stringify(flows.initialization.methods[0].template, null, 2);
2296 + setTimeout(() => { isUpdatingFromCode = false; }, 10);
2297 +
2298 + // Import from LLM functions
2299 + function openImportModal() {
2300 + document.getElementById('importLLMModal').style.display = 'block';
2301 + document.getElementById('llmImportTextarea').value = '';
2302 + document.getElementById('llmImportTextarea').focus();
2303 + }
2304 +
2305 + function closeImportModal() {
2306 + document.getElementById('importLLMModal').style.display = 'none';
2307 + }
2308 +
2309 + function importFromLLM() {
2310 + const textarea = document.getElementById('llmImportTextarea');
2311 + const input = textarea.value.trim();
2312 +
2313 + if (!input) {
2314 + alert('Please paste a request to import');
2315 + return;
2316 + }
2317 +
2318 + try {
2319 + // Step 1: Replace backticks with quotes
2320 + let converted = input.replace(/`/g, '"');
2321 +
2322 + // Step 2: Try to parse as JSON to validate it
2323 + let parsedParams;
2324 + try {
2325 + parsedParams = JSON.parse(converted);
2326 + } catch (e) {
2327 + // If direct parsing fails, try to extract just the parameters object
2328 + // This handles cases where the user might paste including method name or other context
2329 + const match = converted.match(/\{[\s\S]*\}/);
2330 + if (match) {
2331 + parsedParams = JSON.parse(match[0]);
2332 + } else {
2333 + throw new Error('Could not find valid JSON object in the input');
2334 + }
2335 + }
2336 +
2337 + // Step 3: Determine if this is a tools/call request based on current context
2338 + let jsonRpcRequest;
2339 +
2340 + // Check if we have a tool selected
2341 + if (currentMethodSchema && currentMethodSchema.toolName) {
2342 + // It's a tool call
2343 + jsonRpcRequest = {
2344 + jsonrpc: "2.0",
2345 + id: currentRequestId,
2346 + method: "tools/call",
2347 + params: {
2348 + name: currentMethodSchema.toolName,
2349 + arguments: parsedParams
2350 + }
2351 + };
2352 + } else {
2353 + // Try to detect the method from the current selection or default to tools/call
2354 + const activeMethod = document.querySelector('.method-item.active');
2355 + let method = "tools/call";
2356 +
2357 + if (activeMethod) {
2358 + const methodText = activeMethod.querySelector('.method-text');
2359 + if (methodText) {
2360 + method = methodText.textContent;
2361 + }
2362 + }
2363 +
2364 + // For tools/call, wrap in arguments, otherwise use directly as params
2365 + if (method === "tools/call") {
2366 + // Try to guess the tool name from the parameters
2367 + // Common patterns: metrics, nodes, contexts suggest weights tools
2368 + let toolName = "unknown_tool";
2369 +
2370 + if (parsedParams.contexts || parsedParams.metrics) {
2371 + if (parsedParams.method === "ks2" || parsedParams.method === "volume") {
2372 + toolName = "find_correlated_metrics";
2373 + } else if (parsedParams.baseline_after !== undefined || parsedParams.baseline_before !== undefined) {
2374 + toolName = "find_correlated_metrics";
2375 + } else {
2376 + toolName = "list_metrics";
2377 + }
2378 + }
2379 +
2380 + jsonRpcRequest = {
2381 + jsonrpc: "2.0",
2382 + id: currentRequestId,
2383 + method: "tools/call",
2384 + params: {
2385 + name: toolName,
2386 + arguments: parsedParams
2387 + }
2388 + };
2389 +
2390 + alert(`Note: No tool was selected. Guessing tool name as "${toolName}". Please select the correct tool from the Methods list if this is incorrect.`);
2391 + } else {
2392 + jsonRpcRequest = {
2393 + jsonrpc: "2.0",
2394 + id: currentRequestId,
2395 + method: method,
2396 + params: parsedParams
2397 + };
2398 + }
2399 + }
2400 +
2401 + // Step 4: Update the JSON editor
2402 + isUpdatingFromCode = true;
2403 + jsonEditor.value = JSON.stringify(jsonRpcRequest, null, 2);
2404 + setTimeout(() => { isUpdatingFromCode = false; }, 10);
2405 +
2406 + // Step 5: Update the form if in edit mode
2407 + if (currentActiveTab === 'editRequest' && currentMethodSchema) {
2408 + updateFormFromRaw();
2409 + }
2410 +
2411 + // Step 6: Close modal
2412 + closeImportModal();
2413 +
2414 + // Step 7: Switch to Raw JSON tab to show the result
2415 + switchTab('rawRequest');
2416 +
2417 + // Visual feedback
2418 + formatJSON();
2419 +
2420 + } catch (error) {
2421 + alert('Error converting LLM request:\n\n' + error.message + '\n\nPlease ensure the input is a valid JSON-like structure with backticks.');
2422 + }
2423 + }
2424 +
2425 + // Close modal when clicking outside of it
2426 + window.onclick = function(event) {
2427 + const modal = document.getElementById('importLLMModal');
2428 + if (event.target === modal) {
2429 + closeImportModal();
2430 + }
2431 + }
2432 + </script>
2433 +</body>
2434 +</html>
\ No newline at end of file
src/web/mcp/mcp-test-client/json-pretty-printer.js new
+313
@@ -0,0 +1,313 @@
1 +/**
2 + * JSON Pretty Printer with Syntax Highlighting
3 + * Features:
4 + * - Visualizes newlines in strings
5 + * - Detects and formats nested JSON in string values
6 + * - Syntax highlighting with colors
7 + * - Handles large JSON structures efficiently
8 + */
9 +
10 +class JSONPrettyPrinter {
11 + constructor(options = {}) {
12 + this.options = {
13 + indent: 2,
14 + maxDepth: 100,
15 + maxStringLength: 1048576, // 1 MiB
16 + visualizeNewlines: true,
17 + detectNestedJSON: true,
18 + syntaxHighlight: true,
19 + colors: {
20 + key: '#881391',
21 + string: '#0B7500',
22 + number: '#1750EB',
23 + boolean: '#0033B3',
24 + null: '#666666',
25 + punctuation: '#999999',
26 + newline: '#0088cc',
27 + background: {
28 + newline: '#e6f3ff'
29 + }
30 + },
31 + ...options
32 + };
33 + }
34 +
35 + /**
36 + * Pretty print JSON with all features
37 + */
38 + prettyPrint(obj, indent = 0) {
39 + if (indent > this.options.maxDepth) {
40 + console.warn(`Max depth (${this.options.maxDepth}) reached at indent level ${indent}`);
41 + return this._colorize('"[Max depth reached]"', 'string');
42 + }
43 +
44 + if (obj === null) {
45 + return this._colorize('null', 'null');
46 + }
47 +
48 + const type = typeof obj;
49 +
50 + switch (type) {
51 + case 'boolean':
52 + return this._colorize(obj.toString(), 'boolean');
53 +
54 + case 'number':
55 + return this._colorize(obj.toString(), 'number');
56 +
57 + case 'string':
58 + return this._formatString(obj, indent);
59 +
60 + case 'object':
61 + if (Array.isArray(obj)) {
62 + return this._formatArray(obj, indent);
63 + } else {
64 + return this._formatObject(obj, indent);
65 + }
66 +
67 + default:
68 + return this._colorize(JSON.stringify(obj), 'string');
69 + }
70 + }
71 +
72 + /**
73 + * Format a string value with newline visualization and nested JSON detection
74 + */
75 + _formatString(str, indent) {
76 + // Check if string might contain JSON
77 + if (this.options.detectNestedJSON && this._looksLikeJSON(str)) {
78 + console.log(`Detected nested JSON at indent ${indent}, string length: ${str.length}`);
79 + try {
80 + // First, try to parse as-is
81 + const parsed = JSON.parse(str);
82 + console.log(`Successfully parsed nested JSON, formatting at indent ${indent + 1}`);
83 + // If it's valid JSON, format it as nested JSON
84 + const formatted = this.prettyPrint(parsed, indent + 1);
85 + // Wrap in quotes but show it's parsed JSON
86 + return this._colorize('"', 'string') +
87 + this._colorize('[JSON] ', 'newline') +
88 + formatted +
89 + this._colorize('"', 'string');
90 + } catch (e) {
91 + // If direct parsing fails, try unescaping first
92 + console.log('Direct parse failed, trying to unescape first');
93 + console.log('First 200 chars of string:', str.substring(0, 200));
94 + try {
95 + // Unescape the string (handle \" -> ")
96 + const unescaped = str.replace(/\\"/g, '"').replace(/\\\\/g, '\\');
97 + console.log('First 200 chars after unescape:', unescaped.substring(0, 200));
98 + const parsed = JSON.parse(unescaped);
99 + console.log(`Successfully parsed unescaped JSON, formatting at indent ${indent + 1}`);
100 + // If it's valid JSON, format it as nested JSON
101 + const formatted = this.prettyPrint(parsed, indent + 1);
102 + // Wrap in quotes but show it's parsed JSON
103 + return this._colorize('"', 'string') +
104 + this._colorize('[JSON] ', 'newline') +
105 + formatted +
106 + this._colorize('"', 'string');
107 + } catch (e2) {
108 + console.error('Failed to parse even after unescaping:', e2.message);
109 + // Not valid JSON, continue with normal string formatting
110 + }
111 + }
112 + }
113 +
114 + // Escape special characters
115 + let escaped = str
116 + .replace(/\\/g, '\\\\')
117 + .replace(/"/g, '\\"')
118 + .replace(/\t/g, '\\t')
119 + .replace(/\r/g, '\\r');
120 +
121 + // Visualize newlines if enabled
122 + if (this.options.visualizeNewlines && escaped.includes('\\n')) {
123 + // Split by newlines but keep them
124 + const parts = escaped.split(/(\\n)/);
125 + const formatted = parts.map((part, i) => {
126 + if (part === '\\n') {
127 + return `<span class="json-newline-wrapper">` +
128 + `<span class="json-newline">\\n</span>` +
129 + `</span>`;
130 + }
131 + return this._escapeHtml(part);
132 + }).join('');
133 +
134 + return this._colorize('"', 'string') +
135 + `<span class="json-string">` + formatted + `</span>` +
136 + this._colorize('"', 'string');
137 + }
138 +
139 + // Regular string
140 + return this._colorize('"' + this._escapeHtml(escaped) + '"', 'string');
141 + }
142 +
143 + /**
144 + * Format an object
145 + */
146 + _formatObject(obj, indent) {
147 + const keys = Object.keys(obj);
148 + if (keys.length === 0) {
149 + return this._colorize('{}', 'punctuation');
150 + }
151 +
152 + const spaces = ' '.repeat(indent * this.options.indent);
153 + const innerSpaces = ' '.repeat((indent + 1) * this.options.indent);
154 +
155 + let result = this._colorize('{', 'punctuation') + '\n';
156 +
157 + keys.forEach((key, index) => {
158 + result += innerSpaces;
159 + result += this._colorize('"' + this._escapeHtml(key) + '"', 'key');
160 + result += this._colorize(': ', 'punctuation');
161 + result += this.prettyPrint(obj[key], indent + 1);
162 +
163 + if (index < keys.length - 1) {
164 + result += this._colorize(',', 'punctuation');
165 + }
166 + result += '\n';
167 + });
168 +
169 + result += spaces + this._colorize('}', 'punctuation');
170 + return result;
171 + }
172 +
173 + /**
174 + * Format an array
175 + */
176 + _formatArray(arr, indent) {
177 + if (arr.length === 0) {
178 + return this._colorize('[]', 'punctuation');
179 + }
180 +
181 + const spaces = ' '.repeat(indent * this.options.indent);
182 + const innerSpaces = ' '.repeat((indent + 1) * this.options.indent);
183 +
184 + let result = this._colorize('[', 'punctuation') + '\n';
185 +
186 + arr.forEach((item, index) => {
187 + result += innerSpaces;
188 + result += this.prettyPrint(item, indent + 1);
189 +
190 + if (index < arr.length - 1) {
191 + result += this._colorize(',', 'punctuation');
192 + }
193 + result += '\n';
194 + });
195 +
196 + result += spaces + this._colorize(']', 'punctuation');
197 + return result;
198 + }
199 +
200 + /**
201 + * Check if a string looks like it might contain JSON
202 + */
203 + _looksLikeJSON(str) {
204 + if (str.length > this.options.maxStringLength) {
205 + return false;
206 + }
207 +
208 + const trimmed = str.trim();
209 + return (trimmed.startsWith('{') && trimmed.endsWith('}')) ||
210 + (trimmed.startsWith('[') && trimmed.endsWith(']'));
211 + }
212 +
213 + /**
214 + * Apply color to text based on type
215 + */
216 + _colorize(text, type) {
217 + if (!this.options.syntaxHighlight) {
218 + return text;
219 + }
220 +
221 + const color = this.options.colors[type];
222 + if (!color) {
223 + return text;
224 + }
225 +
226 + return `<span class="json-${type}" style="color: ${color}">${text}</span>`;
227 + }
228 +
229 + /**
230 + * Escape HTML special characters
231 + */
232 + _escapeHtml(str) {
233 + const div = document.createElement('div');
234 + div.textContent = str;
235 + return div.innerHTML;
236 + }
237 +
238 + /**
239 + * Convert to HTML with proper styling
240 + */
241 + toHTML(obj) {
242 + const content = this.prettyPrint(obj);
243 +
244 + // Add CSS for newline visualization
245 + const style = `
246 + <style>
247 + .json-pretty {
248 + font-family: 'Courier New', Consolas, monospace;
249 + font-size: 13px;
250 + line-height: 1.5;
251 + white-space: pre;
252 + overflow-x: auto;
253 + }
254 + .json-newline-wrapper {
255 + display: inline;
256 + position: relative;
257 + }
258 + .json-newline {
259 + background-color: ${this.options.colors.background.newline};
260 + color: ${this.options.colors.newline};
261 + padding: 1px 3px;
262 + border-radius: 3px;
263 + font-size: 11px;
264 + font-weight: bold;
265 + margin: 0 1px;
266 + }
267 + .json-string {
268 + color: ${this.options.colors.string};
269 + }
270 + .json-key {
271 + color: ${this.options.colors.key};
272 + }
273 + .json-number {
274 + color: ${this.options.colors.number};
275 + }
276 + .json-boolean {
277 + color: ${this.options.colors.boolean};
278 + }
279 + .json-null {
280 + color: ${this.options.colors.null};
281 + }
282 + .json-punctuation {
283 + color: ${this.options.colors.punctuation};
284 + }
285 + </style>
286 + `;
287 +
288 + return style + '<div class="json-pretty">' + content + '</div>';
289 + }
290 +
291 + /**
292 + * Convert to plain HTML without wrapper (for embedding)
293 + */
294 + toHTMLContent(obj) {
295 + return this.prettyPrint(obj);
296 + }
297 +
298 + /**
299 + * Format for console output (no HTML)
300 + */
301 + toConsole(obj) {
302 + const oldHighlight = this.options.syntaxHighlight;
303 + this.options.syntaxHighlight = false;
304 + const result = this.prettyPrint(obj);
305 + this.options.syntaxHighlight = oldHighlight;
306 + return result;
307 + }
308 +}
309 +
310 +// Export for use
311 +if (typeof module !== 'undefined' && module.exports) {
312 + module.exports = JSONPrettyPrinter;
313 +}
\ No newline at end of file
src/web/mcp/mcp-test-client/mcp-schema-ui-generator.js new
+2467
@@ -0,0 +1,2467 @@
1 +/**
2 + * MCP Schema UI Generator
3 + * A lightweight JSON Schema form generator specifically designed for MCP tool parameters
4 + */
5 +
6 +class MCPSchemaUIGenerator {
7 + constructor(container, options = {}) {
8 + this.container = container;
9 + this.options = {
10 + onChange: options.onChange || (() => {}),
11 + showDescriptions: options.showDescriptions !== false,
12 + showTooltips: options.showTooltips !== false,
13 + ...options
14 + };
15 + this.value = {};
16 + this.schema = null;
17 + this.errors = {};
18 + this.fieldElements = new Map();
19 + this._changeTimer = null;
20 + }
21 +
22 + /**
23 + * Set the schema and render the form
24 + */
25 + setSchema(schema, initialValue = {}) {
26 + // Validate schema against MCP spec before using it
27 + this._validateMCPSchema(schema);
28 +
29 + this.schema = schema;
30 + this.value = this._deepClone(initialValue);
31 + this.errors = {};
32 + this.render();
33 + }
34 +
35 + /**
36 + * Get the current form value
37 + */
38 + getValue() {
39 + const cleanedValue = {};
40 + for (const [key, value] of Object.entries(this.value)) {
41 + // Filter out anyOf helper keys (they have _anyof_ in their name)
42 + if (!key.includes('_anyof_')) {
43 + cleanedValue[key] = value;
44 + }
45 + }
46 + return this._deepClone(cleanedValue);
47 + }
48 +
49 + /**
50 + * Set the form value
51 + */
52 + setValue(value) {
53 + this.value = this._deepClone(value);
54 + this._updateFieldValues();
55 + }
56 +
57 + /**
58 + * Validate the current value against the schema
59 + */
60 + validate() {
61 + this.errors = {};
62 + if (!this.schema) return true;
63 +
64 + // Check required fields
65 + if (this.schema.required && Array.isArray(this.schema.required)) {
66 + for (const field of this.schema.required) {
67 + if (!this.value.hasOwnProperty(field) ||
68 + this.value[field] === null ||
69 + this.value[field] === undefined ||
70 + (typeof this.value[field] === 'string' && this.value[field].trim() === '')) {
71 + this.errors[field] = 'This field is required';
72 + }
73 + }
74 + }
75 +
76 + // Validate individual fields
77 + if (this.schema.properties) {
78 + for (const [key, propSchema] of Object.entries(this.schema.properties)) {
79 + const value = this.value[key];
80 + const error = this._validateField(value, propSchema);
81 + if (error) {
82 + this.errors[key] = error;
83 + }
84 + }
85 + }
86 +
87 + this._updateErrorDisplay();
88 + return Object.keys(this.errors).length === 0;
89 + }
90 +
91 + /**
92 + * Render the form
93 + */
94 + render() {
95 + this.container.innerHTML = '';
96 + this.fieldElements.clear();
97 +
98 + if (!this.schema || !this.schema.properties) {
99 + this.container.innerHTML = '<p>No schema provided</p>';
100 + return;
101 + }
102 +
103 + // Add title if present
104 + if (this.schema.title) {
105 + const title = document.createElement('h4');
106 + title.className = 'mcp-form-title';
107 + title.textContent = this.schema.title;
108 + this.container.appendChild(title);
109 + }
110 +
111 + // Add description if present
112 + if (this.schema.description && this.options.showDescriptions) {
113 + const desc = document.createElement('p');
114 + desc.className = 'mcp-form-description';
115 + desc.textContent = this.schema.description;
116 + this.container.appendChild(desc);
117 + }
118 +
119 + // Create form wrapper
120 + const form = document.createElement('div');
121 + form.className = 'mcp-form';
122 + this.container.appendChild(form);
123 +
124 + // Render each property
125 + for (const [key, propSchema] of Object.entries(this.schema.properties)) {
126 + // Initialize with default value if not already set
127 + if (this.value[key] === undefined && propSchema.default !== undefined) {
128 + this.value[key] = propSchema.default;
129 + }
130 +
131 + const fieldContainer = this._renderField(key, propSchema, this.value[key]);
132 + form.appendChild(fieldContainer);
133 + this.fieldElements.set(key, fieldContainer);
134 + }
135 +
136 + // Add CSS if not already present
137 + this._injectStyles();
138 + }
139 +
140 + /**
141 + * Render a single field
142 + */
143 + _renderField(key, schema, value) {
144 + const container = document.createElement('div');
145 + container.className = 'mcp-field';
146 + container.dataset.field = key;
147 +
148 + // Label
149 + const labelWrapper = document.createElement('div');
150 + labelWrapper.className = 'mcp-field-label-wrapper';
151 +
152 + const label = document.createElement('label');
153 + label.className = 'mcp-field-label';
154 +
155 + // Add tooltip functionality if description exists
156 + if (schema.description && this.options.showTooltips) {
157 + label.className += ' mcp-field-label-hoverable';
158 +
159 + // Add hover handlers for proper positioning
160 + let tooltipEl = null;
161 + label.addEventListener('mouseenter', (e) => {
162 + const rect = label.getBoundingClientRect();
163 + tooltipEl = document.createElement('div');
164 + tooltipEl.className = 'mcp-tooltip-popup';
165 + // Convert newlines to <br/> tags
166 + tooltipEl.innerHTML = schema.description.replace(/\n/g, '<br/>');
167 + tooltipEl.style.position = 'fixed';
168 + tooltipEl.style.left = `${rect.left}px`;
169 + tooltipEl.style.top = `${rect.top - 5}px`;
170 + tooltipEl.style.transform = 'translateY(-100%)';
171 + document.body.appendChild(tooltipEl);
172 + });
173 +
174 + label.addEventListener('mouseleave', () => {
175 + if (tooltipEl) {
176 + tooltipEl.remove();
177 + tooltipEl = null;
178 + }
179 + });
180 + }
181 +
182 + label.textContent = schema.title || key;
183 +
184 + // Mark required fields
185 + if (this.schema.required && this.schema.required.includes(key)) {
186 + const required = document.createElement('span');
187 + required.className = 'mcp-field-required';
188 + required.textContent = '*';
189 + label.appendChild(required);
190 + }
191 +
192 + labelWrapper.appendChild(label);
193 +
194 + // Add JSON field name
195 + const fieldName = document.createElement('span');
196 + fieldName.className = 'mcp-field-json-name';
197 + fieldName.textContent = key;
198 + labelWrapper.appendChild(fieldName);
199 +
200 + container.appendChild(labelWrapper);
201 +
202 + // Description (below label)
203 + if (schema.description && this.options.showDescriptions && !this.options.showTooltips) {
204 + const desc = document.createElement('div');
205 + desc.className = 'mcp-field-description';
206 + desc.innerHTML = schema.description.replace(/\n/g, '<br/>');
207 + container.appendChild(desc);
208 + }
209 +
210 + // Input
211 + const inputContainer = document.createElement('div');
212 + inputContainer.className = 'mcp-field-input-container';
213 +
214 + const input = this._createInput(key, schema, value);
215 + inputContainer.appendChild(input);
216 + container.appendChild(inputContainer);
217 +
218 + // Error message placeholder
219 + const error = document.createElement('div');
220 + error.className = 'mcp-field-error';
221 + container.appendChild(error);
222 +
223 + return container;
224 + }
225 +
226 + /**
227 + * Create appropriate input element based on schema
228 + */
229 + _createInput(key, schema, value) {
230 + // Handle enum
231 + if (schema.enum) {
232 + return this._createSelect(key, schema, value);
233 + }
234 +
235 + // Handle type-specific inputs
236 + switch (schema.type) {
237 + case 'string':
238 + return this._createStringInput(key, schema, value);
239 + case 'number':
240 + case 'integer':
241 + return this._createNumberInput(key, schema, value);
242 + case 'boolean':
243 + return this._createBooleanInput(key, schema, value);
244 + case 'array':
245 + return this._createArrayInput(key, schema, value);
246 + case 'object':
247 + return this._createObjectInput(key, schema, value);
248 + default:
249 + // Handle anyOf/oneOf
250 + if (schema.anyOf || schema.oneOf) {
251 + return this._createMultiTypeInput(key, schema, value);
252 + }
253 + return this._createStringInput(key, schema, value);
254 + }
255 + }
256 +
257 + /**
258 + * Create string input
259 + */
260 + _createStringInput(key, schema, value) {
261 + const input = document.createElement('input');
262 + input.type = 'text';
263 + input.className = 'mcp-input';
264 + input.value = value || schema.default || '';
265 + input.placeholder = schema.placeholder || '';
266 +
267 + if (schema.minLength) input.minLength = schema.minLength;
268 + if (schema.maxLength) input.maxLength = schema.maxLength;
269 + if (schema.pattern) input.pattern = schema.pattern;
270 +
271 + input.addEventListener('input', () => {
272 + this.value[key] = input.value;
273 + this._onChange();
274 + });
275 +
276 + return input;
277 + }
278 +
279 + /**
280 + * Create number input
281 + */
282 + _createNumberInput(key, schema, value) {
283 + const input = document.createElement('input');
284 + input.type = 'number';
285 + input.className = 'mcp-input';
286 + input.value = value !== undefined ? value : (schema.default !== undefined ? schema.default : '');
287 +
288 + if (schema.minimum !== undefined) input.min = schema.minimum;
289 + if (schema.maximum !== undefined) input.max = schema.maximum;
290 + if (schema.multipleOf !== undefined) input.step = schema.multipleOf;
291 +
292 + input.addEventListener('input', () => {
293 + const val = input.value === '' ? undefined : Number(input.value);
294 + this.value[key] = val;
295 + this._onChange();
296 + });
297 +
298 + return input;
299 + }
300 +
301 + /**
302 + * Create boolean input
303 + */
304 + _createBooleanInput(key, schema, value) {
305 + const wrapper = document.createElement('label');
306 + wrapper.className = 'mcp-checkbox-wrapper';
307 +
308 + const input = document.createElement('input');
309 + input.type = 'checkbox';
310 + input.className = 'mcp-checkbox';
311 + input.checked = value !== undefined ? value : (schema.default || false);
312 +
313 + input.addEventListener('change', () => {
314 + this.value[key] = input.checked;
315 + this._onChange();
316 + });
317 +
318 + const text = document.createElement('span');
319 + text.textContent = schema.title || key;
320 +
321 + wrapper.appendChild(input);
322 + wrapper.appendChild(text);
323 +
324 + return wrapper;
325 + }
326 +
327 + /**
328 + * Create select dropdown
329 + */
330 + _createSelect(key, schema, value) {
331 + const select = document.createElement('select');
332 + select.className = 'mcp-select';
333 +
334 + // Always add empty option if there's no default value
335 + // This ensures nothing is pre-selected when there's no default
336 + const hasDefault = schema.default !== undefined;
337 + const isRequired = this.schema.required && this.schema.required.includes(key);
338 +
339 + if (!hasDefault || !isRequired) {
340 + const emptyOption = document.createElement('option');
341 + emptyOption.value = '';
342 + emptyOption.textContent = '-- Not selected --';
343 + select.appendChild(emptyOption);
344 + }
345 +
346 + // Add enum options
347 + for (const option of schema.enum) {
348 + const optionEl = document.createElement('option');
349 + optionEl.value = option;
350 + optionEl.textContent = option;
351 + select.appendChild(optionEl);
352 + }
353 +
354 + // Set the selected value
355 + if (value !== undefined && value !== null && value !== '') {
356 + select.value = value;
357 + } else if (hasDefault) {
358 + select.value = schema.default;
359 + // Also set it in the data model
360 + this.value[key] = schema.default;
361 + } else {
362 + // No value and no default - select the empty option
363 + select.value = '';
364 + }
365 +
366 + select.addEventListener('change', () => {
367 + this.value[key] = select.value || undefined;
368 + this._onChange();
369 + });
370 +
371 + return select;
372 + }
373 +
374 + /**
375 + * Create array input
376 + */
377 + _createArrayInput(key, schema, value) {
378 + // Check if this is an array of enums - use checkbox interface
379 + if (schema.items && schema.items.enum && Array.isArray(schema.items.enum)) {
380 + return this._createArrayEnumCheckboxes(key, schema, value);
381 + }
382 +
383 + // Check if this is a tuple array with oneOf items
384 + if (schema.items && schema.items.type === 'array' && schema.items.items && schema.items.items.oneOf) {
385 + return this._createTupleArrayInput(key, schema, value);
386 + }
387 +
388 + // Regular array interface
389 + const container = document.createElement('div');
390 + container.className = 'mcp-array-container';
391 +
392 + const items = Array.isArray(value) ? value : (schema.default || []);
393 + this.value[key] = [...items];
394 +
395 + const itemsContainer = document.createElement('div');
396 + itemsContainer.className = 'mcp-array-items';
397 +
398 + const renderItems = () => {
399 + itemsContainer.innerHTML = '';
400 + this.value[key].forEach((item, index) => {
401 + const itemEl = this._createArrayItem(key, schema, item, index);
402 + itemsContainer.appendChild(itemEl);
403 + });
404 + };
405 +
406 + renderItems();
407 + container.appendChild(itemsContainer);
408 +
409 + // Add button
410 + const addButton = document.createElement('button');
411 + addButton.type = 'button';
412 + addButton.className = 'mcp-button mcp-button-add';
413 + addButton.textContent = `Add ${schema.title || key}`;
414 + addButton.addEventListener('click', () => {
415 + const newItem = this._getDefaultValue(schema.items);
416 + if (!Array.isArray(this.value[key])) {
417 + this.value[key] = [];
418 + }
419 + this.value[key].push(newItem);
420 + renderItems();
421 + this._onChange();
422 + });
423 +
424 + container.appendChild(addButton);
425 +
426 + return container;
427 + }
428 +
429 + /**
430 + * Create array of enum checkboxes
431 + */
432 + _createArrayEnumCheckboxes(key, schema, value) {
433 + const container = document.createElement('div');
434 + container.className = 'mcp-array-enum-container';
435 +
436 + // Initialize value
437 + const currentValue = Array.isArray(value) ? [...value] : (schema.default ? [...schema.default] : []);
438 + this.value[key] = currentValue;
439 +
440 + // Create checkboxes for each enum option
441 + schema.items.enum.forEach(enumValue => {
442 + const checkboxWrapper = document.createElement('label');
443 + checkboxWrapper.className = 'mcp-array-enum-item';
444 +
445 + const checkbox = document.createElement('input');
446 + checkbox.type = 'checkbox';
447 + checkbox.className = 'mcp-checkbox';
448 + checkbox.value = enumValue;
449 + checkbox.checked = this.value[key].includes(enumValue);
450 +
451 + checkbox.addEventListener('change', () => {
452 + if (!Array.isArray(this.value[key])) {
453 + this.value[key] = [];
454 + }
455 +
456 + if (checkbox.checked) {
457 + // Add to array if not already present
458 + if (!this.value[key].includes(enumValue)) {
459 + this.value[key].push(enumValue);
460 + }
461 + } else {
462 + // Remove from array
463 + const index = this.value[key].indexOf(enumValue);
464 + if (index !== -1) {
465 + this.value[key].splice(index, 1);
466 + }
467 + }
468 + this._onChange();
469 + });
470 +
471 + const label = document.createElement('span');
472 + label.textContent = enumValue;
473 +
474 + checkboxWrapper.appendChild(checkbox);
475 + checkboxWrapper.appendChild(label);
476 + container.appendChild(checkboxWrapper);
477 + });
478 +
479 + return container;
480 + }
481 +
482 + /**
483 + * Create tuple array input (array of arrays with specific types at each position)
484 + */
485 + _createTupleArrayInput(key, schema, value) {
486 + const container = document.createElement('div');
487 + container.className = 'mcp-array-container mcp-tuple-array-container';
488 +
489 + const items = Array.isArray(value) ? value : (schema.default || []);
490 + this.value[key] = [...items];
491 +
492 + const itemsContainer = document.createElement('div');
493 + itemsContainer.className = 'mcp-array-items';
494 +
495 + const renderItems = () => {
496 + itemsContainer.innerHTML = '';
497 + this.value[key].forEach((item, index) => {
498 + const itemEl = this._createTupleArrayItem(key, schema.items, item, index);
499 + itemsContainer.appendChild(itemEl);
500 + });
501 + };
502 +
503 + renderItems();
504 + container.appendChild(itemsContainer);
505 +
506 + // Add button
507 + const addButton = document.createElement('button');
508 + addButton.type = 'button';
509 + addButton.className = 'mcp-button mcp-button-add';
510 + addButton.textContent = `Add ${schema.title || key}`;
511 + addButton.addEventListener('click', () => {
512 + // Create a new tuple with default values for each position
513 + const newTuple = this._createDefaultTuple(schema.items);
514 + if (!Array.isArray(this.value[key])) {
515 + this.value[key] = [];
516 + }
517 + this.value[key].push(newTuple);
518 + renderItems();
519 + this._onChange();
520 + });
521 +
522 + container.appendChild(addButton);
523 +
524 + return container;
525 + }
526 +
527 + /**
528 + * Create a tuple array item (array with specific types at each position)
529 + */
530 + _createTupleArrayItem(arrayKey, tupleSchema, tupleValue, index) {
531 + const container = document.createElement('div');
532 + container.className = 'mcp-array-item mcp-tuple-item';
533 +
534 + const fieldsWrapper = document.createElement('div');
535 + fieldsWrapper.className = 'mcp-tuple-fields';
536 +
537 + // Ensure the tuple value is an array
538 + if (!Array.isArray(tupleValue)) {
539 + tupleValue = this._createDefaultTuple(tupleSchema);
540 + this.value[arrayKey][index] = tupleValue;
541 + }
542 +
543 + // Handle oneOf items - each position in the tuple has a specific type
544 + if (tupleSchema.items && tupleSchema.items.oneOf && Array.isArray(tupleSchema.items.oneOf)) {
545 + tupleSchema.items.oneOf.forEach((positionSchema, positionIndex) => {
546 + const fieldContainer = document.createElement('div');
547 + fieldContainer.className = 'mcp-tuple-field';
548 +
549 + // Create label if we can determine what this position represents
550 + const label = document.createElement('label');
551 + label.className = 'mcp-tuple-field-label';
552 + if (positionIndex === 0) label.textContent = 'Column';
553 + else if (positionIndex === 1) label.textContent = 'Operator';
554 + else if (positionIndex === 2) label.textContent = 'Value';
555 + else label.textContent = `Field ${positionIndex + 1}`;
556 + fieldContainer.appendChild(label);
557 +
558 + // Get the value for this position
559 + const positionValue = tupleValue[positionIndex];
560 +
561 + // For the value field with oneOf, create a special multi-type input
562 + if (positionSchema.oneOf && positionIndex === 2) {
563 + const multiTypeContainer = this._createTupleMultiTypeInput(
564 + arrayKey, index, positionIndex, positionSchema, positionValue
565 + );
566 + fieldContainer.appendChild(multiTypeContainer);
567 + } else {
568 + // Create regular input for this position
569 + const input = this._createInput(`${arrayKey}[${index}][${positionIndex}]`, positionSchema, positionValue);
570 +
571 + // We need to handle the value updates ourselves
572 + const newInput = input.cloneNode(true);
573 + input.parentNode?.replaceChild(newInput, input);
574 +
575 + // For select elements, we need to set the value after cloning
576 + if (newInput.tagName === 'SELECT' && positionValue !== undefined && positionValue !== null) {
577 + newInput.value = positionValue;
578 + }
579 +
580 + const updateValue = () => {
581 + if (!Array.isArray(this.value[arrayKey])) {
582 + this.value[arrayKey] = [];
583 + }
584 + if (!Array.isArray(this.value[arrayKey][index])) {
585 + this.value[arrayKey][index] = [];
586 + }
587 +
588 + let newValue;
589 + if (positionSchema.type === 'number' || positionSchema.type === 'integer') {
590 + newValue = newInput.value === '' ? undefined : Number(newInput.value);
591 + } else if (positionSchema.type === 'boolean') {
592 + newValue = newInput.checked;
593 + } else {
594 + newValue = newInput.value;
595 + }
596 +
597 + this.value[arrayKey][index][positionIndex] = newValue;
598 + this._onChange();
599 + };
600 +
601 + if (newInput.tagName === 'INPUT' || newInput.tagName === 'SELECT' || newInput.tagName === 'TEXTAREA') {
602 + newInput.addEventListener(newInput.type === 'checkbox' ? 'change' : 'input', updateValue);
603 + }
604 +
605 + fieldContainer.appendChild(newInput);
606 + }
607 +
608 + fieldsWrapper.appendChild(fieldContainer);
609 + });
610 + }
611 +
612 + container.appendChild(fieldsWrapper);
613 +
614 + // Remove button
615 + const removeButton = document.createElement('button');
616 + removeButton.type = 'button';
617 + removeButton.className = 'mcp-button mcp-button-remove';
618 + removeButton.innerHTML = '×';
619 + removeButton.title = 'Remove';
620 + removeButton.addEventListener('click', () => {
621 + if (!Array.isArray(this.value[arrayKey])) {
622 + this.value[arrayKey] = [];
623 + }
624 + this.value[arrayKey].splice(index, 1);
625 +
626 + // Re-render the parent array container
627 + // Find the parent array container and its render function
628 + const arrayContainer = container.closest('.mcp-tuple-array-container');
629 + if (arrayContainer) {
630 + const itemsContainer = arrayContainer.querySelector('.mcp-array-items');
631 + if (itemsContainer) {
632 + // Re-render all items
633 + itemsContainer.innerHTML = '';
634 + if (Array.isArray(this.value[arrayKey])) {
635 + this.value[arrayKey].forEach((item, idx) => {
636 + const itemEl = this._createTupleArrayItem(arrayKey, tupleSchema, item, idx);
637 + itemsContainer.appendChild(itemEl);
638 + });
639 + }
640 + }
641 + }
642 +
643 + this._onChange();
644 + });
645 + container.appendChild(removeButton);
646 +
647 + return container;
648 + }
649 +
650 + /**
651 + * Create a default tuple based on the schema
652 + */
653 + _createDefaultTuple(tupleSchema) {
654 + if (tupleSchema.items && tupleSchema.items.oneOf && Array.isArray(tupleSchema.items.oneOf)) {
655 + return tupleSchema.items.oneOf.map(positionSchema => {
656 + return this._getDefaultValue(positionSchema);
657 + });
658 + }
659 + return [];
660 + }
661 +
662 + /**
663 + * Create a multi-type input specifically for tuple arrays
664 + */
665 + _createTupleMultiTypeInput(arrayKey, tupleIndex, positionIndex, schema, value) {
666 + const container = document.createElement('div');
667 + container.className = 'mcp-tuple-multitype-container';
668 +
669 + // Get the options from oneOf
670 + const options = schema.oneOf || [];
671 +
672 + // Create type selector tabs
673 + const tabsContainer = document.createElement('div');
674 + tabsContainer.className = 'mcp-anyof-tabs mcp-tuple-anyof-tabs';
675 +
676 + const inputContainer = document.createElement('div');
677 + inputContainer.className = 'mcp-tuple-multitype-input';
678 +
679 + let activeType = 'string';
680 + let currentInput = null;
681 +
682 + // Try to determine the current type based on value
683 + if (value !== undefined && value !== null) {
684 + if (typeof value === 'boolean') {
685 + activeType = 'boolean';
686 + } else if (typeof value === 'number') {
687 + activeType = 'number';
688 + } else {
689 + activeType = 'string';
690 + }
691 + }
692 +
693 + // Create tabs for each type
694 + const tabs = {};
695 + options.forEach((optionSchema, optionIndex) => {
696 + const tab = document.createElement('button');
697 + tab.type = 'button';
698 + tab.className = 'mcp-anyof-tab';
699 + tab.textContent = optionSchema.type || 'unknown';
700 +
701 + if (optionSchema.type === activeType) {
702 + tab.classList.add('active');
703 + }
704 +
705 + tab.addEventListener('click', () => {
706 + // Update active tab
707 + Object.values(tabs).forEach(t => t.classList.remove('active'));
708 + tab.classList.add('active');
709 +
710 + // Create new input for this type
711 + activeType = optionSchema.type;
712 + this._updateTupleMultiTypeInput(
713 + inputContainer, arrayKey, tupleIndex, positionIndex,
714 + optionSchema, activeType === 'boolean' ? false : (activeType === 'number' ? 0 : '')
715 + );
716 + });
717 +
718 + tabs[optionSchema.type] = tab;
719 + tabsContainer.appendChild(tab);
720 + });
721 +
722 + container.appendChild(tabsContainer);
723 + container.appendChild(inputContainer);
724 +
725 + // Create initial input
726 + const initialSchema = options.find(o => o.type === activeType) || options[0];
727 + this._updateTupleMultiTypeInput(inputContainer, arrayKey, tupleIndex, positionIndex, initialSchema, value);
728 +
729 + return container;
730 + }
731 +
732 + /**
733 + * Update the input in a tuple multi-type container
734 + */
735 + _updateTupleMultiTypeInput(container, arrayKey, tupleIndex, positionIndex, schema, value) {
736 + container.innerHTML = '';
737 +
738 + let input;
739 + if (schema.type === 'boolean') {
740 + input = document.createElement('input');
741 + input.type = 'checkbox';
742 + input.className = 'mcp-checkbox';
743 + input.checked = !!value;
744 + } else if (schema.type === 'number') {
745 + input = document.createElement('input');
746 + input.type = 'number';
747 + input.className = 'mcp-input';
748 + input.value = value !== undefined ? value : '';
749 + } else {
750 + input = document.createElement('input');
751 + input.type = 'text';
752 + input.className = 'mcp-input';
753 + input.value = value || '';
754 + }
755 +
756 + const updateValue = () => {
757 + if (!Array.isArray(this.value[arrayKey])) {
758 + this.value[arrayKey] = [];
759 + }
760 + if (!Array.isArray(this.value[arrayKey][tupleIndex])) {
761 + this.value[arrayKey][tupleIndex] = [];
762 + }
763 +
764 + let newValue;
765 + if (schema.type === 'boolean') {
766 + newValue = input.checked;
767 + } else if (schema.type === 'number') {
768 + newValue = input.value === '' ? undefined : Number(input.value);
769 + } else {
770 + newValue = input.value;
771 + }
772 +
773 + this.value[arrayKey][tupleIndex][positionIndex] = newValue;
774 + this._onChange();
775 + };
776 +
777 + input.addEventListener(schema.type === 'boolean' ? 'change' : 'input', updateValue);
778 + container.appendChild(input);
779 + }
780 +
781 + /**
782 + * Create array item
783 + */
784 + _createArrayItem(arrayKey, schema, value, index) {
785 + const container = document.createElement('div');
786 + container.className = 'mcp-array-item';
787 +
788 + const inputWrapper = document.createElement('div');
789 + inputWrapper.className = 'mcp-array-item-input';
790 +
791 + // Create input based on item schema
792 + const itemSchema = schema.items || { type: 'string' };
793 + const input = this._createInput(`${arrayKey}[${index}]`, itemSchema, value);
794 +
795 + // We need to clear any event listeners that were added by _createInput
796 + // because they would try to set this.value["dimensions[0]"] instead of this.value["dimensions"][0]
797 + const newInput = input.cloneNode(true);
798 + input.parentNode?.replaceChild(newInput, input);
799 +
800 + // For select elements, we need to set the value after cloning
801 + if (newInput.tagName === 'SELECT' && value !== undefined && value !== null) {
802 + newInput.value = value;
803 + }
804 +
805 + // Add our own handler to update the array properly
806 + const updateValue = () => {
807 + if (!Array.isArray(this.value[arrayKey])) {
808 + this.value[arrayKey] = [];
809 + }
810 + if (itemSchema.type === 'number' || itemSchema.type === 'integer') {
811 + this.value[arrayKey][index] = newInput.value === '' ? undefined : Number(newInput.value);
812 + } else if (itemSchema.type === 'boolean') {
813 + this.value[arrayKey][index] = newInput.checked;
814 + } else {
815 + this.value[arrayKey][index] = newInput.value;
816 + }
817 + this._onChange();
818 + };
819 +
820 + if (newInput.tagName === 'INPUT' || newInput.tagName === 'SELECT') {
821 + newInput.addEventListener(newInput.type === 'checkbox' ? 'change' : 'input', updateValue);
822 + }
823 +
824 + inputWrapper.appendChild(newInput);
825 + container.appendChild(inputWrapper);
826 +
827 + // Remove button
828 + const removeButton = document.createElement('button');
829 + removeButton.type = 'button';
830 + removeButton.className = 'mcp-button mcp-button-remove';
831 + removeButton.innerHTML = '×';
832 + removeButton.title = 'Remove';
833 + removeButton.addEventListener('click', () => {
834 + if (!Array.isArray(this.value[arrayKey])) {
835 + this.value[arrayKey] = [];
836 + }
837 + this.value[arrayKey].splice(index, 1);
838 + const arrayContainer = container.parentElement.parentElement;
839 + const renderItems = () => {
840 + const itemsContainer = arrayContainer.querySelector('.mcp-array-items');
841 + itemsContainer.innerHTML = '';
842 + this.value[arrayKey].forEach((item, idx) => {
843 + const itemEl = this._createArrayItem(arrayKey, schema, item, idx);
844 + itemsContainer.appendChild(itemEl);
845 + });
846 + };
847 + renderItems();
848 + this._onChange();
849 + });
850 +
851 + container.appendChild(removeButton);
852 +
853 + return container;
854 + }
855 +
856 + /**
857 + * Create object input
858 + */
859 + _createObjectInput(key, schema, value) {
860 + // Check if this is a dynamic object with additionalProperties
861 + if (schema.additionalProperties && !schema.properties) {
862 + return this._createDynamicObjectInput(key, schema, value);
863 + }
864 +
865 + // For objects with defined properties, create a nested form
866 + if (schema.properties) {
867 + return this._createNestedObjectInput(key, schema, value);
868 + }
869 +
870 + // Fallback to JSON editor for unstructured objects
871 + const textarea = document.createElement('textarea');
872 + textarea.className = 'mcp-textarea';
873 + textarea.rows = 4;
874 +
875 + try {
876 + textarea.value = JSON.stringify(value || schema.default || {}, null, 2);
877 + } catch (e) {
878 + textarea.value = '{}';
879 + }
880 +
881 + textarea.addEventListener('input', () => {
882 + try {
883 + this.value[key] = JSON.parse(textarea.value);
884 + textarea.classList.remove('mcp-input-error');
885 + } catch (e) {
886 + textarea.classList.add('mcp-input-error');
887 + }
888 + this._onChange();
889 + });
890 +
891 + return textarea;
892 + }
893 +
894 + /**
895 + * Create dynamic object input for additionalProperties
896 + */
897 + _createDynamicObjectInput(key, schema, value) {
898 + const container = document.createElement('div');
899 + container.className = 'mcp-dynamic-object-container';
900 +
901 + // Initialize value
902 + const objectValue = value || {};
903 + this.value[key] = objectValue;
904 +
905 + const itemsContainer = document.createElement('div');
906 + itemsContainer.className = 'mcp-dynamic-object-items';
907 +
908 + // Keep track of items with temporary keys to manage renaming
909 + let itemIndex = 0;
910 + const items = new Map();
911 +
912 + // Initialize items from existing value
913 + Object.entries(objectValue).forEach(([k, v]) => {
914 + items.set(itemIndex++, { key: k, value: v });
915 + });
916 +
917 + const updateData = () => {
918 + // Rebuild the object from items
919 + const newObject = {};
920 + items.forEach((item) => {
921 + if (item.key) {
922 + newObject[item.key] = item.value;
923 + }
924 + });
925 + this.value[key] = newObject;
926 + this._onChange();
927 + };
928 +
929 + const renderItems = () => {
930 + itemsContainer.innerHTML = '';
931 +
932 + items.forEach((item, idx) => {
933 + const itemEl = this._createEditableDynamicObjectItem(
934 + key, idx, item, schema.additionalProperties,
935 + updateData, // Just update data, don't re-render
936 + items, // Pass the items Map
937 + renderItems // Pass renderItems separately for when we need full re-render
938 + );
939 + itemsContainer.appendChild(itemEl);
940 + });
941 + };
942 +
943 + renderItems();
944 + container.appendChild(itemsContainer);
945 +
946 + // Add button
947 + const addButton = document.createElement('button');
948 + addButton.type = 'button';
949 + addButton.className = 'mcp-button mcp-button-add';
950 + addButton.textContent = 'Add Item';
951 + addButton.addEventListener('click', () => {
952 + // Add new item with empty key and default value
953 + const defaultValue = this._getDefaultValue(schema.additionalProperties);
954 + items.set(itemIndex++, { key: '', value: defaultValue });
955 + renderItems();
956 + });
957 +
958 + container.appendChild(addButton);
959 +
960 + return container;
961 + }
962 +
963 + /**
964 + * Create an editable dynamic object property item
965 + */
966 + _createEditableDynamicObjectItem(parentKey, itemIndex, item, propSchema, updateCallback, itemsMap, rerenderCallback) {
967 + const container = document.createElement('div');
968 + container.className = 'mcp-dynamic-object-item';
969 +
970 + // Editable key input
971 + const keyInput = document.createElement('input');
972 + keyInput.type = 'text';
973 + keyInput.className = 'mcp-input mcp-dynamic-object-key-input';
974 + keyInput.value = item.key || '';
975 + keyInput.placeholder = 'Property name';
976 +
977 + keyInput.addEventListener('input', () => {
978 + const newKey = keyInput.value.trim();
979 +
980 + // Update the item's key
981 + item.key = newKey;
982 +
983 + // Don't trigger re-render on every keystroke
984 + });
985 +
986 + keyInput.addEventListener('blur', () => {
987 + // Trigger update when focus leaves to ensure value is saved
988 + updateCallback();
989 + });
990 +
991 + container.appendChild(keyInput);
992 +
993 + // Property value input
994 + const valueContainer = document.createElement('div');
995 + valueContainer.className = 'mcp-dynamic-object-value';
996 +
997 + // Create appropriate input based on additionalProperties schema
998 + if (propSchema.type === 'array') {
999 + // For array values, create a mini array editor
1000 + const arrayContainer = this._createEditableMiniArrayInput(
1001 + parentKey, itemIndex, item, propSchema.items || {},
1002 + updateCallback,
1003 + false // Don't re-render parent on array changes
1004 + );
1005 + valueContainer.appendChild(arrayContainer);
1006 + } else {
1007 + // For simple types, create the appropriate input
1008 + const input = this._createInput(`${parentKey}_item_${itemIndex}`, propSchema, item.value);
1009 + if (input.tagName === 'INPUT' || input.tagName === 'SELECT' || input.tagName === 'TEXTAREA') {
1010 + // Remove any existing event listeners from _createInput
1011 + const newInput = input.cloneNode(true);
1012 + input.parentNode?.replaceChild(newInput, input);
1013 +
1014 + const updateValue = () => {
1015 + if (propSchema.type === 'number' || propSchema.type === 'integer') {
1016 + item.value = newInput.value === '' ? undefined : Number(newInput.value);
1017 + } else if (propSchema.type === 'boolean') {
1018 + item.value = newInput.checked;
1019 + } else {
1020 + item.value = newInput.value || undefined;
1021 + }
1022 + };
1023 +
1024 + if (propSchema.type === 'boolean' || newInput.tagName === 'SELECT') {
1025 + // For checkbox and select, update immediately
1026 + newInput.addEventListener('change', () => {
1027 + updateValue();
1028 + updateCallback();
1029 + });
1030 + } else {
1031 + // For text inputs, update value on input but only re-render on blur
1032 + newInput.addEventListener('input', updateValue);
1033 + newInput.addEventListener('blur', () => updateCallback());
1034 + }
1035 +
1036 + valueContainer.appendChild(newInput);
1037 + } else {
1038 + valueContainer.appendChild(input);
1039 + }
1040 + }
1041 +
1042 + container.appendChild(valueContainer);
1043 +
1044 + // Remove button
1045 + const removeButton = document.createElement('button');
1046 + removeButton.type = 'button';
1047 + removeButton.className = 'mcp-button mcp-button-remove';
1048 + removeButton.innerHTML = '×';
1049 + removeButton.title = 'Remove item';
1050 + removeButton.addEventListener('click', () => {
1051 + // Remove from items Map
1052 + if (itemsMap) {
1053 + itemsMap.delete(itemIndex);
1054 + }
1055 + // Need full re-render when removing items
1056 + if (rerenderCallback) {
1057 + rerenderCallback();
1058 + }
1059 + updateCallback();
1060 + });
1061 + container.appendChild(removeButton);
1062 +
1063 + return container;
1064 + }
1065 +
1066 + /**
1067 + * Create editable mini array input for dynamic object properties
1068 + */
1069 + _createEditableMiniArrayInput(parentKey, itemIndex, item, itemSchema, updateCallback, shouldRerenderParent = true) {
1070 + const container = document.createElement('div');
1071 + container.className = 'mcp-mini-array-container';
1072 +
1073 + // Ensure item.value is an array
1074 + if (!Array.isArray(item.value)) {
1075 + item.value = [];
1076 + }
1077 +
1078 + const itemsContainer = document.createElement('div');
1079 + itemsContainer.className = 'mcp-mini-array-items';
1080 +
1081 + // Store references to avoid re-rendering
1082 + const itemElements = new Map();
1083 +
1084 + const addArrayItem = (arrayValue, index) => {
1085 + const itemEl = document.createElement('div');
1086 + itemEl.className = 'mcp-mini-array-item';
1087 +
1088 + // Create input
1089 + const input = this._createInput(`${parentKey}_item_${itemIndex}_arr_${index}`, itemSchema, arrayValue);
1090 +
1091 + if (input.tagName === 'INPUT' || input.tagName === 'SELECT' || input.tagName === 'TEXTAREA') {
1092 + // Remove any existing event listeners from _createInput
1093 + const newInput = input.cloneNode(true);
1094 +
1095 + const updateValue = () => {
1096 + if (itemSchema.type === 'number' || itemSchema.type === 'integer') {
1097 + item.value[index] = newInput.value === '' ? undefined : Number(newInput.value);
1098 + } else if (itemSchema.type === 'boolean') {
1099 + item.value[index] = newInput.checked;
1100 + } else {
1101 + item.value[index] = newInput.value || undefined;
1102 + }
1103 + };
1104 +
1105 + if (itemSchema.type === 'boolean' || newInput.tagName === 'SELECT') {
1106 + newInput.addEventListener('change', () => {
1107 + updateValue();
1108 + this._onChange();
1109 + });
1110 + } else {
1111 + newInput.addEventListener('input', updateValue);
1112 + newInput.addEventListener('blur', () => this._onChange());
1113 + }
1114 +
1115 + itemEl.appendChild(newInput);
1116 + } else {
1117 + itemEl.appendChild(input);
1118 + }
1119 +
1120 + // Remove button
1121 + const removeButton = document.createElement('button');
1122 + removeButton.type = 'button';
1123 + removeButton.className = 'mcp-button mcp-button-remove-small';
1124 + removeButton.innerHTML = '×';
1125 + removeButton.title = 'Remove item';
1126 + removeButton.addEventListener('click', (e) => {
1127 + e.preventDefault();
1128 + e.stopPropagation();
1129 +
1130 + // Remove from array
1131 + item.value.splice(index, 1);
1132 +
1133 + // Remove DOM element
1134 + itemEl.remove();
1135 +
1136 + // Update indices for remaining items
1137 + itemElements.delete(index);
1138 + const newElements = new Map();
1139 + let newIndex = 0;
1140 + itemElements.forEach((el, oldIndex) => {
1141 + if (oldIndex > index) {
1142 + newElements.set(newIndex, el);
1143 + // Update the input name attribute if needed
1144 + const input = el.querySelector('input, select, textarea');
1145 + if (input) {
1146 + input.name = `${parentKey}_item_${itemIndex}_arr_${newIndex}`;
1147 + }
1148 + } else if (oldIndex < index) {
1149 + newElements.set(newIndex, el);
1150 + }
1151 + newIndex++;
1152 + });
1153 + itemElements.clear();
1154 + newElements.forEach((el, idx) => itemElements.set(idx, el));
1155 +
1156 + this._onChange();
1157 + });
1158 +
1159 + itemEl.appendChild(removeButton);
1160 + itemsContainer.appendChild(itemEl);
1161 + itemElements.set(index, itemEl);
1162 + };
1163 +
1164 + // Initial render
1165 + item.value.forEach((arrayValue, index) => {
1166 + addArrayItem(arrayValue, index);
1167 + });
1168 +
1169 + container.appendChild(itemsContainer);
1170 +
1171 + // Add button
1172 + const addButton = document.createElement('button');
1173 + addButton.type = 'button';
1174 + addButton.className = 'mcp-button mcp-button-add-small';
1175 + addButton.textContent = '+ Add';
1176 + addButton.addEventListener('click', (e) => {
1177 + e.preventDefault();
1178 + e.stopPropagation();
1179 +
1180 + const newItem = this._getDefaultValue(itemSchema);
1181 + const newIndex = item.value.length;
1182 + item.value.push(newItem);
1183 +
1184 + // Add DOM element without re-rendering everything
1185 + addArrayItem(newItem, newIndex);
1186 +
1187 + this._onChange();
1188 + });
1189 + container.appendChild(addButton);
1190 +
1191 + return container;
1192 + }
1193 +
1194 + /**
1195 + * Create editable mini array item for dynamic object properties
1196 + */
1197 + _createEditableMiniArrayItem(parentKey, itemIndex, item, arrayItem, index, itemSchema, updateCallback) {
1198 + const container = document.createElement('div');
1199 + container.className = 'mcp-mini-array-item';
1200 +
1201 + // Create input based on schema
1202 + const input = this._createInput(`${parentKey}_item_${itemIndex}_arr_${index}`, itemSchema, arrayItem);
1203 +
1204 + if (input.tagName === 'INPUT' || input.tagName === 'SELECT' || input.tagName === 'TEXTAREA') {
1205 + // Remove any existing event listeners from _createInput
1206 + const newInput = input.cloneNode(true);
1207 + input.parentNode?.replaceChild(newInput, input);
1208 +
1209 + const updateValue = () => {
1210 + // Update the array item value
1211 + if (!Array.isArray(item.value)) {
1212 + item.value = [];
1213 + }
1214 +
1215 + if (itemSchema.type === 'number' || itemSchema.type === 'integer') {
1216 + item.value[index] = newInput.value === '' ? undefined : Number(newInput.value);
1217 + } else if (itemSchema.type === 'boolean') {
1218 + item.value[index] = newInput.checked;
1219 + } else {
1220 + item.value[index] = newInput.value || undefined;
1221 + }
1222 + };
1223 +
1224 + if (itemSchema.type === 'boolean' || newInput.tagName === 'SELECT') {
1225 + // For checkbox and select, update immediately
1226 + newInput.addEventListener('change', () => {
1227 + updateValue();
1228 + updateCallback();
1229 + });
1230 + } else {
1231 + // For text inputs, update value on input but only re-render on blur
1232 + newInput.addEventListener('input', updateValue);
1233 + newInput.addEventListener('blur', () => updateCallback());
1234 + }
1235 +
1236 + container.appendChild(newInput);
1237 + } else {
1238 + container.appendChild(input);
1239 + }
1240 +
1241 + // Remove button
1242 + const removeButton = document.createElement('button');
1243 + removeButton.type = 'button';
1244 + removeButton.className = 'mcp-button mcp-button-remove-small';
1245 + removeButton.innerHTML = '×';
1246 + removeButton.title = 'Remove item';
1247 + removeButton.addEventListener('click', (e) => {
1248 + e.preventDefault();
1249 + e.stopPropagation();
1250 + if (Array.isArray(item.value)) {
1251 + item.value.splice(index, 1);
1252 + // Just trigger local re-render, not parent update
1253 + updateCallback();
1254 + }
1255 + });
1256 + container.appendChild(removeButton);
1257 +
1258 + return container;
1259 + }
1260 +
1261 + /**
1262 + * Create mini array item
1263 + */
1264 + _createMiniArrayItem(parentKey, propKey, value, index, schema, renderCallback) {
1265 + const container = document.createElement('div');
1266 + container.className = 'mcp-mini-array-item';
1267 +
1268 + // Create input based on schema
1269 + const input = this._createInput(`${parentKey}_${propKey}_${index}`, schema, value);
1270 +
1271 + if (input.tagName === 'INPUT' || input.tagName === 'SELECT' || input.tagName === 'TEXTAREA') {
1272 + input.addEventListener('input', () => {
1273 + if (schema.type === 'number' || schema.type === 'integer') {
1274 + this.value[parentKey][propKey][index] = input.value === '' ? undefined : Number(input.value);
1275 + } else if (schema.type === 'boolean') {
1276 + this.value[parentKey][propKey][index] = input.checked;
1277 + } else {
1278 + this.value[parentKey][propKey][index] = input.value || undefined;
1279 + }
1280 + this._onChange();
1281 + });
1282 + }
1283 +
1284 + container.appendChild(input);
1285 +
1286 + // Remove button
1287 + const removeButton = document.createElement('button');
1288 + removeButton.type = 'button';
1289 + removeButton.className = 'mcp-button mcp-button-remove-small';
1290 + removeButton.innerHTML = '×';
1291 + removeButton.addEventListener('click', () => {
1292 + if (this.value[parentKey] && Array.isArray(this.value[parentKey][propKey])) {
1293 + this.value[parentKey][propKey].splice(index, 1);
1294 + renderCallback();
1295 + this._onChange();
1296 + }
1297 + });
1298 + container.appendChild(removeButton);
1299 +
1300 + return container;
1301 + }
1302 +
1303 + /**
1304 + * Create nested object input for objects with defined properties
1305 + */
1306 + _createNestedObjectInput(key, schema, value) {
1307 + const container = document.createElement('div');
1308 + container.className = 'mcp-nested-object';
1309 +
1310 + // Initialize object value
1311 + this.value[key] = value || {};
1312 +
1313 + // Render each property
1314 + for (const [propKey, propSchema] of Object.entries(schema.properties)) {
1315 + const fieldContainer = this._renderField(`${key}.${propKey}`, propSchema, this.value[key][propKey]);
1316 +
1317 + // Update parent object when nested field changes
1318 + const originalOnChange = this.options.onChange;
1319 + this.options.onChange = () => {
1320 + this.value[key][propKey] = this.value[`${key}.${propKey}`];
1321 + originalOnChange(this.getValue());
1322 + };
1323 +
1324 + container.appendChild(fieldContainer);
1325 + }
1326 +
1327 + return container;
1328 + }
1329 +
1330 + /**
1331 + * Create multi-type input for anyOf/oneOf
1332 + */
1333 + _createMultiTypeInput(key, schema, value) {
1334 + const container = document.createElement('div');
1335 + container.className = 'mcp-anyof-container';
1336 +
1337 + // Get the options
1338 + const options = schema.anyOf || schema.oneOf || [];
1339 +
1340 + // Create tabs
1341 + const tabsContainer = document.createElement('div');
1342 + tabsContainer.className = 'mcp-anyof-tabs';
1343 +
1344 + const contentContainer = document.createElement('div');
1345 + contentContainer.className = 'mcp-anyof-content';
1346 +
1347 + let activeIndex = 0;
1348 +
1349 + // Try to determine which schema matches the current value
1350 + if (value !== undefined && value !== null) {
1351 + options.forEach((opt, index) => {
1352 + if (this._valueMatchesSchema(value, opt)) {
1353 + activeIndex = index;
1354 + }
1355 + });
1356 + }
1357 +
1358 + const contents = [];
1359 +
1360 + options.forEach((optionSchema, index) => {
1361 + // Create tab
1362 + const tab = document.createElement('div');
1363 + tab.className = 'mcp-anyof-tab' + (index === activeIndex ? ' active' : '');
1364 +
1365 + // Determine tab label
1366 + let label = optionSchema.title || '';
1367 + if (!label) {
1368 + if (optionSchema.type) {
1369 + label = optionSchema.type;
1370 + // Add more context for certain types
1371 + if (optionSchema.type === 'string' && optionSchema.format) {
1372 + label += ` (${optionSchema.format})`;
1373 + } else if (optionSchema.type === 'number' && optionSchema.description) {
1374 + // Try to extract key info from description
1375 + if (optionSchema.description.includes('timestamp')) {
1376 + label = 'timestamp';
1377 + } else if (optionSchema.description.includes('relative')) {
1378 + label = 'relative';
1379 + }
1380 + }
1381 + } else if (optionSchema.enum) {
1382 + label = 'enum';
1383 + } else {
1384 + label = `Option ${index + 1}`;
1385 + }
1386 + }
1387 + tab.textContent = label;
1388 +
1389 + // Create content
1390 + const content = document.createElement('div');
1391 + content.className = 'mcp-anyof-panel' + (index === activeIndex ? ' active' : '');
1392 +
1393 + // If the option has a description, show it
1394 + if (optionSchema.description) {
1395 + const desc = document.createElement('div');
1396 + desc.className = 'mcp-anyof-description';
1397 + desc.innerHTML = optionSchema.description.replace(/\n/g, '<br/>');
1398 + content.appendChild(desc);
1399 + }
1400 +
1401 + // Create the input for this option
1402 + const optionInput = this._createInput(key + '_anyof_' + index, optionSchema, value);
1403 + content.appendChild(optionInput);
1404 +
1405 + // Handle value updates based on the input type
1406 + if (optionInput.tagName === 'INPUT' || optionInput.tagName === 'SELECT' || optionInput.tagName === 'TEXTAREA') {
1407 + const updateValue = () => {
1408 + if (optionSchema.type === 'number' || optionSchema.type === 'integer') {
1409 + this.value[key] = optionInput.value === '' ? undefined : Number(optionInput.value);
1410 + } else if (optionSchema.type === 'boolean') {
1411 + this.value[key] = optionInput.checked;
1412 + } else {
1413 + this.value[key] = optionInput.value || undefined;
1414 + }
1415 + this._onChange();
1416 + };
1417 +
1418 + optionInput.addEventListener(optionInput.type === 'checkbox' ? 'change' : 'input', updateValue);
1419 + } else if (optionSchema.type === 'array' || optionSchema.type === 'object') {
1420 + // For complex types, we need to handle updates differently
1421 + // The array/object inputs handle their own value updates
1422 + // We just need to ensure the parent value is properly set
1423 + this.value[key] = this.value[key] || this._getDefaultValue(optionSchema);
1424 + }
1425 +
1426 + // Tab click handler
1427 + tab.addEventListener('click', () => {
1428 + // Update active states
1429 + container.querySelectorAll('.mcp-anyof-tab').forEach(t => t.classList.remove('active'));
1430 + container.querySelectorAll('.mcp-anyof-panel').forEach(p => p.classList.remove('active'));
1431 + tab.classList.add('active');
1432 + content.classList.add('active');
1433 +
1434 + // Update the value based on the selected type
1435 + const input = content.querySelector('input, select, textarea');
1436 + if (input && input.value) {
1437 + input.dispatchEvent(new Event('input'));
1438 + }
1439 + });
1440 +
1441 + tabsContainer.appendChild(tab);
1442 + contentContainer.appendChild(content);
1443 + contents.push({ tab, content, input: optionInput });
1444 + });
1445 +
1446 + container.appendChild(tabsContainer);
1447 + container.appendChild(contentContainer);
1448 +
1449 + return container;
1450 + }
1451 +
1452 + /**
1453 + * Check if a value matches a schema type
1454 + */
1455 + _valueMatchesSchema(value, schema) {
1456 + if (schema.type) {
1457 + switch (schema.type) {
1458 + case 'string':
1459 + return typeof value === 'string';
1460 + case 'number':
1461 + case 'integer':
1462 + return typeof value === 'number';
1463 + case 'boolean':
1464 + return typeof value === 'boolean';
1465 + case 'array':
1466 + return Array.isArray(value);
1467 + case 'object':
1468 + return typeof value === 'object' && value !== null && !Array.isArray(value);
1469 + }
1470 + }
1471 + if (schema.enum && schema.enum.includes(value)) {
1472 + return true;
1473 + }
1474 + return false;
1475 + }
1476 +
1477 + /**
1478 + * Get default value for a schema
1479 + */
1480 + _getDefaultValue(schema) {
1481 + if (schema.default !== undefined) return schema.default;
1482 +
1483 + switch (schema.type) {
1484 + case 'string': return '';
1485 + case 'number':
1486 + case 'integer': return 0;
1487 + case 'boolean': return false;
1488 + case 'array': return [];
1489 + case 'object': return {};
1490 + default: return null;
1491 + }
1492 + }
1493 +
1494 + /**
1495 + * Validate a field value
1496 + */
1497 + _validateField(value, schema) {
1498 + if (schema.enum && !schema.enum.includes(value)) {
1499 + return `Value must be one of: ${schema.enum.join(', ')}`;
1500 + }
1501 +
1502 + switch (schema.type) {
1503 + case 'string':
1504 + if (schema.minLength && (!value || value.length < schema.minLength)) {
1505 + return `Minimum length is ${schema.minLength}`;
1506 + }
1507 + if (schema.maxLength && value && value.length > schema.maxLength) {
1508 + return `Maximum length is ${schema.maxLength}`;
1509 + }
1510 + if (schema.pattern && value && !new RegExp(schema.pattern).test(value)) {
1511 + return 'Invalid format';
1512 + }
1513 + break;
1514 + case 'number':
1515 + case 'integer':
1516 + if (value === undefined || value === null || value === '') return null;
1517 + if (schema.minimum !== undefined && value < schema.minimum) {
1518 + return `Minimum value is ${schema.minimum}`;
1519 + }
1520 + if (schema.maximum !== undefined && value > schema.maximum) {
1521 + return `Maximum value is ${schema.maximum}`;
1522 + }
1523 + if (schema.type === 'integer' && !Number.isInteger(value)) {
1524 + return 'Must be an integer';
1525 + }
1526 + break;
1527 + }
1528 +
1529 + return null;
1530 + }
1531 +
1532 + /**
1533 + * Update error display
1534 + */
1535 + _updateErrorDisplay() {
1536 + for (const [field, container] of this.fieldElements) {
1537 + const errorEl = container.querySelector('.mcp-field-error');
1538 + if (this.errors[field]) {
1539 + errorEl.textContent = this.errors[field];
1540 + errorEl.style.display = 'block';
1541 + container.classList.add('has-error');
1542 + } else {
1543 + errorEl.textContent = '';
1544 + errorEl.style.display = 'none';
1545 + container.classList.remove('has-error');
1546 + }
1547 + }
1548 + }
1549 +
1550 + /**
1551 + * Clear all validation errors
1552 + */
1553 + clearErrors() {
1554 + this.errors = {};
1555 + this._updateErrorDisplay();
1556 + }
1557 +
1558 + /**
1559 + * Update field values from current value
1560 + */
1561 + _updateFieldValues() {
1562 + // Re-render the entire form with the new values
1563 + // This is simpler than trying to update individual fields
1564 + if (this.schema) {
1565 + this.render();
1566 + }
1567 + }
1568 +
1569 + /**
1570 + * Handle value change
1571 + */
1572 + _onChange() {
1573 + // Debounce changes to avoid rapid re-renders
1574 + if (this._changeTimer) {
1575 + clearTimeout(this._changeTimer);
1576 + }
1577 +
1578 + this._changeTimer = setTimeout(() => {
1579 + // Don't validate on every change - only validate when explicitly requested
1580 + this.options.onChange(this.getValue());
1581 + }, 50); // 50ms debounce
1582 + }
1583 +
1584 + /**
1585 + * Validate that the schema conforms to MCP specification
1586 + */
1587 + _validateMCPSchema(schema) {
1588 + const errors = [];
1589 +
1590 + // Root schema validation
1591 + if (!schema || typeof schema !== 'object') {
1592 + throw new Error('MCP Schema Validation Error: Schema must be an object');
1593 + }
1594 +
1595 + // Must have type: "object"
1596 + if (schema.type !== 'object') {
1597 + errors.push('Root schema must have type: "object"');
1598 + }
1599 +
1600 + // Must have "type" property
1601 + if (!schema.hasOwnProperty('type')) {
1602 + errors.push('Root schema must have "type" property');
1603 + }
1604 +
1605 + // Validate properties
1606 + if (schema.properties) {
1607 + if (typeof schema.properties !== 'object') {
1608 + errors.push('Properties must be an object');
1609 + } else {
1610 + for (const [propName, propSchema] of Object.entries(schema.properties)) {
1611 + const propErrors = this._validateProperty(propName, propSchema, `properties.${propName}`);
1612 + errors.push(...propErrors);
1613 + }
1614 + }
1615 + }
1616 +
1617 + // Validate required array
1618 + if (schema.required !== undefined) {
1619 + if (!Array.isArray(schema.required)) {
1620 + errors.push('Required must be an array');
1621 + } else {
1622 + for (const requiredProp of schema.required) {
1623 + if (typeof requiredProp !== 'string') {
1624 + errors.push(`Required property "${requiredProp}" must be a string`);
1625 + }
1626 + if (schema.properties && !schema.properties.hasOwnProperty(requiredProp)) {
1627 + errors.push(`Required property "${requiredProp}" is not defined in properties`);
1628 + }
1629 + }
1630 + }
1631 + }
1632 +
1633 + // Check for allowed root properties
1634 + const allowedRootProps = ['type', 'properties', 'required', 'title', 'description', 'additionalProperties'];
1635 + for (const prop of Object.keys(schema)) {
1636 + if (!allowedRootProps.includes(prop)) {
1637 + errors.push(`Unknown root property: "${prop}"`);
1638 + }
1639 + }
1640 +
1641 + if (errors.length > 0) {
1642 + throw new Error(`MCP Schema Validation Errors:\n${errors.map(e => `- ${e}`).join('\n')}`);
1643 + }
1644 + }
1645 +
1646 + /**
1647 + * Validate a property schema
1648 + */
1649 + _validateProperty(propName, propSchema, path) {
1650 + const errors = [];
1651 +
1652 + if (!propSchema || typeof propSchema !== 'object') {
1653 + errors.push(`${path}: Property schema must be an object`);
1654 + return errors;
1655 + }
1656 +
1657 + // Must have a type or be anyOf/oneOf
1658 + if (!propSchema.type && !propSchema.anyOf && !propSchema.oneOf && !propSchema.enum) {
1659 + errors.push(`${path}: Property must have "type", "anyOf", "oneOf", or "enum"`);
1660 + }
1661 +
1662 + // Validate type
1663 + if (propSchema.type) {
1664 + const validTypes = ['string', 'number', 'integer', 'boolean', 'array', 'object', 'null'];
1665 + if (!validTypes.includes(propSchema.type)) {
1666 + errors.push(`${path}: Invalid type "${propSchema.type}". Must be one of: ${validTypes.join(', ')}`);
1667 + }
1668 + }
1669 +
1670 + // Type-specific validation
1671 + if (propSchema.type === 'array') {
1672 + if (propSchema.items) {
1673 + if (typeof propSchema.items !== 'object') {
1674 + errors.push(`${path}.items: Array items must be an object schema`);
1675 + } else {
1676 + const itemErrors = this._validateProperty(`${propName}[]`, propSchema.items, `${path}.items`);
1677 + errors.push(...itemErrors);
1678 + }
1679 + }
1680 + }
1681 +
1682 + if (propSchema.type === 'object') {
1683 + if (propSchema.properties) {
1684 + for (const [subPropName, subPropSchema] of Object.entries(propSchema.properties)) {
1685 + const subErrors = this._validateProperty(subPropName, subPropSchema, `${path}.properties.${subPropName}`);
1686 + errors.push(...subErrors);
1687 + }
1688 + }
1689 + }
1690 +
1691 + // Validate enum
1692 + if (propSchema.enum) {
1693 + if (!Array.isArray(propSchema.enum)) {
1694 + errors.push(`${path}.enum: Enum must be an array`);
1695 + } else if (propSchema.enum.length === 0) {
1696 + errors.push(`${path}.enum: Enum array cannot be empty`);
1697 + }
1698 + }
1699 +
1700 + // Validate anyOf/oneOf
1701 + if (propSchema.anyOf) {
1702 + if (!Array.isArray(propSchema.anyOf)) {
1703 + errors.push(`${path}.anyOf: anyOf must be an array`);
1704 + } else {
1705 + propSchema.anyOf.forEach((subSchema, index) => {
1706 + const subErrors = this._validateProperty(`${propName}[${index}]`, subSchema, `${path}.anyOf[${index}]`);
1707 + errors.push(...subErrors);
1708 + });
1709 + }
1710 + }
1711 +
1712 + if (propSchema.oneOf) {
1713 + if (!Array.isArray(propSchema.oneOf)) {
1714 + errors.push(`${path}.oneOf: oneOf must be an array`);
1715 + } else {
1716 + propSchema.oneOf.forEach((subSchema, index) => {
1717 + const subErrors = this._validateProperty(`${propName}[${index}]`, subSchema, `${path}.oneOf[${index}]`);
1718 + errors.push(...subErrors);
1719 + });
1720 + }
1721 + }
1722 +
1723 + // Check for allowed property keywords
1724 + const allowedProps = [
1725 + // Core keywords
1726 + 'type', 'enum', 'const',
1727 + // String keywords
1728 + 'minLength', 'maxLength', 'pattern', 'format',
1729 + // Number keywords
1730 + 'minimum', 'maximum', 'exclusiveMinimum', 'exclusiveMaximum', 'multipleOf',
1731 + // Array keywords
1732 + 'items', 'minItems', 'maxItems', 'uniqueItems',
1733 + // Object keywords
1734 + 'properties', 'required', 'additionalProperties', 'patternProperties',
1735 + // Composition keywords
1736 + 'anyOf', 'oneOf', 'allOf', 'not',
1737 + // Metadata keywords
1738 + 'title', 'description', 'default', 'examples',
1739 + // Additional validation
1740 + 'if', 'then', 'else'
1741 + ];
1742 +
1743 + for (const prop of Object.keys(propSchema)) {
1744 + if (!allowedProps.includes(prop)) {
1745 + errors.push(`${path}: Unknown property keyword: "${prop}"`);
1746 + }
1747 + }
1748 +
1749 + // Validate default value matches type
1750 + if (propSchema.default !== undefined) {
1751 + const defaultError = this._validateDefaultValue(propSchema.default, propSchema, path);
1752 + if (defaultError) {
1753 + errors.push(defaultError);
1754 + }
1755 + }
1756 +
1757 + // Validate string constraints
1758 + if (propSchema.type === 'string') {
1759 + if (propSchema.minLength !== undefined && (typeof propSchema.minLength !== 'number' || propSchema.minLength < 0)) {
1760 + errors.push(`${path}.minLength: Must be a non-negative number`);
1761 + }
1762 + if (propSchema.maxLength !== undefined && (typeof propSchema.maxLength !== 'number' || propSchema.maxLength < 0)) {
1763 + errors.push(`${path}.maxLength: Must be a non-negative number`);
1764 + }
1765 + if (propSchema.pattern !== undefined && typeof propSchema.pattern !== 'string') {
1766 + errors.push(`${path}.pattern: Must be a string`);
1767 + }
1768 + }
1769 +
1770 + // Validate number constraints
1771 + if (propSchema.type === 'number' || propSchema.type === 'integer') {
1772 + if (propSchema.minimum !== undefined && typeof propSchema.minimum !== 'number') {
1773 + errors.push(`${path}.minimum: Must be a number`);
1774 + }
1775 + if (propSchema.maximum !== undefined && typeof propSchema.maximum !== 'number') {
1776 + errors.push(`${path}.maximum: Must be a number`);
1777 + }
1778 + if (propSchema.multipleOf !== undefined && (typeof propSchema.multipleOf !== 'number' || propSchema.multipleOf <= 0)) {
1779 + errors.push(`${path}.multipleOf: Must be a positive number`);
1780 + }
1781 + }
1782 +
1783 + // Validate array constraints
1784 + if (propSchema.type === 'array') {
1785 + if (propSchema.minItems !== undefined && (typeof propSchema.minItems !== 'number' || propSchema.minItems < 0)) {
1786 + errors.push(`${path}.minItems: Must be a non-negative number`);
1787 + }
1788 + if (propSchema.maxItems !== undefined && (typeof propSchema.maxItems !== 'number' || propSchema.maxItems < 0)) {
1789 + errors.push(`${path}.maxItems: Must be a non-negative number`);
1790 + }
1791 + }
1792 +
1793 + return errors;
1794 + }
1795 +
1796 + /**
1797 + * Validate that a default value matches the schema type
1798 + */
1799 + _validateDefaultValue(defaultValue, schema, path) {
1800 + // Handle null
1801 + if (defaultValue === null) {
1802 + if (schema.type === 'null' || (Array.isArray(schema.type) && schema.type.includes('null'))) {
1803 + return null;
1804 + }
1805 + return `${path}.default: Default value null is not valid for type ${schema.type}`;
1806 + }
1807 +
1808 + // Handle enum
1809 + if (schema.enum) {
1810 + if (!schema.enum.includes(defaultValue)) {
1811 + return `${path}.default: Default value must be one of enum values: ${schema.enum.join(', ')}`;
1812 + }
1813 + return null;
1814 + }
1815 +
1816 + // Validate based on type
1817 + switch (schema.type) {
1818 + case 'string':
1819 + if (typeof defaultValue !== 'string') {
1820 + return `${path}.default: Default value must be a string, got ${typeof defaultValue}`;
1821 + }
1822 + // Check string constraints
1823 + if (schema.minLength !== undefined && defaultValue.length < schema.minLength) {
1824 + return `${path}.default: Default value length ${defaultValue.length} is less than minLength ${schema.minLength}`;
1825 + }
1826 + if (schema.maxLength !== undefined && defaultValue.length > schema.maxLength) {
1827 + return `${path}.default: Default value length ${defaultValue.length} exceeds maxLength ${schema.maxLength}`;
1828 + }
1829 + if (schema.pattern && !new RegExp(schema.pattern).test(defaultValue)) {
1830 + return `${path}.default: Default value does not match pattern ${schema.pattern}`;
1831 + }
1832 + break;
1833 +
1834 + case 'number':
1835 + if (typeof defaultValue !== 'number' || isNaN(defaultValue)) {
1836 + return `${path}.default: Default value must be a number, got ${typeof defaultValue}`;
1837 + }
1838 + // Check number constraints
1839 + if (schema.minimum !== undefined && defaultValue < schema.minimum) {
1840 + return `${path}.default: Default value ${defaultValue} is less than minimum ${schema.minimum}`;
1841 + }
1842 + if (schema.maximum !== undefined && defaultValue > schema.maximum) {
1843 + return `${path}.default: Default value ${defaultValue} exceeds maximum ${schema.maximum}`;
1844 + }
1845 + if (schema.multipleOf !== undefined && defaultValue % schema.multipleOf !== 0) {
1846 + return `${path}.default: Default value ${defaultValue} is not a multiple of ${schema.multipleOf}`;
1847 + }
1848 + break;
1849 +
1850 + case 'integer':
1851 + if (typeof defaultValue !== 'number' || !Number.isInteger(defaultValue)) {
1852 + return `${path}.default: Default value must be an integer, got ${typeof defaultValue}`;
1853 + }
1854 + // Check integer constraints (same as number)
1855 + if (schema.minimum !== undefined && defaultValue < schema.minimum) {
1856 + return `${path}.default: Default value ${defaultValue} is less than minimum ${schema.minimum}`;
1857 + }
1858 + if (schema.maximum !== undefined && defaultValue > schema.maximum) {
1859 + return `${path}.default: Default value ${defaultValue} exceeds maximum ${schema.maximum}`;
1860 + }
1861 + break;
1862 +
1863 + case 'boolean':
1864 + if (typeof defaultValue !== 'boolean') {
1865 + return `${path}.default: Default value must be a boolean, got ${typeof defaultValue}`;
1866 + }
1867 + break;
1868 +
1869 + case 'array':
1870 + if (!Array.isArray(defaultValue)) {
1871 + return `${path}.default: Default value must be an array, got ${typeof defaultValue}`;
1872 + }
1873 + // Check array constraints
1874 + if (schema.minItems !== undefined && defaultValue.length < schema.minItems) {
1875 + return `${path}.default: Default array length ${defaultValue.length} is less than minItems ${schema.minItems}`;
1876 + }
1877 + if (schema.maxItems !== undefined && defaultValue.length > schema.maxItems) {
1878 + return `${path}.default: Default array length ${defaultValue.length} exceeds maxItems ${schema.maxItems}`;
1879 + }
1880 + // Could validate items too, but that might be too complex for now
1881 + break;
1882 +
1883 + case 'object':
1884 + if (typeof defaultValue !== 'object' || defaultValue === null || Array.isArray(defaultValue)) {
1885 + return `${path}.default: Default value must be an object, got ${typeof defaultValue}`;
1886 + }
1887 + // Could validate properties too, but that might be too complex for now
1888 + break;
1889 +
1890 + case 'null':
1891 + if (defaultValue !== null) {
1892 + return `${path}.default: Default value must be null, got ${typeof defaultValue}`;
1893 + }
1894 + break;
1895 + }
1896 +
1897 + return null;
1898 + }
1899 +
1900 + /**
1901 + * Deep clone an object
1902 + */
1903 + _deepClone(obj) {
1904 + return JSON.parse(JSON.stringify(obj));
1905 + }
1906 +
1907 + /**
1908 + * Inject CSS styles
1909 + */
1910 + _injectStyles() {
1911 + if (document.getElementById('mcp-schema-ui-styles')) return;
1912 +
1913 + const style = document.createElement('style');
1914 + style.id = 'mcp-schema-ui-styles';
1915 + style.textContent = `
1916 + .mcp-form {
1917 + font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
1918 + font-size: 14px;
1919 + line-height: 1.5;
1920 + }
1921 +
1922 + .mcp-form-title {
1923 + margin: 0 0 12px 0;
1924 + font-size: 16px;
1925 + font-weight: 600;
1926 + color: #333;
1927 + }
1928 +
1929 + .mcp-form-description {
1930 + margin: 0 0 16px 0;
1931 + color: #666;
1932 + font-size: 13px;
1933 + }
1934 +
1935 + .mcp-field {
1936 + margin-bottom: 16px;
1937 + }
1938 +
1939 + .mcp-field-label-wrapper {
1940 + display: flex;
1941 + align-items: center;
1942 + gap: 6px;
1943 + margin-bottom: 6px;
1944 + justify-content: space-between;
1945 + }
1946 +
1947 + .mcp-field-label {
1948 + font-weight: 500;
1949 + color: #333;
1950 + display: flex;
1951 + align-items: center;
1952 + gap: 4px;
1953 + }
1954 +
1955 + .mcp-field-label-hoverable {
1956 + cursor: help;
1957 + position: relative;
1958 + }
1959 +
1960 + .mcp-field-label-hoverable:hover {
1961 + color: #0088cc;
1962 + text-decoration: underline;
1963 + text-decoration-style: dotted;
1964 + text-underline-offset: 2px;
1965 + }
1966 +
1967 + .mcp-field-required {
1968 + color: #dc3545;
1969 + font-weight: normal;
1970 + }
1971 +
1972 + .mcp-field-json-name {
1973 + font-family: 'Courier New', monospace;
1974 + font-size: 12px;
1975 + color: #6c757d;
1976 + margin-left: auto;
1977 + }
1978 +
1979 + .mcp-tooltip-popup {
1980 + background: rgba(0, 0, 0, 0.9);
1981 + color: white;
1982 + padding: 8px 12px;
1983 + border-radius: 4px;
1984 + font-size: 12px;
1985 + line-height: 1.4;
1986 + white-space: pre-wrap;
1987 + max-width: 300px;
1988 + pointer-events: none;
1989 + z-index: 10000;
1990 + box-shadow: 0 2px 8px rgba(0, 0, 0, 0.2);
1991 + animation: mcp-tooltip-fade-in 0.2s ease-out;
1992 + }
1993 +
1994 + @keyframes mcp-tooltip-fade-in {
1995 + from {
1996 + opacity: 0;
1997 + transform: translateY(-95%);
1998 + }
1999 + to {
2000 + opacity: 1;
2001 + transform: translateY(-100%);
2002 + }
2003 + }
2004 +
2005 + .mcp-field-description {
2006 + font-size: 12px;
2007 + color: #6c757d;
2008 + margin-bottom: 6px;
2009 + }
2010 +
2011 + .mcp-field-error {
2012 + color: #dc3545;
2013 + font-size: 12px;
2014 + margin-top: 4px;
2015 + display: none;
2016 + }
2017 +
2018 + .mcp-field.has-error .mcp-input,
2019 + .mcp-field.has-error .mcp-select,
2020 + .mcp-field.has-error .mcp-textarea {
2021 + border-color: #dc3545;
2022 + }
2023 +
2024 + .mcp-input,
2025 + .mcp-select,
2026 + .mcp-textarea {
2027 + width: 100%;
2028 + padding: 6px 10px;
2029 + border: 1px solid #ced4da;
2030 + border-radius: 4px;
2031 + font-size: 14px;
2032 + transition: border-color 0.15s ease-in-out;
2033 + }
2034 +
2035 + .mcp-input:focus,
2036 + .mcp-select:focus,
2037 + .mcp-textarea:focus {
2038 + outline: none;
2039 + border-color: #80bdff;
2040 + box-shadow: 0 0 0 2px rgba(0, 123, 255, 0.1);
2041 + }
2042 +
2043 + .mcp-input-error {
2044 + border-color: #dc3545 !important;
2045 + }
2046 +
2047 + .mcp-checkbox-wrapper {
2048 + display: flex;
2049 + align-items: center;
2050 + gap: 8px;
2051 + cursor: pointer;
2052 + }
2053 +
2054 + .mcp-checkbox {
2055 + width: 16px;
2056 + height: 16px;
2057 + cursor: pointer;
2058 + }
2059 +
2060 + .mcp-array-container {
2061 + border: 1px solid #dee2e6;
2062 + border-radius: 4px;
2063 + padding: 12px;
2064 + background: #f8f9fa;
2065 + }
2066 +
2067 + .mcp-array-items {
2068 + margin-bottom: 8px;
2069 + }
2070 +
2071 + .mcp-array-item {
2072 + display: flex;
2073 + gap: 8px;
2074 + margin-bottom: 8px;
2075 + align-items: flex-start;
2076 + }
2077 +
2078 + .mcp-array-item-input {
2079 + flex: 1;
2080 + }
2081 +
2082 + .mcp-button {
2083 + padding: 6px 12px;
2084 + border: 1px solid #ced4da;
2085 + border-radius: 4px;
2086 + background: white;
2087 + font-size: 13px;
2088 + cursor: pointer;
2089 + transition: all 0.15s ease;
2090 + }
2091 +
2092 + .mcp-button:hover {
2093 + background: #f8f9fa;
2094 + border-color: #adb5bd;
2095 + }
2096 +
2097 + .mcp-button-add {
2098 + color: #28a745;
2099 + border-color: #28a745;
2100 + }
2101 +
2102 + .mcp-button-add:hover {
2103 + background: #28a745;
2104 + color: white;
2105 + }
2106 +
2107 + .mcp-button-remove {
2108 + width: 28px;
2109 + height: 28px;
2110 + padding: 0;
2111 + color: #dc3545;
2112 + border-color: #dc3545;
2113 + font-size: 18px;
2114 + line-height: 1;
2115 + }
2116 +
2117 + .mcp-button-remove:hover {
2118 + background: #dc3545;
2119 + color: white;
2120 + }
2121 +
2122 + /* AnyOf/OneOf tabs */
2123 + .mcp-anyof-container {
2124 + border: 1px solid #dee2e6;
2125 + border-radius: 4px;
2126 + overflow: hidden;
2127 + }
2128 +
2129 + .mcp-anyof-tabs {
2130 + display: flex;
2131 + background: #f8f9fa;
2132 + border-bottom: 1px solid #dee2e6;
2133 + }
2134 +
2135 + .mcp-anyof-tab {
2136 + padding: 8px 16px;
2137 + cursor: pointer;
2138 + font-size: 13px;
2139 + color: #666;
2140 + border-right: 1px solid #dee2e6;
2141 + transition: all 0.2s;
2142 + background: #f8f9fa;
2143 + }
2144 +
2145 + .mcp-anyof-tab:last-child {
2146 + border-right: none;
2147 + }
2148 +
2149 + .mcp-anyof-tab:hover {
2150 + background: #e9ecef;
2151 + color: #333;
2152 + }
2153 +
2154 + .mcp-anyof-tab.active {
2155 + background: white;
2156 + color: #0088cc;
2157 + font-weight: 600;
2158 + border-bottom: 2px solid white;
2159 + margin-bottom: -1px;
2160 + }
2161 +
2162 + .mcp-anyof-content {
2163 + background: white;
2164 + min-height: 60px;
2165 + }
2166 +
2167 + .mcp-anyof-panel {
2168 + display: none;
2169 + padding: 12px;
2170 + }
2171 +
2172 + .mcp-anyof-panel.active {
2173 + display: block;
2174 + }
2175 +
2176 + .mcp-anyof-description {
2177 + font-size: 12px;
2178 + color: #666;
2179 + margin-bottom: 10px;
2180 + line-height: 1.4;
2181 + background: #f8f9fa;
2182 + padding: 8px 10px;
2183 + border-radius: 4px;
2184 + }
2185 +
2186 + /* Dynamic object styles */
2187 + .mcp-dynamic-object-container {
2188 + border: 1px solid #dee2e6;
2189 + border-radius: 4px;
2190 + padding: 8px;
2191 + background: #f8f9fa;
2192 + }
2193 +
2194 + .mcp-dynamic-object-items {
2195 + margin-bottom: 8px;
2196 + }
2197 +
2198 + .mcp-dynamic-object-item {
2199 + display: grid;
2200 + grid-template-columns: 150px 1fr auto;
2201 + gap: 8px;
2202 + align-items: start;
2203 + padding: 8px;
2204 + background: white;
2205 + border: 1px solid #dee2e6;
2206 + border-radius: 4px;
2207 + margin-bottom: 6px;
2208 + }
2209 +
2210 + .mcp-dynamic-object-key {
2211 + font-weight: 600;
2212 + color: #333;
2213 + padding: 6px 0;
2214 + word-break: break-word;
2215 + }
2216 +
2217 + .mcp-dynamic-object-value {
2218 + min-width: 0;
2219 + }
2220 +
2221 + .mcp-dynamic-object-add {
2222 + display: flex;
2223 + gap: 8px;
2224 + padding: 8px;
2225 + background: white;
2226 + border: 1px solid #dee2e6;
2227 + border-radius: 4px;
2228 + }
2229 +
2230 + .mcp-dynamic-key-input {
2231 + flex: 1;
2232 + max-width: 200px;
2233 + }
2234 +
2235 + /* Mini array styles for dynamic objects */
2236 + .mcp-mini-array-container {
2237 + background: #f8f9fa;
2238 + padding: 6px;
2239 + border-radius: 4px;
2240 + border: 1px solid #e9ecef;
2241 + }
2242 +
2243 + .mcp-mini-array-items {
2244 + display: flex;
2245 + flex-direction: column;
2246 + gap: 4px;
2247 + margin-bottom: 6px;
2248 + }
2249 +
2250 + .mcp-mini-array-item {
2251 + display: flex;
2252 + gap: 6px;
2253 + align-items: center;
2254 + }
2255 +
2256 + .mcp-mini-array-item input,
2257 + .mcp-mini-array-item select {
2258 + flex: 1;
2259 + min-width: 0;
2260 + }
2261 +
2262 + .mcp-button-add-small,
2263 + .mcp-button-remove-small {
2264 + padding: 2px 8px;
2265 + font-size: 12px;
2266 + line-height: 1.5;
2267 + white-space: nowrap;
2268 + min-width: fit-content;
2269 + }
2270 +
2271 + .mcp-button-add-small {
2272 + color: #28a745;
2273 + border-color: #28a745;
2274 + }
2275 +
2276 + .mcp-button-add-small:hover {
2277 + background: #28a745;
2278 + color: white;
2279 + }
2280 +
2281 + .mcp-button-remove-small {
2282 + width: 24px;
2283 + height: 24px;
2284 + padding: 0;
2285 + color: #dc3545;
2286 + border-color: #dc3545;
2287 + }
2288 +
2289 + .mcp-button-remove-small:hover {
2290 + background: #dc3545;
2291 + color: white;
2292 + }
2293 +
2294 + /* Array enum checkbox styles */
2295 + .mcp-array-enum-container {
2296 + display: flex;
2297 + flex-direction: column;
2298 + gap: 8px;
2299 + padding: 8px;
2300 + background: #f8f9fa;
2301 + border: 1px solid #dee2e6;
2302 + border-radius: 4px;
2303 + }
2304 +
2305 + .mcp-array-enum-item {
2306 + display: flex;
2307 + align-items: center;
2308 + gap: 8px;
2309 + cursor: pointer;
2310 + padding: 4px 0;
2311 + }
2312 +
2313 + .mcp-array-enum-item:hover {
2314 + background: rgba(0, 123, 255, 0.05);
2315 + border-radius: 3px;
2316 + margin: 0 -4px;
2317 + padding: 4px;
2318 + }
2319 +
2320 + .mcp-array-enum-item input[type="checkbox"] {
2321 + width: 16px;
2322 + height: 16px;
2323 + cursor: pointer;
2324 + margin: 0;
2325 + }
2326 +
2327 + .mcp-array-enum-item span {
2328 + user-select: none;
2329 + color: #333;
2330 + font-size: 14px;
2331 + }
2332 +
2333 + /* Nested object styles */
2334 + .mcp-nested-object {
2335 + padding: 12px;
2336 + background: #f8f9fa;
2337 + border: 1px solid #dee2e6;
2338 + border-radius: 4px;
2339 + }
2340 +
2341 + .mcp-nested-object .mcp-field {
2342 + margin-bottom: 12px;
2343 + }
2344 +
2345 + .mcp-nested-object .mcp-field:last-child {
2346 + margin-bottom: 0;
2347 + }
2348 +
2349 + /* Tuple array styles */
2350 + .mcp-tuple-array-container {
2351 + background: #f0f0f0;
2352 + }
2353 +
2354 + .mcp-tuple-item {
2355 + background: white;
2356 + border: 1px solid #ddd;
2357 + border-radius: 4px;
2358 + padding: 8px;
2359 + }
2360 +
2361 + .mcp-tuple-fields {
2362 + display: flex;
2363 + gap: 12px;
2364 + align-items: flex-start;
2365 + flex: 1;
2366 + }
2367 +
2368 + .mcp-tuple-field {
2369 + flex: 1;
2370 + min-width: 0;
2371 + }
2372 +
2373 + .mcp-tuple-field:first-child {
2374 + flex: 1.6; /* Column name - 20% narrower than before */
2375 + }
2376 +
2377 + .mcp-tuple-field:nth-child(2) {
2378 + flex: 0.75; /* Operator - half width */
2379 + }
2380 +
2381 + .mcp-tuple-field:nth-child(3) {
2382 + flex: 2; /* Value */
2383 + }
2384 +
2385 + .mcp-tuple-field-label {
2386 + font-size: 11px;
2387 + color: #666;
2388 + margin-bottom: 4px;
2389 + font-weight: 500;
2390 + text-transform: uppercase;
2391 + letter-spacing: 0.5px;
2392 + }
2393 +
2394 + .mcp-tuple-field input,
2395 + .mcp-tuple-field select {
2396 + width: 100%;
2397 + box-sizing: border-box;
2398 + }
2399 +
2400 + .mcp-array-item.mcp-tuple-item {
2401 + display: flex;
2402 + gap: 8px;
2403 + align-items: center;
2404 + }
2405 +
2406 + .mcp-array-item.mcp-tuple-item .mcp-button-remove {
2407 + flex-shrink: 0;
2408 + align-self: flex-start;
2409 + margin-top: 20px; /* Align with inputs */
2410 + }
2411 +
2412 + /* Tuple multi-type input styles */
2413 + .mcp-tuple-multitype-container {
2414 + display: flex;
2415 + flex-direction: column;
2416 + gap: 4px;
2417 + }
2418 +
2419 + .mcp-tuple-anyof-tabs {
2420 + display: flex;
2421 + gap: 2px;
2422 + background: #f0f0f0;
2423 + padding: 2px;
2424 + border-radius: 4px;
2425 + }
2426 +
2427 + .mcp-tuple-anyof-tabs .mcp-anyof-tab {
2428 + padding: 2px 8px;
2429 + font-size: 11px;
2430 + border: none;
2431 + background: transparent;
2432 + color: #666;
2433 + cursor: pointer;
2434 + border-radius: 3px;
2435 + transition: all 0.15s ease;
2436 + }
2437 +
2438 + .mcp-tuple-anyof-tabs .mcp-anyof-tab:hover {
2439 + background: #e0e0e0;
2440 + }
2441 +
2442 + .mcp-tuple-anyof-tabs .mcp-anyof-tab.active {
2443 + background: white;
2444 + color: #333;
2445 + font-weight: 500;
2446 + box-shadow: 0 1px 2px rgba(0,0,0,0.1);
2447 + }
2448 +
2449 + .mcp-tuple-multitype-input {
2450 + min-height: 32px;
2451 + display: flex;
2452 + align-items: center;
2453 + }
2454 +
2455 + .mcp-tuple-multitype-input input[type="checkbox"] {
2456 + margin: 0;
2457 + }
2458 + `;
2459 +
2460 + document.head.appendChild(style);
2461 + }
2462 +}
2463 +
2464 +// Export for use
2465 +if (typeof module !== 'undefined' && module.exports) {
2466 + module.exports = MCPSchemaUIGenerator;
2467 +}
\ No newline at end of file
src/web/mcp/mcp-tools-alert-transitions.c new
+349
@@ -0,0 +1,349 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-tools-alert-transitions.h"
4 +#include "mcp-params.h"
5 +#include "database/contexts/rrdcontext.h"
6 +
7 +// Schema for alert transitions
8 +void mcp_tool_list_alert_transitions_schema(BUFFER *buffer) {
9 + // Tool metadata
10 + buffer_json_member_add_object(buffer, "inputSchema");
11 + buffer_json_member_add_string(buffer, "type", "object");
12 + buffer_json_member_add_string(buffer, "title", "List alert transitions");
13 +
14 + buffer_json_member_add_object(buffer, "properties");
15 +
16 + // Time range
17 + mcp_schema_add_time_params(
18 + buffer,
19 + "alert transitions",
20 + false);
21 +
22 + mcp_schema_add_array_param(
23 + buffer, "alerts",
24 + "Filter by alert names",
25 + "Array of specific alert names to filter by. "
26 + "Each alert name must be an exact match - no wildcards or patterns allowed. "
27 + "Use '" MCP_TOOL_LIST_ALL_ALERTS "' to discover available alert names. "
28 + "If not specified, all alerts are included. "
29 + "Examples: [\"disk_space_usage\", \"cpu_iowait\", \"ram_in_use\"]");
30 +
31 + // Nodes filter
32 + mcp_schema_add_array_param(
33 + buffer, "nodes", "Filter nodes",
34 + "Show only alerts transitions for these nodes.\n"
35 + "Use 'list_nodes' to discover available nodes.\n"
36 + "If not specified, alerts transitions from all nodes are included. "
37 + "Examples: [\"node1\", \"node2\"], [\"web-server-01\", \"db-server-01\"]");
38 +
39 + mcp_schema_add_array_param(
40 + buffer, "metrics",
41 + "Filter by metrics",
42 + "Array of specific metric names to filter by. "
43 + "Each metric must be an exact match - no wildcards or patterns allowed. "
44 + "Use '" MCP_TOOL_LIST_METRICS "' to discover available metrics. "
45 + "If not specified, all metrics are included. "
46 + "Examples: [\"system.cpu\", \"system.load\"], [\"disk.io\", \"disk.space\"]");
47 +
48 + mcp_schema_add_array_param(
49 + buffer, "instances",
50 + "Filter by instances",
51 + "Query only the given instances.\n"
52 + "Use the '" MCP_TOOL_GET_METRICS_DETAILS "' tool to discover available instances for a metric.\n"
53 + "If no instances are specified, all instances of the metric are queried.\n"
54 + "Example: [\"instance1\", \"instance2\", \"instance3\"]\n."
55 + "IMPORTANT: when you have a choice, prefer to filter by labels instead of instances, because many monitored "
56 + "components may change instance names over time.");
57 +
58 + // Status filter (required multi-select enum)
59 + buffer_json_member_add_object(buffer, "status");
60 + {
61 + buffer_json_member_add_string(buffer, "type", "array");
62 + buffer_json_member_add_string(buffer, "title", "Filter by status");
63 + buffer_json_member_add_string(
64 + buffer, "description",
65 + "Select the alert statuses of interest. At least one status must be selected.\n"
66 + " - CRITICAL: the highest severity, indicates a critical issue that needs immediate attention.\n"
67 + " - WARNING: indicates a potential issue that should be monitored but is not critical.\n"
68 + " - CLEAR: the normal state state for alerts, indicating that the alert is not triggered.\n"
69 + " - UNDEFINED: the alerts failed to be evaluated (some variable of it is undefined, division by zero, etc).\n"
70 + " - UNINITIALIZED: the alert has not been initialized for the first time yet, no data available.\n"
71 + " - REMOVED: the alert was removed (happens during netdata shutdown, child disconnect, health reload).\n"
72 + "Multiple statuses can be selected. Example: [\"CRITICAL\", \"WARNING\"]");
73 +
74 + // Define items schema with enum values
75 + buffer_json_member_add_object(buffer, "items");
76 + {
77 + buffer_json_member_add_string(buffer, "type", "string");
78 + buffer_json_member_add_array(buffer, "enum");
79 + buffer_json_add_array_item_string(buffer, "CRITICAL");
80 + buffer_json_add_array_item_string(buffer, "WARNING");
81 + buffer_json_add_array_item_string(buffer, "CLEAR");
82 + buffer_json_add_array_item_string(buffer, "UNDEFINED");
83 + buffer_json_add_array_item_string(buffer, "UNINITIALIZED");
84 + buffer_json_add_array_item_string(buffer, "REMOVED");
85 + buffer_json_array_close(buffer);
86 + }
87 + buffer_json_object_close(buffer); // items
88 + }
89 + buffer_json_object_close(buffer); // status
90 +
91 + mcp_schema_add_array_param(
92 + buffer, "classifications",
93 + "Filter by classifications",
94 + "Array of specific alert classifications to filter by. "
95 + "Each classification must be an exact match - no wildcards or patterns allowed. "
96 + "Use '" MCP_TOOL_LIST_ALL_ALERTS "' to discover available classifications. "
97 + "If not specified, all classifications are included. "
98 + "Examples: [\"Errors\", \"Latency\", \"Utilization\"]");
99 +
100 + mcp_schema_add_array_param(
101 + buffer, "types",
102 + "Filter by types",
103 + "Array of specific alert types to filter by. "
104 + "Each type must be an exact match - no wildcards or patterns allowed. "
105 + "Use '" MCP_TOOL_LIST_ALL_ALERTS "' to discover available types. "
106 + "If not specified, all types are included. "
107 + "Examples: [\"System\", \"Web Server\", \"Database\"]");
108 +
109 + mcp_schema_add_array_param(
110 + buffer, "components",
111 + "Filter by components",
112 + "Array of specific components to filter by. "
113 + "Each component must be an exact match - no wildcards or patterns allowed. "
114 + "Use '" MCP_TOOL_LIST_ALL_ALERTS "' to discover available components. "
115 + "If not specified, all components are included. "
116 + "Examples: [\"Network\", \"Disk\", \"Memory\"]");
117 +
118 + mcp_schema_add_array_param(
119 + buffer, "roles",
120 + "Filter by roles",
121 + "Array of specific roles to filter by. "
122 + "Each role must be an exact match - no wildcards or patterns allowed. "
123 + "Use '" MCP_TOOL_LIST_ALL_ALERTS "' to discover available roles. "
124 + "If not specified, all roles are included. "
125 + "Examples: [\"sysadmin\", \"webmaster\", \"dba\"]");
126 +
127 + // Cardinality limit
128 + mcp_schema_add_cardinality_limit(
129 + buffer,
130 + "Number of most recent alert transitions to return",
131 + MCP_ALERTS_CARDINALITY_LIMIT, // default value
132 + 1, // minimum
133 + MAX(MCP_ALERTS_CARDINALITY_LIMIT, MCP_ALERTS_CARDINALITY_LIMIT_MAX)); // max value
134 +
135 + // Pagination cursor
136 + mcp_schema_add_string_param(buffer, "cursor",
137 + "Pagination cursor",
138 + "Pagination cursor from previous response. Use the 'nextCursor' value from the previous response to get the next page of results.",
139 + NULL, false);
140 +
141 + // Timeout parameter
142 + mcp_schema_add_timeout(buffer, "timeout",
143 + "Query timeout",
144 + "Maximum time to wait for the query to complete (in seconds)",
145 + 60, 1, 3600, false);
146 +
147 + buffer_json_object_close(buffer); // properties
148 +
149 + // Required fields
150 + buffer_json_member_add_array(buffer, "required");
151 + buffer_json_add_array_item_string(buffer, "status");
152 + buffer_json_array_close(buffer);
153 +
154 + buffer_json_object_close(buffer); // inputSchema
155 +}
156 +
157 +// Execute alert transitions query
158 +MCP_RETURN_CODE mcp_tool_list_alert_transitions_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
159 + if (!mcpc || id == 0)
160 + return MCP_RC_ERROR;
161 +
162 + // Extract nodes array
163 + const char *nodes_pattern = NULL;
164 + CLEAN_BUFFER *nodes_buffer = NULL;
165 +
166 + nodes_buffer = mcp_params_parse_array_to_pattern(params, "nodes", false, false, MCP_TOOL_LIST_NODES, mcpc->error);
167 + if (buffer_strlen(mcpc->error) > 0) {
168 + return MCP_RC_BAD_REQUEST;
169 + }
170 + if (nodes_buffer)
171 + nodes_pattern = buffer_tostring(nodes_buffer);
172 +
173 + // Extract time parameters
174 + time_t after = mcp_params_parse_time(params, "after", MCP_DEFAULT_AFTER_TIME);
175 + time_t before = mcp_params_parse_time(params, "before", MCP_DEFAULT_BEFORE_TIME);
176 +
177 + // Extract cardinality limit
178 + size_t cardinality_limit = mcp_params_extract_size(params, "cardinality_limit", 1, 1, 100, mcpc->error);
179 + if (buffer_strlen(mcpc->error) > 0) {
180 + return MCP_RC_BAD_REQUEST;
181 + }
182 +
183 + // Extract pagination cursor (global_id_anchor)
184 + usec_t global_id_anchor = 0;
185 + const char *cursor = mcp_params_extract_string(params, "cursor", NULL);
186 + if (cursor) {
187 + // Parse cursor as usec_t
188 + char *endptr;
189 + unsigned long long value = strtoull(cursor, &endptr, 10);
190 + if (*endptr != '\0' || endptr == cursor) {
191 + buffer_sprintf(mcpc->error, "Invalid cursor value");
192 + return MCP_RC_BAD_REQUEST;
193 + }
194 + global_id_anchor = (usec_t)value;
195 + }
196 +
197 + // Extract status array (required parameter)
198 + CLEAN_BUFFER *status_buffer = NULL;
199 + const char *status_pattern = NULL;
200 +
201 + status_buffer = mcp_params_parse_array_to_pattern(params, "status", true, false, NULL, mcpc->error);
202 + if (buffer_strlen(mcpc->error) > 0) {
203 + buffer_strcat(mcpc->error, ". You must select at least one alert status to filter by.");
204 + return MCP_RC_BAD_REQUEST;
205 + }
206 + if (status_buffer)
207 + status_pattern = buffer_tostring(status_buffer);
208 +
209 + // Extract instances array
210 + const char *instances_pattern = NULL;
211 + CLEAN_BUFFER *instances_buffer = NULL;
212 +
213 + instances_buffer = mcp_params_parse_array_to_pattern(params, "instances", false, false, NULL, mcpc->error);
214 + if (buffer_strlen(mcpc->error) > 0) {
215 + return MCP_RC_BAD_REQUEST;
216 + }
217 + if (instances_buffer)
218 + instances_pattern = buffer_tostring(instances_buffer);
219 +
220 + // Extract metrics array
221 + const char *metrics_pattern = NULL;
222 + CLEAN_BUFFER *metrics_buffer = NULL;
223 +
224 + metrics_buffer = mcp_params_parse_array_to_pattern(params, "metrics", false, false, MCP_TOOL_LIST_METRICS, mcpc->error);
225 + if (buffer_strlen(mcpc->error) > 0) {
226 + return MCP_RC_BAD_REQUEST;
227 + }
228 + if (metrics_buffer)
229 + metrics_pattern = buffer_tostring(metrics_buffer);
230 +
231 + // Extract alerts array
232 + const char *alerts_pattern = NULL;
233 + CLEAN_BUFFER *alerts_buffer = NULL;
234 +
235 + alerts_buffer = mcp_params_parse_array_to_pattern(params, "alerts", false, false, MCP_TOOL_LIST_ALL_ALERTS, mcpc->error);
236 + if (buffer_strlen(mcpc->error) > 0) {
237 + return MCP_RC_BAD_REQUEST;
238 + }
239 + if (alerts_buffer)
240 + alerts_pattern = buffer_tostring(alerts_buffer);
241 +
242 + // Extract classifications array
243 + const char *classifications_pattern = NULL;
244 + CLEAN_BUFFER *classifications_buffer = NULL;
245 +
246 + classifications_buffer = mcp_params_parse_array_to_pattern(params, "classifications", false, false, MCP_TOOL_LIST_ALL_ALERTS, mcpc->error);
247 + if (buffer_strlen(mcpc->error) > 0) {
248 + return MCP_RC_BAD_REQUEST;
249 + }
250 + if (classifications_buffer)
251 + classifications_pattern = buffer_tostring(classifications_buffer);
252 +
253 + // Extract types array
254 + const char *types_pattern = NULL;
255 + CLEAN_BUFFER *types_buffer = NULL;
256 +
257 + types_buffer = mcp_params_parse_array_to_pattern(params, "types", false, false, MCP_TOOL_LIST_ALL_ALERTS, mcpc->error);
258 + if (buffer_strlen(mcpc->error) > 0) {
259 + return MCP_RC_BAD_REQUEST;
260 + }
261 + if (types_buffer)
262 + types_pattern = buffer_tostring(types_buffer);
263 +
264 + // Extract components array
265 + const char *components_pattern = NULL;
266 + CLEAN_BUFFER *components_buffer = NULL;
267 +
268 + components_buffer = mcp_params_parse_array_to_pattern(params, "components", false, false, MCP_TOOL_LIST_ALL_ALERTS, mcpc->error);
269 + if (buffer_strlen(mcpc->error) > 0) {
270 + return MCP_RC_BAD_REQUEST;
271 + }
272 + if (components_buffer)
273 + components_pattern = buffer_tostring(components_buffer);
274 +
275 + // Extract roles array
276 + const char *roles_pattern = NULL;
277 + CLEAN_BUFFER *roles_buffer = NULL;
278 +
279 + roles_buffer = mcp_params_parse_array_to_pattern(params, "roles", false, false, MCP_TOOL_LIST_ALL_ALERTS, mcpc->error);
280 + if (buffer_strlen(mcpc->error) > 0) {
281 + return MCP_RC_BAD_REQUEST;
282 + }
283 + if (roles_buffer)
284 + roles_pattern = buffer_tostring(roles_buffer);
285 +
286 + // Extract timeout parameter
287 + int timeout = mcp_params_extract_timeout(params, "timeout", 60, 1, 3600, mcpc->error);
288 + if (buffer_strlen(mcpc->error) > 0) {
289 + return MCP_RC_BAD_REQUEST;
290 + }
291 +
292 + // Create request structure
293 + struct api_v2_contexts_request req = {
294 + .scope_nodes = nodes_pattern,
295 + .scope_contexts = NULL, // Not used for alert transitions
296 + .nodes = NULL,
297 + .contexts = NULL,
298 + .after = after,
299 + .before = before,
300 + .timeout_ms = timeout * 1000, // Convert seconds to milliseconds
301 + .options = CONTEXTS_OPTION_CONFIGURATIONS | CONTEXTS_OPTION_MCP | CONTEXTS_OPTION_RFC3339 | CONTEXTS_OPTION_JSON_LONG_KEYS | CONTEXTS_OPTION_MINIFY,
302 + .cardinality_limit = cardinality_limit,
303 + .alerts = {
304 + .last = cardinality_limit,
305 + .global_id_anchor = global_id_anchor,
306 + .facets = {
307 + [ATF_STATUS] = status_pattern,
308 + [ATF_CLASS] = classifications_pattern,
309 + [ATF_TYPE] = types_pattern,
310 + [ATF_COMPONENT] = components_pattern,
311 + [ATF_ROLE] = roles_pattern,
312 + [ATF_NODE] = NULL,
313 + [ATF_ALERT_NAME] = alerts_pattern,
314 + [ATF_CHART_NAME] = instances_pattern,
315 + [ATF_CONTEXT] = metrics_pattern,
316 + },
317 + }
318 + };
319 +
320 + // Execute the query
321 + CONTEXTS_V2_MODE mode = CONTEXTS_V2_NODES | CONTEXTS_V2_ALERT_TRANSITIONS;
322 + CLEAN_BUFFER *t = buffer_create(0, NULL);
323 + int response = rrdcontext_to_json_v2(t, &req, mode);
324 +
325 + if (response != HTTP_RESP_OK) {
326 + buffer_sprintf(mcpc->error, "Query failed with response code %d", response);
327 + return MCP_RC_ERROR;
328 + }
329 +
330 + // Initialize success response
331 + mcp_init_success_result(mcpc, id);
332 +
333 + // Start building a content-array for the result
334 + buffer_json_member_add_array(mcpc->result, "content");
335 + {
336 + // Return text content for LLM compatibility
337 + buffer_json_add_array_item_object(mcpc->result);
338 + {
339 + buffer_json_member_add_string(mcpc->result, "type", "text");
340 + buffer_json_member_add_string(mcpc->result, "text", buffer_tostring(t));
341 + }
342 + buffer_json_object_close(mcpc->result); // Close text content
343 + }
344 + buffer_json_array_close(mcpc->result); // Close content array
345 + buffer_json_object_close(mcpc->result); // Close result object
346 + buffer_json_finalize(mcpc->result); // Finalize the JSON
347 +
348 + return MCP_RC_OK;
349 +}
\ No newline at end of file
src/web/mcp/mcp-tools-alert-transitions.h new
+12
@@ -0,0 +1,12 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_ALERT_TRANSITIONS_H
4 +#define NETDATA_MCP_TOOLS_ALERT_TRANSITIONS_H
5 +
6 +#include "mcp.h"
7 +
8 +// Function declarations for alert transitions tool
9 +void mcp_tool_list_alert_transitions_schema(BUFFER *buffer);
10 +MCP_RETURN_CODE mcp_tool_list_alert_transitions_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
11 +
12 +#endif //NETDATA_MCP_TOOLS_ALERT_TRANSITIONS_H
\ No newline at end of file
src/web/mcp/mcp-tools-configured-alerts.c new
+122
@@ -0,0 +1,122 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-tools-configured-alerts.h"
4 +#include "mcp-params.h"
5 +#include "health/health_internals.h"
6 +
7 +// Schema for list_configured_alerts - no parameters
8 +void mcp_tool_list_configured_alerts_schema(BUFFER *buffer) {
9 + // Tool metadata
10 + buffer_json_member_add_object(buffer, "inputSchema");
11 + buffer_json_member_add_string(buffer, "type", "object");
12 + buffer_json_member_add_string(buffer, "title", "List configured alerts");
13 +
14 + buffer_json_member_add_object(buffer, "properties");
15 + // No properties - this tool accepts no parameters
16 + buffer_json_object_close(buffer); // properties
17 +
18 + buffer_json_object_close(buffer); // inputSchema
19 +}
20 +
21 +// Execute list_configured_alerts - no filtering, returns all prototypes
22 +MCP_RETURN_CODE mcp_tool_list_configured_alerts_execute(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, MCP_REQUEST_ID id) {
23 + if (!mcpc || id == 0)
24 + return MCP_RC_ERROR;
25 +
26 + // Create a temporary buffer for the result
27 + CLEAN_BUFFER *t = buffer_create(0, NULL);
28 + size_t count = 0;
29 +
30 + // Build the JSON response in tabular format
31 + buffer_json_initialize(t, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
32 + buffer_json_member_add_uint64(t, "format_version", 1);
33 +
34 + // Add header for tabular data
35 + buffer_json_member_add_array(t, "configured_alerts_header");
36 + {
37 + buffer_json_add_array_item_string(t, "name");
38 + buffer_json_add_array_item_string(t, "applies_to");
39 + buffer_json_add_array_item_string(t, "on");
40 + buffer_json_add_array_item_string(t, "summary");
41 + buffer_json_add_array_item_string(t, "component");
42 + buffer_json_add_array_item_string(t, "classification");
43 + buffer_json_add_array_item_string(t, "type");
44 + buffer_json_add_array_item_string(t, "recipient");
45 + }
46 + buffer_json_array_close(t); // configured_alerts_header
47 +
48 + // Add tabular data
49 + buffer_json_member_add_array(t, "configured_alerts");
50 + {
51 + // Iterate through all alert prototypes
52 + RRD_ALERT_PROTOTYPE *ap;
53 + dfe_start_read(health_globals.prototypes.dict, ap) {
54 + // Only use the first rule for each prototype
55 + if (ap) {
56 + buffer_json_add_array_item_array(t); // Start row
57 + {
58 + // name
59 + buffer_json_add_array_item_string(t, string2str(ap->config.name));
60 +
61 + // alert_type (template or alarm)
62 + buffer_json_add_array_item_string(t, ap->match.is_template ? "context" : "instance");
63 +
64 + // on (context or instance)
65 + if(ap->match.is_template)
66 + buffer_json_add_array_item_string(t, string2str(ap->match.on.context));
67 + else
68 + buffer_json_add_array_item_string(t, string2str(ap->match.on.chart));
69 +
70 + // summary
71 + buffer_json_add_array_item_string(t, string2str(ap->config.summary));
72 +
73 + // component
74 + buffer_json_add_array_item_string(t, string2str(ap->config.component));
75 +
76 + // classification
77 + buffer_json_add_array_item_string(t, string2str(ap->config.classification));
78 +
79 + // type
80 + buffer_json_add_array_item_string(t, string2str(ap->config.type));
81 +
82 + // recipient
83 + buffer_json_add_array_item_string(t, string2str(ap->config.recipient));
84 + }
85 + buffer_json_array_close(t); // End row
86 +
87 + count++;
88 + }
89 + }
90 + dfe_done(ap);
91 + }
92 + buffer_json_array_close(t); // configured_alerts
93 +
94 + // Add summary
95 + buffer_json_member_add_object(t, "summary");
96 + {
97 + buffer_json_member_add_uint64(t, "total_prototypes", count);
98 + }
99 + buffer_json_object_close(t); // summary
100 +
101 + buffer_json_finalize(t);
102 +
103 + // Initialize success response
104 + mcp_init_success_result(mcpc, id);
105 +
106 + // Start building a content array for the result
107 + buffer_json_member_add_array(mcpc->result, "content");
108 + {
109 + // Return text content for LLM compatibility
110 + buffer_json_add_array_item_object(mcpc->result);
111 + {
112 + buffer_json_member_add_string(mcpc->result, "type", "text");
113 + buffer_json_member_add_string(mcpc->result, "text", buffer_tostring(t));
114 + }
115 + buffer_json_object_close(mcpc->result); // Close text content
116 + }
117 + buffer_json_array_close(mcpc->result); // Close content array
118 + buffer_json_object_close(mcpc->result); // Close result object
119 + buffer_json_finalize(mcpc->result); // Finalize the JSON
120 +
121 + return MCP_RC_OK;
122 +}
\ No newline at end of file
src/web/mcp/mcp-tools-configured-alerts.h new
+16
@@ -0,0 +1,16 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_CONFIGURED_ALERTS_H
4 +#define NETDATA_MCP_TOOLS_CONFIGURED_ALERTS_H
5 +
6 +#include "mcp-tools.h"
7 +
8 +#define MCP_TOOL_LIST_CONFIGURED_ALERTS "list_configured_alerts"
9 +
10 +// Schema generator function
11 +void mcp_tool_list_configured_alerts_schema(BUFFER *buffer);
12 +
13 +// Execution function
14 +MCP_RETURN_CODE mcp_tool_list_configured_alerts_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
15 +
16 +#endif //NETDATA_MCP_TOOLS_CONFIGURED_ALERTS_H
\ No newline at end of file
src/web/mcp/mcp-tools-execute-function-internal.h new
+174
@@ -0,0 +1,174 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_INTERNAL_H
4 +#define NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_INTERNAL_H
5 +
6 +#include "mcp.h"
7 +#include "database/contexts/rrdcontext.h"
8 +
9 +// Pagination units enumeration - only supported types enable cursor pagination
10 +typedef enum mcp_pagination_units {
11 + MCP_PAGINATION_UNITS_UNKNOWN = 0, // Unknown units - pagination disabled
12 + MCP_PAGINATION_UNITS_TIMESTAMP_USEC, // Microsecond timestamps
13 +} MCP_PAGINATION_UNITS;
14 +
15 +// Define operator types for faster comparison
16 +typedef enum {
17 + OP_EQUALS, // ==
18 + OP_NOT_EQUALS, // != or <>
19 + OP_LESS, // <
20 + OP_LESS_EQUALS, // <=
21 + OP_GREATER, // >
22 + OP_GREATER_EQUALS, // >=
23 + OP_MATCH, // simple pattern
24 + OP_NOT_MATCH, // a negative match of a simple pattern
25 + OP_UNKNOWN // unknown operator
26 +} OPERATOR_TYPE;
27 +
28 +// Maximum number of conditions we expect to handle
29 +#define MAX_CONDITIONS 20
30 +#define MAX_COLUMNS 300 // Maximum number of columns we can handle
31 +#define MAX_SELECTED_COLUMNS 100 // Maximum number of columns that can be selected
32 +
33 +// Result status for table processing
34 +typedef enum {
35 + MCP_TABLE_OK, // Success
36 + MCP_TABLE_ERROR_INVALID_CONDITIONS, // Condition format/parsing error
37 + MCP_TABLE_ERROR_NO_MATCHES_WITH_MISSING_COLUMNS, // No matches, some columns not found
38 + MCP_TABLE_ERROR_NO_MATCHES, // No matches with valid columns
39 + MCP_TABLE_ERROR_INVALID_SORT_ORDER, // Invalid sort order parameter
40 + MCP_TABLE_ERROR_COLUMNS_NOT_FOUND, // Requested columns not found
41 + MCP_TABLE_ERROR_SORT_COLUMN_NOT_FOUND, // Sort column not found
42 + MCP_TABLE_ERROR_TOO_MANY_COLUMNS, // Exceeds MAX_COLUMNS
43 + MCP_TABLE_NOT_JSON, // Response is not valid JSON
44 + MCP_TABLE_NOT_PROCESSABLE, // JSON but not a processable table format
45 + MCP_TABLE_EMPTY_RESULT, // Function returned no rows
46 + MCP_TABLE_INFO_MISSING_COLUMNS_FOUND_RESULTS, // Missing columns but found via wildcard
47 + MCP_TABLE_RESPONSE_TOO_BIG // Result too big, guidance added
48 +} MCP_TABLE_RESULT_STATUS;
49 +
50 +// Function response types
51 +typedef enum {
52 + FN_TYPE_UNKNOWN = 0,
53 + FN_TYPE_TABLE, // Regular table (has_history=false)
54 + FN_TYPE_TABLE_WITH_HISTORY, // Logs table (has_history=true)
55 + FN_TYPE_NOT_TABLE // Not a table format
56 +} MCP_FUNCTION_TYPE;
57 +
58 +// Value types for conditions
59 +typedef enum {
60 + COND_VALUE_STRING,
61 + COND_VALUE_NUMBER,
62 + COND_VALUE_BOOLEAN,
63 + COND_VALUE_NULL
64 +} CONDITION_VALUE_TYPE;
65 +
66 +// Structure to hold preprocessed condition information
67 +typedef struct condition_s {
68 + int column_index; // Index of the column in the row (-1 for wildcard search)
69 + const char *column_name; // Name of the column (referenced from json-c, not owned)
70 + OPERATOR_TYPE op; // Operator type
71 + CONDITION_VALUE_TYPE v_type; // Type of the value
72 + union {
73 + const char *v_str; // String value (referenced from json-c, not owned)
74 + bool v_bool; // Boolean value
75 + double v_num; // Numeric value (using double to handle both int and float)
76 + };
77 + SIMPLE_PATTERN *pattern; // Pre-compiled pattern for MATCH operations (owned - must be freed)
78 +} CONDITION;
79 +
80 +// Structure to hold an array of conditions
81 +typedef struct {
82 + CONDITION items[MAX_CONDITIONS]; // Fixed array of conditions
83 + size_t count; // Number of conditions currently in use
84 + bool has_missing_columns; // True if any column was not found
85 +} CONDITION_ARRAY;
86 +
87 +// Structure to hold function data throughout processing
88 +typedef struct {
89 + // Request context - all parsed parameters
90 + struct {
91 + // Core request data
92 + MCP_CLIENT *mcpc; // The MCP client
93 + struct json_object *params; // The raw parameters as given by the client
94 +
95 + // Parsed required parameters
96 + const char *function; // Function name to execute
97 + const char *node; // Node name/id/guid
98 + RRDHOST *host; // The resolved host
99 + time_t timeout; // Timeout in seconds
100 +
101 + // Transaction tracking
102 + nd_uuid_t transaction_uuid; // Transaction UUID
103 + char transaction[UUID_STR_LEN]; // Transaction UUID string
104 +
105 + // Authentication
106 + USER_AUTH *auth; // User authentication info
107 +
108 + // Parsed optional parameters for table filtering
109 + struct {
110 + const char *column; // Column to sort by (referenced from json-c, not owned)
111 + bool descending; // true for DESC, false for ASC
112 + } sort; // Sort configuration
113 +
114 + size_t limit; // Row limit (0 = no limit)
115 + struct {
116 + const char *array[MAX_SELECTED_COLUMNS]; // Column names to include (referenced from json-c, not owned)
117 + size_t count; // Number of columns selected
118 + } columns; // Selected columns
119 +
120 + CONDITION_ARRAY conditions; // Preprocessed conditions
121 +
122 + // Time-based and history parameters
123 + time_t after; // Start time for the query (0 = not specified)
124 + time_t before; // End time for the query (0 = not specified)
125 + const char *cursor; // Pagination cursor (MCP standard) (referenced from json-c, not owned)
126 + usec_t anchor; // Internal anchor timestamp converted from cursor (0 = not specified)
127 + size_t last; // Number of last rows (0 = not specified)
128 + const char *direction; // Query direction: "forward" or "backward" (referenced from json-c, not owned)
129 + const char *query; // Full-text search query (referenced from json-c, not owned)
130 + } request;
131 +
132 + // Pagination settings (copied from registry_entry to avoid keeping it locked)
133 + struct {
134 + bool enabled; // whether pagination is supported
135 + MCP_PAGINATION_UNITS units; // units of the pagination column
136 + STRING *column; // column name in data (owned copy)
137 + } pagination;
138 +
139 + // Input data from the function
140 + struct {
141 + struct json_object *jobj; // The parsed JSON object
142 + BUFFER *json; // The original JSON response
143 + MCP_FUNCTION_TYPE type; // Type of response
144 + size_t rows; // Number of rows in original data
145 + size_t columns; // Number of columns available
146 + } input;
147 +
148 + // Output data after processing
149 + struct {
150 + MCP_TABLE_RESULT_STATUS status; // Result of processing
151 + BUFFER *result; // Response to send to the client
152 + size_t rows; // Number of rows after filtering
153 + size_t columns; // Number of columns selected
154 + } output;
155 +} MCP_FUNCTION_DATA;
156 +
157 +// Function prototypes shared between execute-function and execute-function-logs
158 +
159 +// Initialize MCP_FUNCTION_DATA structure
160 +void mcp_functions_data_init(MCP_FUNCTION_DATA *data);
161 +
162 +// Clean up MCP_FUNCTION_DATA structure
163 +void mcp_functions_data_cleanup(MCP_FUNCTION_DATA *data);
164 +
165 +// Analyze the JSON response and determine its type
166 +MCP_FUNCTION_TYPE mcp_functions_analyze_response(struct json_object *json_obj, int *out_status);
167 +
168 +// Convert string operator to enum type
169 +OPERATOR_TYPE mcp_functions_string_to_operator(const char *op_str);
170 +
171 +// Free patterns in the condition array
172 +void mcp_functions_free_condition_patterns(CONDITION_ARRAY *condition_array);
173 +
174 +#endif // NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_INTERNAL_H
\ No newline at end of file
src/web/mcp/mcp-tools-execute-function-registry.c new
+445
@@ -0,0 +1,445 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-tools-execute-function-registry.h"
4 +#include "database/rrdfunctions.h"
5 +
6 +// Parameter type string mappings
7 +ENUM_STR_MAP_DEFINE(MCP_REQUIRED_PARAMS_TYPE) = {
8 + { MCP_REQUIRED_PARAMS_TYPE_SELECT, "select" },
9 + { MCP_REQUIRED_PARAMS_TYPE_MULTISELECT, "multiselect" },
10 + { 0, NULL }
11 +};
12 +
13 +ENUM_STR_DEFINE_FUNCTIONS(MCP_REQUIRED_PARAMS_TYPE, MCP_REQUIRED_PARAMS_TYPE_SELECT, "select")
14 +
15 +// Pagination units string mappings
16 +ENUM_STR_MAP_DEFINE(MCP_PAGINATION_UNITS) = {
17 + { MCP_PAGINATION_UNITS_TIMESTAMP_USEC, "timestamp_usec" },
18 + { 0, NULL }
19 +};
20 +
21 +ENUM_STR_DEFINE_FUNCTIONS(MCP_PAGINATION_UNITS, MCP_PAGINATION_UNITS_UNKNOWN, "unknown")
22 +
23 +// Static dictionary to store function registry entries
24 +static DICTIONARY *functions_registry = NULL;
25 +
26 +// Cleanup function for registry entries
27 +static void registry_entry_cleanup(MCP_FUNCTION_REGISTRY_ENTRY *entry) {
28 + if (!entry)
29 + return;
30 +
31 + // Free STRING pointers
32 + string_freez(entry->help);
33 +
34 + // Free pagination fields
35 + string_freez(entry->pagination.key);
36 + string_freez(entry->pagination.column);
37 +
38 + // Free parameters
39 + for (size_t i = 0; i < entry->required_params_count; i++) {
40 + MCP_FUNCTION_PARAM *param = &entry->required_params[i];
41 + string_freez(param->id);
42 + string_freez(param->name);
43 + string_freez(param->help);
44 +
45 + // Free options
46 + for (size_t j = 0; j < param->options_count; j++) {
47 + string_freez(param->options[j].id);
48 + string_freez(param->options[j].name);
49 + string_freez(param->options[j].info);
50 + }
51 + freez(param->options);
52 + }
53 + freez(entry->required_params);
54 +}
55 +
56 +// Dictionary callbacks
57 +static void registry_entry_insert_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
58 + MCP_FUNCTION_REGISTRY_ENTRY *entry = (MCP_FUNCTION_REGISTRY_ENTRY *)value;
59 +
60 + rw_spinlock_init(&entry->spinlock);
61 + spinlock_init(&entry->update_spinlock);
62 +}
63 +
64 +static bool registry_entry_conflict_callback(const DICTIONARY_ITEM *item __maybe_unused, void *old_value, void *new_value, void *data __maybe_unused) {
65 + MCP_FUNCTION_REGISTRY_ENTRY *old_entry = (MCP_FUNCTION_REGISTRY_ENTRY *)old_value;
66 + MCP_FUNCTION_REGISTRY_ENTRY *new_entry = (MCP_FUNCTION_REGISTRY_ENTRY *)new_value;
67 +
68 + // Get write lock on the old entry
69 + rw_spinlock_write_lock(&old_entry->spinlock);
70 +
71 + // Swap all members between old and new
72 + // This moves the old data to new_entry and new data to old_entry
73 + SWAP(old_entry->type, new_entry->type);
74 + SWAP(old_entry->has_history, new_entry->has_history);
75 + SWAP(old_entry->update_every, new_entry->update_every);
76 + SWAP(old_entry->version, new_entry->version);
77 + SWAP(old_entry->supports_post, new_entry->supports_post);
78 + SWAP(old_entry->help, new_entry->help);
79 + SWAP(old_entry->required_params_count, new_entry->required_params_count);
80 + SWAP(old_entry->required_params, new_entry->required_params);
81 + SWAP(old_entry->pagination, new_entry->pagination);
82 + SWAP(old_entry->last_update, new_entry->last_update);
83 + SWAP(old_entry->expires, new_entry->expires);
84 +
85 + // Release the lock
86 + rw_spinlock_write_unlock(&old_entry->spinlock);
87 +
88 + // Clean up the new entry (which now contains the old data)
89 + registry_entry_cleanup(new_entry);
90 +
91 + // Return false to reject the new value (we've already updated old_value)
92 + return false;
93 +}
94 +
95 +static void registry_entry_delete_callback(const DICTIONARY_ITEM *item __maybe_unused, void *value, void *data __maybe_unused) {
96 + MCP_FUNCTION_REGISTRY_ENTRY *entry = (MCP_FUNCTION_REGISTRY_ENTRY *)value;
97 +
98 + // Clean up the entry data
99 + registry_entry_cleanup(entry);
100 +}
101 +
102 +// Initialize the functions-registry
103 +void mcp_functions_registry_init(void) {
104 + if (functions_registry)
105 + return;
106 +
107 + functions_registry = dictionary_create_advanced(
108 + DICT_OPTION_DONT_OVERWRITE_VALUE | DICT_OPTION_FIXED_SIZE,
109 + NULL,
110 + sizeof(MCP_FUNCTION_REGISTRY_ENTRY));
111 +
112 + dictionary_register_insert_callback(functions_registry, registry_entry_insert_callback, NULL);
113 + dictionary_register_delete_callback(functions_registry, registry_entry_delete_callback, NULL);
114 + dictionary_register_conflict_callback(functions_registry, registry_entry_conflict_callback, NULL);
115 +}
116 +
117 +// Clean up the functions-registry
118 +void mcp_functions_registry_cleanup(void) {
119 + if (!functions_registry)
120 + return;
121 +
122 + dictionary_destroy(functions_registry);
123 + functions_registry = NULL;
124 +}
125 +
126 +// Parse JSON info response and populate registry entry
127 +static int parse_function_info(struct json_object *json_obj, MCP_FUNCTION_REGISTRY_ENTRY *entry) {
128 + struct json_object *jobj;
129 +
130 + // Parse version (v3+ supports POST)
131 + entry->version = 1; // default to v1
132 + if (json_object_object_get_ex(json_obj, "v", &jobj)) {
133 + entry->version = json_object_get_int(jobj);
134 + }
135 + entry->supports_post = (entry->version >= 3);
136 +
137 + // Parse type
138 + if (json_object_object_get_ex(json_obj, "type", &jobj)) {
139 + const char *type_str = json_object_get_string(jobj);
140 + if (strcmp(type_str, "table") == 0) {
141 + entry->type = FN_TYPE_TABLE;
142 + } else {
143 + entry->type = FN_TYPE_UNKNOWN;
144 + }
145 + }
146 +
147 + // Parse has_history
148 + if (json_object_object_get_ex(json_obj, "has_history", &jobj)) {
149 + entry->has_history = json_object_get_boolean(jobj);
150 + if (entry->has_history && entry->type == FN_TYPE_TABLE) {
151 + entry->type = FN_TYPE_TABLE_WITH_HISTORY;
152 + }
153 + }
154 +
155 + // Parse update_every
156 + if (json_object_object_get_ex(json_obj, "update_every", &jobj)) {
157 + entry->update_every = json_object_get_int(jobj);
158 + }
159 +
160 + // Parse help
161 + if (json_object_object_get_ex(json_obj, "help", &jobj)) {
162 + entry->help = string_strdupz(json_object_get_string(jobj));
163 + }
164 +
165 + // Parse accepted_params to detect supported optional parameters
166 + if (json_object_object_get_ex(json_obj, "accepted_params", &jobj)) {
167 + if (json_object_is_type(jobj, json_type_array)) {
168 + size_t accepted_count = json_object_array_length(jobj);
169 + for (size_t i = 0; i < accepted_count; i++) {
170 + struct json_object *param_obj = json_object_array_get_idx(jobj, i);
171 + if (param_obj && json_object_is_type(param_obj, json_type_string)) {
172 + const char *param_name = json_object_get_string(param_obj);
173 +
174 + // Check for timeframe parameters
175 + if (strcmp(param_name, "after") == 0 || strcmp(param_name, "before") == 0) {
176 + entry->has_timeframe = true;
177 + }
178 + else if (strcmp(param_name, "last") == 0) {
179 + entry->has_last = true;
180 + }
181 + else if (strcmp(param_name, "data_only") == 0) {
182 + entry->has_data_only = true;
183 + }
184 + else if (strcmp(param_name, "direction") == 0) {
185 + entry->has_direction = true;
186 + }
187 + else if (strcmp(param_name, "query") == 0) {
188 + entry->has_query = true;
189 + }
190 + else if (strcmp(param_name, "slice") == 0) {
191 + entry->has_slice = true;
192 + }
193 + }
194 + }
195 + }
196 + }
197 +
198 + // Parse pagination object
199 + if (json_object_object_get_ex(json_obj, "pagination", &jobj)) {
200 + if (json_object_is_type(jobj, json_type_object)) {
201 + struct json_object *field;
202 +
203 + // Parse enabled flag
204 + if (json_object_object_get_ex(jobj, "enabled", &field) &&
205 + json_object_is_type(field, json_type_boolean)) {
206 + entry->pagination.enabled = json_object_get_boolean(field);
207 + }
208 +
209 + // Only parse other fields if pagination is enabled
210 + if (entry->pagination.enabled) {
211 + // Parse key
212 + if (json_object_object_get_ex(jobj, "key", &field) &&
213 + json_object_is_type(field, json_type_string)) {
214 + entry->pagination.key = string_strdupz(json_object_get_string(field));
215 + }
216 +
217 + // Parse column
218 + if (json_object_object_get_ex(jobj, "column", &field) &&
219 + json_object_is_type(field, json_type_string)) {
220 + entry->pagination.column = string_strdupz(json_object_get_string(field));
221 + }
222 +
223 + // Parse units
224 + if (json_object_object_get_ex(jobj, "units", &field) &&
225 + json_object_is_type(field, json_type_string)) {
226 + const char *units_str = json_object_get_string(field);
227 + entry->pagination.units = MCP_PAGINATION_UNITS_2id(units_str);
228 +
229 + // If units are unknown, disable pagination
230 + if (entry->pagination.units == MCP_PAGINATION_UNITS_UNKNOWN) {
231 + entry->pagination.enabled = false;
232 + }
233 + }
234 + }
235 + }
236 + }
237 +
238 + // Parse required_params
239 + if (json_object_object_get_ex(json_obj, "required_params", &jobj)) {
240 + if (json_object_is_type(jobj, json_type_array)) {
241 + entry->required_params_count = json_object_array_length(jobj);
242 + if (entry->required_params_count > 0) {
243 + entry->required_params = callocz(entry->required_params_count, sizeof(MCP_FUNCTION_PARAM));
244 +
245 + for (size_t i = 0; i < entry->required_params_count; i++) {
246 + struct json_object *param_obj = json_object_array_get_idx(jobj, i);
247 + MCP_FUNCTION_PARAM *param = &entry->required_params[i];
248 +
249 + struct json_object *field;
250 +
251 + // Parse param fields
252 + if (json_object_object_get_ex(param_obj, "id", &field))
253 + param->id = string_strdupz(json_object_get_string(field));
254 +
255 + if (json_object_object_get_ex(param_obj, "name", &field))
256 + param->name = string_strdupz(json_object_get_string(field));
257 +
258 + if (json_object_object_get_ex(param_obj, "help", &field))
259 + param->help = string_strdupz(json_object_get_string(field));
260 +
261 + if (json_object_object_get_ex(param_obj, "type", &field)) {
262 + const char *type_str = json_object_get_string(field);
263 + param->type = MCP_REQUIRED_PARAMS_TYPE_2id(type_str);
264 + }
265 +
266 + if (json_object_object_get_ex(param_obj, "unique_view", &field))
267 + param->unique_view = json_object_get_boolean(field);
268 +
269 + // Parse options
270 + if (json_object_object_get_ex(param_obj, "options", &field)) {
271 + if (json_object_is_type(field, json_type_array)) {
272 + param->options_count = json_object_array_length(field);
273 + if (param->options_count > 0) {
274 + param->options = callocz(param->options_count, sizeof(MCP_FUNCTION_PARAM_OPTION));
275 +
276 + for (size_t j = 0; j < param->options_count; j++) {
277 + struct json_object *opt_obj = json_object_array_get_idx(field, j);
278 + MCP_FUNCTION_PARAM_OPTION *opt = &param->options[j];
279 +
280 + struct json_object *opt_field;
281 + if (json_object_object_get_ex(opt_obj, "id", &opt_field))
282 + opt->id = string_strdupz(json_object_get_string(opt_field));
283 +
284 + if (json_object_object_get_ex(opt_obj, "name", &opt_field))
285 + opt->name = string_strdupz(json_object_get_string(opt_field));
286 +
287 + if (json_object_object_get_ex(opt_obj, "info", &opt_field))
288 + opt->info = string_strdupz(json_object_get_string(opt_field));
289 + }
290 + }
291 + }
292 + }
293 + }
294 + }
295 + }
296 + }
297 +
298 + time_t now = now_realtime_sec();
299 + entry->last_update = now;
300 + entry->expires = now + MCP_FUNCTIONS_REGISTRY_TTL;
301 +
302 + return 0;
303 +}
304 +
305 +// Fetch function info from the node (private function)
306 +static MCP_FUNCTION_REGISTRY_ENTRY *mcp_function_get_info(RRDHOST *host, const char *function_name, BUFFER *error) {
307 + if (!host || !function_name) {
308 + buffer_strcat(error, "Invalid host or function name");
309 + return NULL;
310 + }
311 +
312 + // Prepare the info request
313 + char info_function[256];
314 + snprintfz(info_function, sizeof(info_function), "%s info", function_name);
315 +
316 + USER_AUTH auth = {
317 + .user_role = HTTP_USER_ROLE_ADMIN,
318 + .access = HTTP_ACCESS_ALL,
319 + .method = USER_AUTH_METHOD_GOD,
320 + .client_ip = "mcp-info",
321 + .client_name = "mcp-tools-execute-function-registry",
322 + };
323 +
324 + // Create a source buffer from user_auth
325 + CLEAN_BUFFER *source = buffer_create(0, NULL);
326 + user_auth_to_source_buffer(&auth, source);
327 + buffer_strcat(source, ",modelcontextprotocol");
328 +
329 + // Call the function with info parameter
330 + BUFFER *response = buffer_create(0, NULL);
331 + int code = rrd_function_run(
332 + host,
333 + response,
334 + 10,
335 + auth.access,
336 + info_function,
337 + true,
338 + NULL,
339 + NULL,
340 + NULL,
341 + NULL,
342 + NULL,
343 + NULL,
344 + NULL,
345 + NULL,
346 + buffer_tostring(source),
347 + false);
348 +
349 + if (code != HTTP_RESP_OK) {
350 + buffer_sprintf(error, "Failed to get function info: HTTP %d", code);
351 + buffer_free(response);
352 + return NULL;
353 + }
354 +
355 + // Parse JSON response
356 + struct json_tokener *tokener = json_tokener_new();
357 + struct json_object *json_obj = json_tokener_parse_ex(tokener, buffer_tostring(response), buffer_strlen(response));
358 + json_tokener_free(tokener);
359 + buffer_free(response);
360 +
361 + if (!json_obj) {
362 + buffer_strcat(error, "Failed to parse JSON response");
363 + return NULL;
364 + }
365 +
366 + // Check if it's a special info response with required_params
367 + struct json_object *required_params_obj;
368 + if (!json_object_object_get_ex(json_obj, "required_params", &required_params_obj)) {
369 + // This function doesn't support parameters
370 + MCP_FUNCTION_REGISTRY_ENTRY *entry = callocz(1, sizeof(MCP_FUNCTION_REGISTRY_ENTRY));
371 + parse_function_info(json_obj, entry);
372 + json_object_put(json_obj);
373 + return entry;
374 + }
375 +
376 + // Create and populate registry entry
377 + MCP_FUNCTION_REGISTRY_ENTRY *entry = callocz(1, sizeof(MCP_FUNCTION_REGISTRY_ENTRY));
378 + if (parse_function_info(json_obj, entry) != 0) {
379 + json_object_put(json_obj);
380 + freez(entry);
381 + buffer_strcat(error, "Failed to parse function info");
382 + return NULL;
383 + }
384 +
385 + json_object_put(json_obj);
386 + return entry;
387 +}
388 +
389 +// Create a dictionary key from host and function name
390 +static void create_registry_key(BUFFER *key_buffer, RRDHOST *host, const char *function_name) {
391 + buffer_flush(key_buffer);
392 + buffer_sprintf(key_buffer, "%s|%s", rrdhost_hostname(host), function_name);
393 +}
394 +
395 +// Get a registry entry for a function
396 +MCP_FUNCTION_REGISTRY_ENTRY *mcp_functions_registry_get(RRDHOST *host, const char *function_name, BUFFER *error) {
397 + if (!functions_registry) {
398 + buffer_strcat(error, "Functions registry not initialized");
399 + return NULL;
400 + }
401 +
402 + // Create a dictionary key
403 + CLEAN_BUFFER *key = buffer_create(0, NULL);
404 + create_registry_key(key, host, function_name);
405 + const char *key_str = buffer_tostring(key);
406 +
407 + time_t now = now_realtime_sec();
408 +
409 + // Try to get an existing entry
410 + MCP_FUNCTION_REGISTRY_ENTRY *entry = dictionary_get(functions_registry, key_str);
411 + MCP_FUNCTION_REGISTRY_ENTRY *old_entry = NULL;
412 +
413 + if(entry && entry->last_update + MCP_FUNCTIONS_REGISTRY_TTL < now && spinlock_trylock(&entry->update_spinlock)) {
414 + old_entry = entry;
415 + entry = NULL;
416 + }
417 +
418 + if(!entry) {
419 + MCP_FUNCTION_REGISTRY_ENTRY *new_info = mcp_function_get_info(host, function_name, error);
420 + if(new_info) {
421 + entry = dictionary_set(functions_registry, key_str, new_info, sizeof(MCP_FUNCTION_REGISTRY_ENTRY));
422 + freez(new_info);
423 + }
424 + else if(old_entry)
425 + entry = old_entry;
426 + else
427 + return NULL;
428 + }
429 +
430 + if(old_entry)
431 + spinlock_unlock(&old_entry->update_spinlock);
432 +
433 + if(entry)
434 + rw_spinlock_read_lock(&entry->spinlock);
435 +
436 + return entry;
437 +}
438 +
439 +// Release a registry entry
440 +void mcp_functions_registry_release(MCP_FUNCTION_REGISTRY_ENTRY *entry) {
441 + if (!entry)
442 + return;
443 +
444 + rw_spinlock_read_unlock(&entry->spinlock);
445 +}
\ No newline at end of file
src/web/mcp/mcp-tools-execute-function-registry.h new
+87
@@ -0,0 +1,87 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_REGISTRY_H
4 +#define NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_REGISTRY_H
5 +
6 +#include "mcp.h"
7 +#include "mcp-tools-execute-function-internal.h"
8 +
9 +// Registry entry TTL in seconds (10 minutes)
10 +#define MCP_FUNCTIONS_REGISTRY_TTL 600
11 +
12 +// Parameter type enumeration
13 +typedef enum mcp_required_params_type {
14 + MCP_REQUIRED_PARAMS_TYPE_SELECT = 0,
15 + MCP_REQUIRED_PARAMS_TYPE_MULTISELECT
16 +} MCP_REQUIRED_PARAMS_TYPE;
17 +
18 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(MCP_REQUIRED_PARAMS_TYPE)
19 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(MCP_PAGINATION_UNITS)
20 +
21 +// Parameter option structure
22 +typedef struct mcp_function_param_option {
23 + STRING *id;
24 + STRING *name;
25 + STRING *info; // Additional information about the option (e.g., file count, size, coverage)
26 +} MCP_FUNCTION_PARAM_OPTION;
27 +
28 +// Parameter structure
29 +typedef struct mcp_function_param {
30 + STRING *id;
31 + STRING *name;
32 + STRING *help;
33 + MCP_REQUIRED_PARAMS_TYPE type;
34 + bool unique_view;
35 + size_t options_count;
36 + MCP_FUNCTION_PARAM_OPTION *options;
37 +} MCP_FUNCTION_PARAM;
38 +
39 +// Pagination structure for MCP-compliant cursor-based pagination
40 +typedef struct mcp_function_pagination {
41 + bool enabled; // whether pagination is supported
42 + STRING *key; // parameter name for pagination (e.g., "anchor")
43 + STRING *column; // column name in data (e.g., "timestamp")
44 + MCP_PAGINATION_UNITS units; // units of the column - only known types supported
45 +} MCP_FUNCTION_PAGINATION;
46 +
47 +// Registry entry structure
48 +typedef struct mcp_function_registry_entry {
49 + RW_SPINLOCK spinlock; // Read/write lock for thread-safe access
50 + SPINLOCK update_spinlock; // Spinlock to coordinate updates without blocking readers
51 + MCP_FUNCTION_TYPE type; // Function type (table, table with history, etc.)
52 + bool has_history; // whether the function supports history
53 + int update_every; // update interval in seconds
54 + STRING *help; // help text
55 + int version; // Function version (v3+ supports POST)
56 + bool supports_post; // true if version >= 3
57 + size_t required_params_count; // number of required parameters
58 + MCP_FUNCTION_PARAM *required_params; // array of required parameters
59 + // Supported optional parameters (detected from accepted_params)
60 + bool has_timeframe; // supports after, before parameters
61 + bool has_last; // supports last parameter (row limit)
62 + bool has_data_only; // supports data_only parameter
63 + bool has_direction; // supports direction parameter
64 + bool has_query; // supports query parameter for full-text search
65 + bool has_slice; // supports slice parameter for database-level filtering
66 + MCP_FUNCTION_PAGINATION pagination; // MCP-compliant cursor pagination support
67 + time_t last_update; // timestamp of last info update
68 + time_t expires; // expiration timestamp
69 +} MCP_FUNCTION_REGISTRY_ENTRY;
70 +
71 +// Initialize the functions registry
72 +void mcp_functions_registry_init(void);
73 +
74 +// Cleanup the functions registry
75 +void mcp_functions_registry_cleanup(void);
76 +
77 +// Get a registry entry for a function (read-locked)
78 +// This will fetch info if the entry doesn't exist or has expired
79 +// The returned entry is read-locked and MUST be released with mcp_functions_registry_release()
80 +// Returns NULL on error (with error details in the error buffer)
81 +MCP_FUNCTION_REGISTRY_ENTRY *mcp_functions_registry_get(RRDHOST *host, const char *function_name, BUFFER *error);
82 +
83 +// Release a registry entry (unlocks the read lock)
84 +// MUST be called after mcp_functions_registry_get() as soon as possible
85 +void mcp_functions_registry_release(MCP_FUNCTION_REGISTRY_ENTRY *entry);
86 +
87 +#endif // NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_REGISTRY_H
\ No newline at end of file
src/web/mcp/mcp-tools-execute-function.c new
+2701
@@ -0,0 +1,2701 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-tools-execute-function.h"
4 +#include "mcp-tools-execute-function-internal.h"
5 +#include "mcp-tools-execute-function-registry.h"
6 +#include "mcp-params.h"
7 +#include "database/rrdfunctions.h"
8 +
9 +// Analyze the JSON response and determine its type
10 +MCP_FUNCTION_TYPE mcp_functions_analyze_response(struct json_object *json_obj, int *out_status) {
11 + if (!json_obj) return FN_TYPE_UNKNOWN;
12 +
13 + struct json_object *type_obj = NULL;
14 + struct json_object *has_history_obj = NULL;
15 + struct json_object *status_obj = NULL;
16 +
17 + // Type is required
18 + if (!json_object_object_get_ex(json_obj, "type", &type_obj)) {
19 + return FN_TYPE_NOT_TABLE;
20 + }
21 +
22 + const char *type = json_object_get_string(type_obj);
23 + if (!type || strcmp(type, "table") != 0) {
24 + return FN_TYPE_NOT_TABLE;
25 + }
26 +
27 + // has_history is optional - assume false if missing
28 + bool has_history = false;
29 + if (json_object_object_get_ex(json_obj, "has_history", &has_history_obj)) {
30 + has_history = json_object_get_boolean(has_history_obj);
31 + }
32 +
33 + // Status is optional - assume 200 if missing
34 + int status = 200;
35 + if (json_object_object_get_ex(json_obj, "status", &status_obj)) {
36 + status = json_object_get_int(status_obj);
37 + }
38 +
39 + if (out_status) *out_status = status;
40 +
41 + // Return appropriate type based on has_history
42 + if (has_history) {
43 + return FN_TYPE_TABLE_WITH_HISTORY;
44 + } else {
45 + return FN_TYPE_TABLE;
46 + }
47 +}
48 +
49 +// Helper function to create a filtered copy of a column definition
50 +static struct json_object *create_filtered_column(struct json_object *col_obj, const char *col_id) {
51 + struct json_object *col_copy = json_object_new_object();
52 +
53 + // Handle type conversion for better LLM understanding
54 + struct json_object *type_obj = NULL;
55 + if (json_object_object_get_ex(col_obj, "type", &type_obj) &&
56 + json_object_is_type(type_obj, json_type_string)) {
57 + const char *type_str = json_object_get_string(type_obj);
58 + RRDF_FIELD_TYPE field_type = RRDF_FIELD_TYPE_2id(type_str);
59 +
60 + // Replace the original type with a simplified scalar type for LLM
61 + json_object_object_add(col_copy, "type",
62 + json_object_new_string(field_type_to_json_scalar_type(field_type)));
63 + }
64 +
65 + // Copy only necessary properties
66 + struct json_object_iterator col_it = json_object_iter_begin(col_obj);
67 + struct json_object_iterator col_itEnd = json_object_iter_end(col_obj);
68 +
69 + while (!json_object_iter_equal(&col_it, &col_itEnd)) {
70 + const char *field_key = json_object_iter_peek_name(&col_it);
71 + struct json_object *field_val = json_object_iter_peek_value(&col_it);
72 +
73 + // Skip properties we don't need for LLM
74 + if (strcmp(field_key, "visible") != 0 &&
75 + strcmp(field_key, "visualization") != 0 &&
76 + strcmp(field_key, "value_options") != 0 &&
77 + strcmp(field_key, "sort") != 0 &&
78 + strcmp(field_key, "sortable") != 0 &&
79 + strcmp(field_key, "sticky") != 0 &&
80 + strcmp(field_key, "summary") != 0 &&
81 + strcmp(field_key, "filter") != 0 &&
82 + strcmp(field_key, "full_width") != 0 &&
83 + strcmp(field_key, "wrap") != 0 &&
84 + strcmp(field_key, "default_expanded_filter") != 0 &&
85 + strcmp(field_key, "unique_key") != 0 &&
86 + // Skip type as we've already handled it
87 + strcmp(field_key, "type") != 0) {
88 +
89 + // Skip "name" field if it's the same as the column id
90 + if (strcmp(field_key, "name") == 0 && col_id) {
91 + if (json_object_is_type(field_val, json_type_string)) {
92 + const char *name_str = json_object_get_string(field_val);
93 + if (name_str && strcmp(col_id, name_str) == 0) {
94 + // Name is same as id, skip it
95 + json_object_iter_next(&col_it);
96 + continue;
97 + }
98 + }
99 + }
100 +
101 + json_object_object_add(col_copy, field_key, json_object_get(field_val));
102 + }
103 +
104 + json_object_iter_next(&col_it);
105 + }
106 +
107 + return col_copy;
108 +}
109 +
110 +
111 +// Structure to hold column type and transform information
112 +typedef struct column_transform_info {
113 + RRDF_FIELD_TYPE type;
114 + RRDF_FIELD_TRANSFORM transform;
115 +} COLUMN_TRANSFORM_INFO;
116 +
117 +// Check if a column represents a timestamp
118 +static inline bool is_transformable_timestamp(RRDF_FIELD_TYPE type, RRDF_FIELD_TRANSFORM transform) {
119 + return (type == RRDF_FIELD_TYPE_TIMESTAMP ||
120 + transform == RRDF_FIELD_TRANSFORM_DATETIME_MS ||
121 + transform == RRDF_FIELD_TRANSFORM_DATETIME_USEC);
122 +}
123 +
124 +// Check if a column represents a duration
125 +static inline bool is_transformable_duration(RRDF_FIELD_TYPE type, RRDF_FIELD_TRANSFORM transform) {
126 + (void)transform; // unused for duration
127 + return (type == RRDF_FIELD_TYPE_DURATION);
128 +}
129 +
130 +// Check if a column type/transform combination should be transformed to string
131 +static inline bool is_transformable_to_string(RRDF_FIELD_TYPE type, RRDF_FIELD_TRANSFORM transform) {
132 + return is_transformable_timestamp(type, transform) || is_transformable_duration(type, transform);
133 +}
134 +
135 +// Transform a value based on its type and transform settings
136 +// Returns either a new object (caller owns) or NULL if no transformation needed
137 +// IMPORTANT: When this returns a non-NULL value, it's a NEW object that the caller owns.
138 +// When it returns NULL, the caller should use json_object_get() on the original value
139 +// if they need to store it somewhere that takes ownership.
140 +static struct json_object *transform_value_for_mcp(struct json_object *val, RRDF_FIELD_TYPE type, RRDF_FIELD_TRANSFORM transform) {
141 + if (!val || json_object_is_type(val, json_type_null))
142 + return NULL;
143 +
144 + // If it's not an integer, no transformation possible
145 + if (!json_object_is_type(val, json_type_int))
146 + return NULL;
147 +
148 + int64_t num_val = json_object_get_int64(val);
149 +
150 + if (is_transformable_timestamp(type, transform)) {
151 + // Convert to microseconds based on transform
152 + usec_t usec_val;
153 + if (transform == RRDF_FIELD_TRANSFORM_DATETIME_MS) {
154 + usec_val = (usec_t)num_val * USEC_PER_MS;
155 + } else if (transform == RRDF_FIELD_TRANSFORM_DATETIME_USEC) {
156 + usec_val = (usec_t)num_val;
157 + } else {
158 + // Default: seconds
159 + usec_val = (usec_t)num_val * USEC_PER_SEC;
160 + }
161 +
162 + // Format as RFC3339
163 + char datetime_buf[RFC3339_MAX_LENGTH];
164 + rfc3339_datetime_ut(datetime_buf, sizeof(datetime_buf), usec_val, 0, true);
165 + return json_object_new_string(datetime_buf);
166 + } else if (is_transformable_duration(type, transform)) {
167 + // Duration is always in seconds
168 + char duration_buf[256];
169 + duration_snprintf_time_t(duration_buf, sizeof(duration_buf), (time_t)num_val);
170 + return json_object_new_string(duration_buf);
171 + }
172 +
173 + // Value is 0 or negative, no transformation
174 + return NULL;
175 +}
176 +
177 +// Extract column transform information from column definitions
178 +static void extract_column_transforms(struct json_object *columns_obj,
179 + const int *column_indices,
180 + char **column_names,
181 + size_t selected_count,
182 + COLUMN_TRANSFORM_INFO *col_transforms) {
183 + for (size_t i = 0; i < selected_count; i++) {
184 + int col_idx = column_indices[i];
185 + const char *col_name = column_names[col_idx];
186 +
187 + struct json_object *col_obj = NULL;
188 + if (json_object_object_get_ex(columns_obj, col_name, &col_obj)) {
189 + // Get type
190 + struct json_object *type_obj = NULL;
191 + if (json_object_object_get_ex(col_obj, "type", &type_obj) &&
192 + json_object_is_type(type_obj, json_type_string)) {
193 + const char *type_str = json_object_get_string(type_obj);
194 + col_transforms[i].type = RRDF_FIELD_TYPE_2id(type_str);
195 + }
196 +
197 + // Get transform from value_options
198 + struct json_object *value_options_obj = NULL;
199 + if (json_object_object_get_ex(col_obj, "value_options", &value_options_obj) &&
200 + json_object_is_type(value_options_obj, json_type_object)) {
201 + // value_options is an object, extract the "transform" field
202 + struct json_object *transform_obj = NULL;
203 + if (json_object_object_get_ex(value_options_obj, "transform", &transform_obj) &&
204 + json_object_is_type(transform_obj, json_type_string)) {
205 + const char *transform_str = json_object_get_string(transform_obj);
206 + col_transforms[i].transform = RRDF_FIELD_TRANSFORM_2id(transform_str);
207 + }
208 + }
209 + }
210 + }
211 +}
212 +
213 +// Convert string operator to enum type
214 +OPERATOR_TYPE mcp_functions_string_to_operator(const char *op_str) {
215 + if (!op_str)
216 + return OP_UNKNOWN;
217 +
218 + if (strcmp(op_str, "==") == 0)
219 + return OP_EQUALS;
220 +
221 + if (strcmp(op_str, "!=") == 0 || strcmp(op_str, "<>") == 0)
222 + return OP_NOT_EQUALS;
223 +
224 + if (strcmp(op_str, "<") == 0)
225 + return OP_LESS;
226 +
227 + if (strcmp(op_str, "<=") == 0)
228 + return OP_LESS_EQUALS;
229 +
230 + if (strcmp(op_str, ">") == 0)
231 + return OP_GREATER;
232 +
233 + if (strcmp(op_str, ">=") == 0)
234 + return OP_GREATER_EQUALS;
235 +
236 + if (strcmp(op_str, "match") == 0 || strcmp(op_str, "like") == 0 || strcmp(op_str, "in") == 0)
237 + return OP_MATCH;
238 +
239 + if (strcmp(op_str, "not match") == 0 || strcmp(op_str, "not like") == 0 || strcmp(op_str, "not in") == 0)
240 + return OP_NOT_MATCH;
241 +
242 + return OP_UNKNOWN;
243 +}
244 +
245 +// Free patterns in the condition array
246 +void mcp_functions_free_condition_patterns(CONDITION_ARRAY *condition_array) {
247 + if (!condition_array)
248 + return;
249 +
250 + for (size_t i = 0; i < condition_array->count; i++) {
251 + if (condition_array->items[i].pattern)
252 + simple_pattern_free(condition_array->items[i].pattern);
253 + }
254 +}
255 +
256 +// Check if a single value matches a condition
257 +static bool value_matches_condition(struct json_object *value, const CONDITION *condition)
258 +{
259 + if (!value || !condition)
260 + return false;
261 +
262 + // Handle NULL values
263 + if (json_object_is_type(value, json_type_null)) {
264 + if (condition->v_type == COND_VALUE_NULL) {
265 + return (condition->op == OP_EQUALS);
266 + } else {
267 + return (condition->op == OP_NOT_EQUALS);
268 + }
269 + }
270 +
271 + // Handle MATCH and NOT MATCH operators (pattern matching) - always convert to strings
272 + if (condition->op == OP_MATCH || condition->op == OP_NOT_MATCH) {
273 + if (condition->pattern) {
274 + const char *val_str = json_object_get_string(value);
275 + bool pattern_match = simple_pattern_matches(condition->pattern, val_str);
276 + return (condition->op == OP_MATCH) ? pattern_match : !pattern_match;
277 + }
278 + return false;
279 + }
280 +
281 + // Handle comparisons based on condition value type
282 + if (condition->v_type == COND_VALUE_NUMBER) {
283 + // Try to get numeric value from JSON
284 + double val_num = 0.0;
285 + if (json_object_is_type(value, json_type_int) || json_object_is_type(value, json_type_double)) {
286 + val_num = json_object_get_double(value);
287 + } else if (json_object_is_type(value, json_type_string)) {
288 + // Try to parse string as number
289 + char *endptr;
290 + const char *str = json_object_get_string(value);
291 + val_num = strtod(str, &endptr);
292 + if (endptr == str || *endptr != '\0') {
293 + // Not a valid number, do string comparison
294 + goto string_compare;
295 + }
296 + } else {
297 + // Can't convert to number, treat as not equal
298 + return (condition->op == OP_NOT_EQUALS);
299 + }
300 +
301 + switch (condition->op) {
302 + case OP_EQUALS:
303 + return (val_num == condition->v_num);
304 + case OP_NOT_EQUALS:
305 + return (val_num != condition->v_num);
306 + case OP_LESS:
307 + return (val_num < condition->v_num);
308 + case OP_LESS_EQUALS:
309 + return (val_num <= condition->v_num);
310 + case OP_GREATER:
311 + return (val_num > condition->v_num);
312 + case OP_GREATER_EQUALS:
313 + return (val_num >= condition->v_num);
314 + default:
315 + return false;
316 + }
317 + } else if (condition->v_type == COND_VALUE_BOOLEAN) {
318 + bool val_bool = json_object_get_boolean(value);
319 +
320 + switch (condition->op) {
321 + case OP_EQUALS:
322 + return (val_bool == condition->v_bool);
323 + case OP_NOT_EQUALS:
324 + return (val_bool != condition->v_bool);
325 + default:
326 + // Boolean doesn't support ordering comparisons
327 + return false;
328 + }
329 + }
330 +
331 +string_compare:
332 + {
333 + // String comparisons (including when condition is string or as fallback)
334 + const char *val_str = json_object_get_string(value);
335 + const char *cond_str = condition->v_str;
336 +
337 + // Handle NULL condition string
338 + if (!cond_str) {
339 + if (condition->v_type == COND_VALUE_NULL) {
340 + return (condition->op == OP_EQUALS && !val_str);
341 + }
342 + // Use empty string for comparison
343 + cond_str = "";
344 + }
345 +
346 + if (!val_str)
347 + val_str = "";
348 +
349 + int cmp = strcmp(val_str, cond_str);
350 +
351 + switch (condition->op) {
352 + case OP_EQUALS:
353 + return (cmp == 0);
354 + case OP_NOT_EQUALS:
355 + return (cmp != 0);
356 + case OP_LESS:
357 + return (cmp < 0);
358 + case OP_LESS_EQUALS:
359 + return (cmp <= 0);
360 + case OP_GREATER:
361 + return (cmp > 0);
362 + case OP_GREATER_EQUALS:
363 + return (cmp >= 0);
364 + default:
365 + return false;
366 + }
367 + }
368 +}
369 +
370 +// Check if a row matches all the conditions
371 +static bool row_matches_conditions(struct json_object *row, const CONDITION_ARRAY *conditions) {
372 + if (!conditions || conditions->count == 0)
373 + return true; // No conditions means everything matches
374 +
375 + for (size_t i = 0; i < conditions->count; i++) {
376 + const CONDITION *current = &conditions->items[i];
377 + bool condition_match = false;
378 +
379 + // Special case: column_index == -1 means search all columns
380 + if (current->column_index == -1) {
381 + // Search across all columns for a match
382 + size_t row_length = json_object_array_length(row);
383 + for (size_t col_idx = 0; col_idx < row_length; col_idx++) {
384 + struct json_object *row_val = json_object_array_get_idx(row, col_idx);
385 + if (!row_val) continue;
386 +
387 + // Check if this column value matches the condition
388 + if (value_matches_condition(row_val, current)) {
389 + condition_match = true;
390 + break; // Found a match, no need to check other columns
391 + }
392 + }
393 + }
394 + else {
395 + // Normal case: specific column index
396 + struct json_object *row_val = json_object_array_get_idx(row, current->column_index);
397 +
398 + // Handle null values
399 + if (!row_val) {
400 + return false;
401 + }
402 +
403 + condition_match = value_matches_condition(row_val, current);
404 + }
405 +
406 + // If any condition doesn't match, return false
407 + if (!condition_match) {
408 + return false;
409 + }
410 + }
411 +
412 + // All conditions matched
413 + return true;
414 +}
415 +
416 +// Resolve column indices for conditions that were parsed during request parsing
417 +static void resolve_condition_columns(CONDITION_ARRAY *condition_array, struct json_object *columns_obj) {
418 + if (!condition_array || !columns_obj || condition_array->count == 0)
419 + return;
420 +
421 + condition_array->has_missing_columns = false;
422 +
423 + for (size_t i = 0; i < condition_array->count; i++) {
424 + CONDITION *curr = &condition_array->items[i];
425 +
426 + // Check for special column names that mean "search all columns"
427 + bool is_all_columns = (strcmp(curr->column_name, "*") == 0 || strcmp(curr->column_name, "") == 0);
428 +
429 + // Find column in column definitions
430 + struct json_object *col_obj = NULL;
431 + if (is_all_columns || !json_object_object_get_ex(columns_obj, curr->column_name, &col_obj)) {
432 + // Column not found or explicitly all columns - mark it as a wildcard search (use -1 as special index)
433 + curr->column_index = -1;
434 +
435 + // Only report as missing if it's not a special "all columns" indicator
436 + if (!is_all_columns) {
437 + condition_array->has_missing_columns = true;
438 + }
439 + } else {
440 + // Get column index
441 + struct json_object *index_obj = NULL;
442 + if (json_object_object_get_ex(col_obj, "index", &index_obj)) {
443 + curr->column_index = json_object_get_int(index_obj);
444 + } else {
445 + // Column found but no index - treat as missing
446 + curr->column_index = -1;
447 + condition_array->has_missing_columns = true;
448 + }
449 + }
450 + }
451 +}
452 +
453 +// Flags to indicate what additional content should be added for errors
454 +typedef enum {
455 + MCP_TABLE_ADD_NOTHING = 0,
456 + MCP_TABLE_ADD_COLUMNS = 1 << 0,
457 + MCP_TABLE_ADD_RAW_DATA = 1 << 1,
458 + MCP_TABLE_ADD_FILTERING_INSTRUCTIONS = 1 << 2
459 +} MCP_TABLE_ADDITIONAL_CONTENT;
460 +
461 +// Forward declaration
462 +static usec_t calculate_next_cursor_from_input(MCP_FUNCTION_DATA *data);
463 +
464 +// Initialize MCP_FUNCTION_DATA structure
465 +void mcp_functions_data_init(MCP_FUNCTION_DATA *data) {
466 + memset(data, 0, sizeof(MCP_FUNCTION_DATA));
467 + data->output.result = buffer_create(0, NULL);
468 +}
469 +
470 +// Clean up MCP_FUNCTION_DATA structure
471 +void mcp_functions_data_cleanup(MCP_FUNCTION_DATA *data) {
472 + if (!data) return;
473 +
474 + // Free the parsed JSON object
475 + if (data->input.jobj) {
476 + json_object_put(data->input.jobj);
477 + data->input.jobj = NULL;
478 + }
479 +
480 + // Free the output result buffer
481 + if (data->output.result) {
482 + buffer_free(data->output.result);
483 + data->output.result = NULL;
484 + }
485 +
486 + // Free condition patterns
487 + mcp_functions_free_condition_patterns(&data->request.conditions);
488 +
489 + // Free pagination column string
490 + string_freez(data->pagination.column);
491 +
492 + // Free the input JSON buffer
493 + if (data->input.json) {
494 + buffer_free(data->input.json);
495 + data->input.json = NULL;
496 + }
497 +}
498 +
499 +// Reset output for reprocessing
500 +static void mcp_function_data_reset_output(MCP_FUNCTION_DATA *data) {
501 + buffer_flush(data->output.result);
502 + data->output.status = MCP_TABLE_OK;
503 + data->output.rows = 0;
504 + data->output.columns = 0;
505 +}
506 +
507 +// Helper to create a filtered columns object for error messages
508 +static struct json_object *create_filtered_columns_for_errors(struct json_object *columns_obj) {
509 + if (!columns_obj) return NULL;
510 +
511 + struct json_object *filtered = json_object_new_object();
512 + struct json_object_iterator it = json_object_iter_begin(columns_obj);
513 + struct json_object_iterator itEnd = json_object_iter_end(columns_obj);
514 +
515 + while (!json_object_iter_equal(&it, &itEnd)) {
516 + const char *col_name = json_object_iter_peek_name(&it);
517 + struct json_object *col_obj = json_object_iter_peek_value(&it);
518 +
519 + if (col_obj) {
520 + struct json_object *filtered_col = create_filtered_column(col_obj, col_name);
521 + json_object_object_add(filtered, col_name, filtered_col);
522 + }
523 +
524 + json_object_iter_next(&it);
525 + }
526 +
527 + return filtered;
528 +}
529 +
530 +// Helper function to generate comprehensive error messages for LLMs
531 +static MCP_TABLE_ADDITIONAL_CONTENT generate_table_error_message(MCP_FUNCTION_DATA *data) {
532 + buffer_flush(data->request.mcpc->error);
533 + MCP_TABLE_ADDITIONAL_CONTENT additional_content = MCP_TABLE_ADD_NOTHING;
534 +
535 + switch (data->output.status) {
536 + case MCP_TABLE_ERROR_INVALID_CONDITIONS:
537 + buffer_sprintf(data->request.mcpc->error,
538 + "Error processing conditions: %s\n\n"
539 + "Conditions should be formatted as:\n"
540 + "```json\n"
541 + "\"conditions\": [\n"
542 + " [\"column_name\", \"operator\", value],\n"
543 + " [\"another_column\", \"another_operator\", another_value]\n"
544 + "]\n"
545 + "```",
546 + buffer_tostring(data->output.result)
547 + );
548 + additional_content = MCP_TABLE_ADD_FILTERING_INSTRUCTIONS;
549 + break;
550 +
551 + case MCP_TABLE_ERROR_NO_MATCHES_WITH_MISSING_COLUMNS:
552 + buffer_strcat(data->request.mcpc->error,
553 + "No rows matched the specified conditions.\n\n"
554 + "Note: Some columns were not found, so a full-text search was performed across all columns, but no matches were found."
555 + );
556 + additional_content = MCP_TABLE_ADD_COLUMNS | MCP_TABLE_ADD_FILTERING_INSTRUCTIONS;
557 + break;
558 +
559 + case MCP_TABLE_ERROR_NO_MATCHES:
560 + buffer_strcat(data->request.mcpc->error,
561 + "No results match the specified conditions.\n\n"
562 + "Tips:\n"
563 + "• Verify the column names in your conditions\n"
564 + "• Check the values and operators used\n"
565 + "• For 'match' operators, ensure your pattern format is correct\n"
566 + "• To match multiple values, use 'match' with patterns separated by the pipe (|) character: '*value1*|*value2*'\n"
567 + "• Try broadening your filter criteria"
568 + );
569 + additional_content = MCP_TABLE_ADD_COLUMNS | MCP_TABLE_ADD_FILTERING_INSTRUCTIONS;
570 + break;
571 +
572 + case MCP_TABLE_ERROR_INVALID_SORT_ORDER:
573 + buffer_sprintf(data->request.mcpc->error,
574 + "Invalid sort_order: '%s'. Valid options are 'asc' (ascending) or 'desc' (descending).\n\n"
575 + "Example:\n"
576 + "```json\n"
577 + "\"sort_order\": \"desc\"\n"
578 + "```",
579 + buffer_tostring(data->output.result)
580 + );
581 + additional_content = MCP_TABLE_ADD_NOTHING;
582 + break;
583 +
584 + case MCP_TABLE_ERROR_COLUMNS_NOT_FOUND:
585 + buffer_sprintf(data->request.mcpc->error,
586 + "Column(s) not found: %s",
587 + buffer_tostring(data->output.result)
588 + );
589 + additional_content = MCP_TABLE_ADD_COLUMNS;
590 + break;
591 +
592 + case MCP_TABLE_ERROR_SORT_COLUMN_NOT_FOUND:
593 + buffer_sprintf(data->request.mcpc->error,
594 + "Sort column '%s' not found.",
595 + buffer_tostring(data->output.result)
596 + );
597 + additional_content = MCP_TABLE_ADD_COLUMNS;
598 + break;
599 +
600 + case MCP_TABLE_ERROR_TOO_MANY_COLUMNS:
601 + buffer_sprintf(data->request.mcpc->error,
602 + "Error: Table has %zu columns, which exceeds the maximum supported (%d). Showing raw output.",
603 + data->input.columns, MAX_COLUMNS
604 + );
605 + additional_content = MCP_TABLE_ADD_RAW_DATA;
606 + break;
607 +
608 + case MCP_TABLE_NOT_JSON:
609 + buffer_strcat(data->request.mcpc->error,
610 + "This response is not valid JSON. Showing raw output.");
611 + additional_content = MCP_TABLE_ADD_RAW_DATA;
612 + break;
613 +
614 + case MCP_TABLE_NOT_PROCESSABLE:
615 + buffer_strcat(data->request.mcpc->error,
616 + "The function returned JSON but it's not a table format we can filter. Showing raw output.");
617 + additional_content = MCP_TABLE_ADD_RAW_DATA;
618 + break;
619 +
620 + case MCP_TABLE_EMPTY_RESULT: {
621 + buffer_strcat(data->request.mcpc->error,
622 + "The function returned an empty result (no rows).");
623 +
624 + // Add contextual tips based on what parameters were used
625 + bool has_history = (data->input.type == FN_TYPE_TABLE_WITH_HISTORY);
626 + bool has_query = (data->request.query && *data->request.query);
627 + bool has_conditions = (data->request.conditions.count > (has_query ? 1 : 0));
628 +
629 + if(has_history || has_query || has_conditions) {
630 + buffer_strcat(data->request.mcpc->error, "\n\nTips:");
631 + }
632 +
633 + if (has_history)
634 + buffer_strcat(data->request.mcpc->error, "\n• Expand the search time window by adjusting 'after' and 'before' parameters");
635 +
636 + if (has_query)
637 + buffer_strcat(data->request.mcpc->error,
638 + "\n• Review the search query terms in 'q' parameter"
639 + "\n - Use wildcards: '*error*', '*warning*', '*fail*'"
640 + "\n - Combine terms: '*systemd*|*kernel*', '*eth0*|*eth1*'");
641 +
642 + if (has_conditions) {
643 + buffer_strcat(data->request.mcpc->error,
644 + "\n• Review the conditions - ensure column names and values match");
645 + }
646 +
647 + additional_content = MCP_TABLE_ADD_COLUMNS;
648 + break;
649 + }
650 +
651 + case MCP_TABLE_INFO_MISSING_COLUMNS_FOUND_RESULTS:
652 + buffer_strcat(data->request.mcpc->error,
653 + "Note: Not all columns in the conditions were found, so a full-text search was performed across all columns, and matching results were found.");
654 + additional_content = MCP_TABLE_ADD_NOTHING;
655 + break;
656 +
657 + case MCP_TABLE_RESPONSE_TOO_BIG:
658 + buffer_sprintf(data->request.mcpc->error,
659 + "The response is too big, having %zu rows and %zu columns. Limiting to 1 row for readability.",
660 + data->input.rows, data->input.columns
661 + );
662 + additional_content = MCP_TABLE_ADD_FILTERING_INSTRUCTIONS;
663 + break;
664 +
665 + default:
666 + additional_content = MCP_TABLE_ADD_NOTHING;
667 + break;
668 + }
669 +
670 + return additional_content;
671 +}
672 +
673 +// Helper to add filtering instructions as a separate content entry
674 +static void add_filtering_instructions_to_mcp_result(MCP_CLIENT *mcpc, bool has_history) {
675 + buffer_json_add_array_item_object(mcpc->result);
676 + {
677 + buffer_json_member_add_string(mcpc->result, "type", "text");
678 +
679 + if (has_history) {
680 + // Instructions for history/logs functions (limited capabilities)
681 + buffer_json_member_add_string(mcpc->result, "text",
682 + "FILTERING INSTRUCTIONS:\n"
683 + "• **columns**: Select specific columns to reduce width (e.g., [\"Column1\", \"Column2\", \"Column3\"])\n"
684 + "• **conditions**: Filter rows using exact matches and value sets:\n"
685 + " - Single value: [\"column\", \"==\", \"exact_value\"]\n"
686 + " - Multiple values: [\"column\", \"match\", \"value1|value2|value3\"] (values are OR'd)\n"
687 + "• **limit**: Control number of rows returned (e.g., 10)\n"
688 + "• **q**: Full-text search across all columns (simple patterns like \"*term1*|*term2*\")\n"
689 + "• **direction**: Controls time-based sorting (\"forward\" or \"backward\")\n"
690 + "\n"
691 + "Example filtering:\n"
692 + "```json\n"
693 + "{\n"
694 + " \"columns\": [\"MESSAGE\", \"PRIORITY\", \"_HOSTNAME\"],\n"
695 + " \"conditions\": [\n"
696 + " [\"PRIORITY\", \"match\", \"1|2|3\"],\n"
697 + " [\"_HOSTNAME\", \"=\", \"server1\"]\n"
698 + " ],\n"
699 + " \"q\": \"*systemd*|*logind*|*dbus*\",\n"
700 + " \"direction\": \"backward\",\n"
701 + " \"limit\": 20\n"
702 + "}\n"
703 + "```\n"
704 + "\n"
705 + "Valid operators for history functions: == (exact match), match (value set with | separator)\n"
706 + "Invalid operators: !=, <>, not match, <, <=, >, >=\n"
707 + "Full-text search: Use 'q' parameter with wildcards like '*pattern1*|*pattern2*' to search all columns\n"
708 + "Sorting: Use 'direction' parameter only - column sorting is not supported for history functions"
709 + );
710 + } else {
711 + // Instructions for regular functions (full capabilities)
712 + buffer_json_member_add_string(mcpc->result, "text",
713 + "FILTERING INSTRUCTIONS:\n"
714 + "• **columns**: Select specific columns to reduce width (e.g., [\"Column1\", \"Column2\", \"Column3\"])\n"
715 + "• **conditions**: Filter rows using [ [column1, operator1, value1], [column2, operator2, value2], ... ]\n"
716 + "• **limit**: Control number of rows returned (e.g., 10)\n"
717 + "• **q**: Full-text search across all columns (supports wildcards like \"*term1*|*term2*\")\n"
718 + "• **sort_column** + **sort_order**: Order results by a column ('asc' or 'desc')\n"
719 + "\n"
720 + "Example filtering:\n"
721 + "```json\n"
722 + "{\n"
723 + " \"columns\": [\"CmdLine\", \"CPU\", \"Memory\", \"Status\"],\n"
724 + " \"conditions\": [\n"
725 + " [\"Memory\", \">\", 1.0],\n"
726 + " [\"CmdLine\", \"match\", \"*systemd*|*postgresql*|*docker*\"],\n"
727 + " ],\n"
728 + " \"sort_column\": \"CPU\",\n"
729 + " \"sort_order\": \"desc\",\n"
730 + " \"limit\": 10\n"
731 + "}\n"
732 + "```\n"
733 + "\n"
734 + "Operators: ==, !=, <, <=, >, >=, match (simple pattern), not match (simple pattern)\n"
735 + "Simple patterns: '*this*|*that*|*other*' (wildcard search to find strings that include 'this', or 'that', or 'other')\n"
736 + );
737 + }
738 + }
739 + buffer_json_object_close(mcpc->result);
740 +}
741 +
742 +// Helper to add columns info as a separate content entry
743 +static void add_columns_info_to_mcp_result(MCP_CLIENT *mcpc, struct json_object *columns_obj) {
744 + if (!columns_obj) return;
745 +
746 + struct json_object *filtered_columns = create_filtered_columns_for_errors(columns_obj);
747 + if (filtered_columns) {
748 + // Create wrapper object
749 + struct json_object *wrapper = json_object_new_object();
750 + json_object_object_add(wrapper, "available_columns", filtered_columns);
751 +
752 + const char *columns_json = json_object_to_json_string_ext(wrapper, JSON_C_TO_STRING_PRETTY);
753 +
754 + buffer_json_add_array_item_object(mcpc->result);
755 + {
756 + buffer_json_member_add_string(mcpc->result, "type", "text");
757 + buffer_json_member_add_string(mcpc->result, "text", columns_json);
758 + }
759 + buffer_json_object_close(mcpc->result);
760 +
761 + json_object_put(wrapper);
762 + }
763 +}
764 +
765 +// Helper to add messages to MCP result based on table result status
766 +static void add_table_messages_to_mcp_result(MCP_FUNCTION_DATA *data,
767 + struct json_object *columns_obj) {
768 + // Generate the appropriate error message and get additional content flags
769 + MCP_TABLE_ADDITIONAL_CONTENT additional_content = generate_table_error_message(data);
770 +
771 + // Add the message if there's an error or guidance
772 + if (data->output.status != MCP_TABLE_OK && buffer_strlen(data->request.mcpc->error) > 0) {
773 + buffer_json_add_array_item_object(data->request.mcpc->result);
774 + {
775 + buffer_json_member_add_string(data->request.mcpc->result, "type", "text");
776 + buffer_json_member_add_string(data->request.mcpc->result, "text", buffer_tostring(data->request.mcpc->error));
777 + }
778 + buffer_json_object_close(data->request.mcpc->result);
779 + }
780 +
781 + // Add columns info if requested
782 + if ((additional_content & MCP_TABLE_ADD_COLUMNS) && columns_obj) {
783 + add_columns_info_to_mcp_result(data->request.mcpc, columns_obj);
784 + }
785 +
786 + // Add filtering instructions if requested
787 + if (additional_content & MCP_TABLE_ADD_FILTERING_INSTRUCTIONS) {
788 + bool has_history = (data->input.type == FN_TYPE_TABLE_WITH_HISTORY);
789 + add_filtering_instructions_to_mcp_result(data->request.mcpc, has_history);
790 + }
791 +
792 + // Add raw data if requested
793 + if ((additional_content & MCP_TABLE_ADD_RAW_DATA) &&
794 + buffer_strlen(data->output.result) > 0) {
795 + buffer_json_add_array_item_object(data->request.mcpc->result);
796 + {
797 + buffer_json_member_add_string(data->request.mcpc->result, "type", "text");
798 + buffer_json_member_add_string(data->request.mcpc->result, "text", buffer_tostring(data->output.result));
799 + }
800 + buffer_json_object_close(data->request.mcpc->result);
801 + }
802 +}
803 +
804 +void mcp_tool_execute_function_schema(BUFFER *buffer) {
805 + // Tool input schema
806 + buffer_json_member_add_object(buffer, "inputSchema");
807 + buffer_json_member_add_string(buffer, "type", "object");
808 + buffer_json_member_add_string(
809 + buffer, "title",
810 + "Execute a function on a specific node. "
811 + "Functions provide live information and they are automatically routed "
812 + "and executed to Netdata running on the given node.");
813 +
814 + // Properties
815 + buffer_json_member_add_object(buffer, "properties");
816 +
817 + buffer_json_member_add_object(buffer, "node");
818 + {
819 + buffer_json_member_add_string(buffer, "type", "string");
820 + buffer_json_member_add_string(buffer, "title", "The node on which to execute the function");
821 + buffer_json_member_add_string(
822 + buffer, "description",
823 + "The hostname or machine_guid or node_id of the node where the function should be executed. "
824 + "The node needs to be online (live) and reachable.");
825 + }
826 + buffer_json_object_close(buffer); // node
827 +
828 + buffer_json_member_add_object(buffer, "function");
829 + {
830 + buffer_json_member_add_string(buffer, "type", "string");
831 + buffer_json_member_add_string(buffer, "title", "The name of the function to execute.");
832 + buffer_json_member_add_string(buffer, "description", "The function name, as available in the node_details tool output");
833 + }
834 + buffer_json_object_close(buffer); // function
835 +
836 + mcp_schema_add_timeout(
837 + buffer, "timeout", "Execution timeout in seconds",
838 + "Maximum time to wait for function execution (default: 60)",
839 + 60, 1, 3600, false);
840 +
841 + buffer_json_member_add_object(buffer, "columns");
842 + {
843 + buffer_json_member_add_string(buffer, "type", "array");
844 + buffer_json_member_add_string(buffer, "title", "Columns to include");
845 + buffer_json_member_add_string(
846 + buffer, "description",
847 + "Array of column names to include in the result. "
848 + "Each function has its own columns, so first check the function without this parameter.");
849 +
850 + buffer_json_member_add_object(buffer, "items");
851 + {
852 + buffer_json_member_add_string(buffer, "type", "string");
853 + }
854 + buffer_json_object_close(buffer); // items
855 + }
856 + buffer_json_object_close(buffer); // columns
857 +
858 + mcp_schema_add_string_param(
859 + buffer, "sort_column", "Column to sort by",
860 + "Name of the column to sort the results by.",
861 + NULL, false);
862 +
863 + buffer_json_member_add_object(buffer, "sort_order");
864 + {
865 + buffer_json_member_add_string(buffer, "type", "string");
866 + buffer_json_member_add_string(buffer, "title", "Sort order");
867 + buffer_json_member_add_string(
868 + buffer, "description",
869 + "Order to sort results: 'asc' for ascending, 'desc' for descending");
870 + buffer_json_member_add_string(buffer, "default", "desc");
871 + buffer_json_member_add_array(buffer, "enum");
872 + buffer_json_add_array_item_string(buffer, "asc");
873 + buffer_json_add_array_item_string(buffer, "desc");
874 + buffer_json_array_close(buffer);
875 + }
876 + buffer_json_object_close(buffer); // sort_order
877 +
878 + mcp_schema_add_size_param(
879 + buffer, "limit", "Row limit",
880 + "Maximum number of rows to return",
881 + 0, 0, SIZE_MAX, false);
882 +
883 + // Time-based parameters for functions with history
884 + mcp_schema_add_time_param(
885 + buffer, "after", "Start time",
886 + "Start time for query window (timestamp in seconds or RFC3339 datetime string)",
887 + "now", 0, false);
888 +
889 + mcp_schema_add_time_param(
890 + buffer, "before", "End time",
891 + "End time for query window (timestamp in seconds or RFC3339 datetime string)",
892 + "now", 0, false);
893 +
894 + mcp_schema_add_string_param(
895 + buffer, "cursor", "Pagination cursor",
896 + "Opaque cursor for pagination (follows MCP standard)",
897 + NULL, false);
898 +
899 + mcp_schema_add_size_param(
900 + buffer, "limit", "Entries to return",
901 + "Number of entries to return",
902 + 0, 0, SIZE_MAX, false);
903 +
904 + buffer_json_member_add_object(buffer, "direction");
905 + {
906 + buffer_json_member_add_string(buffer, "type", "string");
907 + buffer_json_member_add_string(buffer, "title", "Query direction");
908 + buffer_json_member_add_string(
909 + buffer, "description",
910 + "Direction for query processing: 'forward' (oldest first) or 'backward' (newest first)");
911 + buffer_json_member_add_string(buffer, "default", "backward");
912 + buffer_json_member_add_array(buffer, "enum");
913 + buffer_json_add_array_item_string(buffer, "forward");
914 + buffer_json_add_array_item_string(buffer, "backward");
915 + buffer_json_array_close(buffer);
916 + }
917 + buffer_json_object_close(buffer); // direction
918 +
919 + mcp_schema_add_string_param(
920 + buffer, "q", "Full-text search",
921 + "Full-text search to filter results. Use pipe character (|) to separate multiple search patterns. "
922 + "Example: '*fail*|*error*|*systemd*'. "
923 + "Wildcards (*) are supported for pattern matching.",
924 + NULL, false);
925 +
926 + buffer_json_member_add_object(buffer, "conditions");
927 + {
928 + buffer_json_member_add_string(buffer, "type", "array");
929 + buffer_json_member_add_string(buffer, "title", "Filter conditions");
930 + buffer_json_member_add_string(
931 + buffer, "description",
932 + "Array of conditions to filter rows. "
933 + "Each condition is an array of [column, operator, value] where operator "
934 + "can be ==, !=, <>, <, <=, >, >=, match, not match. "
935 + "Use '*' or '' (empty string) as column name to search across all columns.");
936 +
937 + buffer_json_member_add_object(buffer, "items");
938 + {
939 + buffer_json_member_add_string(buffer, "type", "array");
940 + buffer_json_member_add_object(buffer, "items");
941 + {
942 + buffer_json_member_add_array(buffer, "oneOf");
943 + {
944 + // First item of the condition array - column name
945 + buffer_json_add_array_item_object(buffer);
946 + {
947 + buffer_json_member_add_string(buffer, "type", "string");
948 + }
949 + buffer_json_object_close(buffer);
950 +
951 + // Second item - operator
952 + buffer_json_add_array_item_object(buffer);
953 + {
954 + buffer_json_member_add_string(buffer, "type", "string");
955 + buffer_json_member_add_array(buffer, "enum");
956 + {
957 + buffer_json_add_array_item_string(buffer, "==");
958 + buffer_json_add_array_item_string(buffer, "!=");
959 + buffer_json_add_array_item_string(buffer, "<>");
960 + buffer_json_add_array_item_string(buffer, "<");
961 + buffer_json_add_array_item_string(buffer, "<=");
962 + buffer_json_add_array_item_string(buffer, ">");
963 + buffer_json_add_array_item_string(buffer, ">=");
964 + buffer_json_add_array_item_string(buffer, "match");
965 + buffer_json_add_array_item_string(buffer, "not match");
966 + }
967 + buffer_json_array_close(buffer);
968 + }
969 + buffer_json_object_close(buffer);
970 +
971 + // Third item - value (can be string, number, or boolean)
972 + buffer_json_add_array_item_object(buffer);
973 + {
974 + buffer_json_member_add_array(buffer, "oneOf");
975 + {
976 + buffer_json_add_array_item_object(buffer);
977 + {
978 + buffer_json_member_add_string(buffer, "type", "string");
979 + }
980 + buffer_json_object_close(buffer);
981 +
982 + buffer_json_add_array_item_object(buffer);
983 + {
984 + buffer_json_member_add_string(buffer, "type", "number");
985 + }
986 + buffer_json_object_close(buffer);
987 +
988 + buffer_json_add_array_item_object(buffer);
989 + {
990 + buffer_json_member_add_string(buffer, "type", "boolean");
991 + }
992 + buffer_json_object_close(buffer);
993 + }
994 + buffer_json_array_close(buffer);
995 + }
996 + buffer_json_object_close(buffer); // third item
997 + }
998 + buffer_json_array_close(buffer); // oneOf
999 + }
1000 + buffer_json_object_close(buffer); // inner items
1001 + }
1002 + buffer_json_object_close(buffer); // items
1003 + }
1004 + buffer_json_object_close(buffer); // conditions
1005 +
1006 + buffer_json_member_add_object(buffer, "selections");
1007 + {
1008 + buffer_json_member_add_string(buffer, "type", "object");
1009 + buffer_json_member_add_string(buffer, "title", "Function parameter selections");
1010 + buffer_json_member_add_string(
1011 + buffer, "description",
1012 + "Key-value pairs where each key is a parameter name and the value depends on the parameter type: "
1013 + "for 'select' type parameters, use a single string value; "
1014 + "for 'multiselect' type parameters, use an array of strings. "
1015 + "Functions that require selections will prompt you with available options when called without this parameter. "
1016 + "Example: {\"param1\": \"single_value\", \"param2\": [\"value1\", \"value2\"]}");
1017 + }
1018 + buffer_json_object_close(buffer); // selections
1019 +
1020 + buffer_json_object_close(buffer); // properties
1021 +
1022 + // Required fields
1023 + buffer_json_member_add_array(buffer, "required");
1024 + buffer_json_add_array_item_string(buffer, "node");
1025 + buffer_json_add_array_item_string(buffer, "function");
1026 + buffer_json_array_close(buffer); // required
1027 +
1028 + buffer_json_object_close(buffer); // inputSchema
1029 +}
1030 +
1031 +/**
1032 + * Filter and sort a table-formatted JSON result based on parameters
1033 + *
1034 + * @param data Function data structure with request context, input and output
1035 + * @param max_size_threshold Maximum size in bytes before truncation is recommended
1036 + *
1037 + * @return void (result is returned in data->output)
1038 + */
1039 +static void mcp_process_table_result(MCP_FUNCTION_DATA *data, size_t max_size_threshold)
1040 +{
1041 + // Reset output for processing
1042 + mcp_function_data_reset_output(data);
1043 +
1044 + // Clear error buffer for any error messages that might be generated
1045 + buffer_flush(data->request.mcpc->error);
1046 +
1047 + // Get original JSON string from input
1048 + const char *json_str = buffer_tostring(data->input.json);
1049 + size_t result_size = buffer_strlen(data->input.json);
1050 +
1051 + if (!data->input.jobj) {
1052 + buffer_strcat(data->output.result, json_str); // Return original if JSON is NULL
1053 + data->output.status = MCP_TABLE_NOT_JSON;
1054 + return;
1055 + }
1056 +
1057 + // Check if it's a processable table format
1058 + if (data->input.type != FN_TYPE_TABLE && data->input.type != FN_TYPE_TABLE_WITH_HISTORY) {
1059 + buffer_strcat(data->output.result, json_str); // Not a processable table format
1060 + data->output.status = MCP_TABLE_NOT_PROCESSABLE;
1061 + return;
1062 + }
1063 +
1064 + // Get data and columns
1065 + struct json_object *data_obj = NULL;
1066 + struct json_object *columns_obj = NULL;
1067 +
1068 + if (!json_object_object_get_ex(data->input.jobj, "data", &data_obj) ||
1069 + !json_object_object_get_ex(data->input.jobj, "columns", &columns_obj)) {
1070 + buffer_strcat(data->output.result, json_str); // Missing required elements
1071 + return;
1072 + }
1073 +
1074 + size_t row_count = json_object_array_length(data_obj);
1075 + size_t column_count = json_object_object_length(columns_obj);
1076 +
1077 + // Store original counts in input
1078 + data->input.rows = row_count;
1079 + data->input.columns = column_count;
1080 +
1081 + // Check if we need to show guidance - only when no filtering is specified
1082 + if (result_size > max_size_threshold && data->request.columns.count == 0 &&
1083 + (data->request.conditions.count == 0 || (data->request.conditions.count == 1 && data->request.query && *data->request.query))) {
1084 + // Store first row in result for guidance
1085 + data->output.status = MCP_TABLE_RESPONSE_TOO_BIG;
1086 + data->request.limit = 1;
1087 + }
1088 +
1089 + // Even with no filtering parameters, we need to process to remove unwanted fields
1090 +
1091 + // Sort direction is already parsed in data->request.sort.descending
1092 +
1093 + // Use fixed-size arrays for column selection
1094 + uint8_t column_selected[MAX_COLUMNS] = {0}; // 1 if column is selected, 0 if not
1095 + char *column_names[MAX_COLUMNS] = {0}; // Names of selected columns (references)
1096 + int column_indices[MAX_COLUMNS] = {0}; // Index mapping for ordered access
1097 + size_t selected_count = 0;
1098 +
1099 + // Check if we have too many columns
1100 + if (column_count > MAX_COLUMNS) {
1101 + data->output.status = MCP_TABLE_ERROR_TOO_MANY_COLUMNS;
1102 + data->output.columns = column_count;
1103 +
1104 + return;
1105 + }
1106 +
1107 + if (data->request.columns.count > 0) {
1108 + // Process the pre-parsed column names
1109 + bool all_columns_found = true;
1110 + CLEAN_BUFFER *missing_columns = buffer_create(0, NULL);
1111 +
1112 + for (size_t i = 0; i < data->request.columns.count; i++) {
1113 + const char *col = data->request.columns.array[i];
1114 +
1115 + // Find column and add to selected set
1116 + struct json_object *col_obj = NULL;
1117 + if (json_object_object_get_ex(columns_obj, col, &col_obj)) {
1118 + // Get index of this column
1119 + struct json_object *index_obj = NULL;
1120 + if (json_object_object_get_ex(col_obj, "index", &index_obj)) {
1121 + int idx = json_object_get_int(index_obj);
1122 + if (idx >= 0 && idx < MAX_COLUMNS) {
1123 + column_selected[idx] = 1;
1124 + column_names[idx] = (char*)col; // Store reference only - columns_obj must stay alive
1125 + }
1126 + }
1127 + } else {
1128 + // Column not found
1129 + all_columns_found = false;
1130 + if (buffer_strlen(missing_columns) > 0) {
1131 + buffer_strcat(missing_columns, ", ");
1132 + }
1133 + buffer_strcat(missing_columns, col);
1134 + }
1135 + }
1136 +
1137 + // If any columns were not found, set error status
1138 + if (!all_columns_found) {
1139 + data->output.status = MCP_TABLE_ERROR_COLUMNS_NOT_FOUND;
1140 + buffer_strcat(data->output.result, buffer_tostring(missing_columns));
1141 +
1142 + return;
1143 + }
1144 + } else {
1145 + // If no columns specified, include all
1146 + struct json_object_iterator it = json_object_iter_begin(columns_obj);
1147 + struct json_object_iterator itEnd = json_object_iter_end(columns_obj);
1148 +
1149 + while (!json_object_iter_equal(&it, &itEnd)) {
1150 + const char *col_key = json_object_iter_peek_name(&it);
1151 + struct json_object *col_val = json_object_iter_peek_value(&it);
1152 +
1153 + struct json_object *index_obj = NULL;
1154 + if (json_object_object_get_ex(col_val, "index", &index_obj)) {
1155 + int idx = json_object_get_int(index_obj);
1156 + if (idx >= 0 && idx < MAX_COLUMNS) {
1157 + column_selected[idx] = 1;
1158 + column_names[idx] = (char*)col_key; // Store reference only - columns_obj must stay alive
1159 + }
1160 + }
1161 +
1162 + json_object_iter_next(&it);
1163 + }
1164 + }
1165 +
1166 + // Create ordered index mapping for selected columns
1167 + for (int i = 0; i < MAX_COLUMNS; i++) {
1168 + if (column_selected[i]) {
1169 + column_indices[selected_count] = i;
1170 + selected_count++;
1171 + }
1172 + }
1173 +
1174 + // Find sort column index if sort requested
1175 + int sort_idx = -1;
1176 +
1177 + if (data->request.sort.column) {
1178 + struct json_object *sort_col_obj = NULL;
1179 + if (json_object_object_get_ex(columns_obj, data->request.sort.column, &sort_col_obj)) {
1180 + struct json_object *index_obj = NULL;
1181 + if (json_object_object_get_ex(sort_col_obj, "index", &index_obj)) {
1182 + sort_idx = json_object_get_int(index_obj);
1183 + }
1184 + } else {
1185 + // Column not found
1186 + data->output.status = MCP_TABLE_ERROR_SORT_COLUMN_NOT_FOUND;
1187 + buffer_strcat(data->output.result, data->request.sort.column);
1188 +
1189 + return;
1190 + }
1191 + }
1192 +
1193 + // Create new arrays for filtered/sorted data
1194 + struct json_object **rows = (struct json_object **)callocz(row_count, sizeof(struct json_object *));
1195 + size_t row_idx = 0;
1196 +
1197 + // Resolve column indices for pre-parsed conditions if they exist
1198 + if (data->request.conditions.count > 0) {
1199 + resolve_condition_columns(&data->request.conditions, columns_obj);
1200 + }
1201 +
1202 + // Copy rows for filtering/sorting, applying filters if specified
1203 + for (size_t i = 0; i < row_count; i++) {
1204 + struct json_object *row = json_object_array_get_idx(data_obj, i);
1205 + if (!row)
1206 + continue;
1207 +
1208 + bool include_row = true;
1209 +
1210 + // Apply preprocessed conditions if they exist
1211 + if (data->request.conditions.count > 0) {
1212 + include_row = row_matches_conditions(row, &data->request.conditions);
1213 + }
1214 +
1215 + if (include_row) {
1216 + rows[row_idx++] = row;
1217 + }
1218 + }
1219 +
1220 + // If we have missing columns and found no matches, provide helpful error
1221 + if (data->request.conditions.has_missing_columns && row_idx == 0) {
1222 + data->output.status = MCP_TABLE_ERROR_NO_MATCHES_WITH_MISSING_COLUMNS;
1223 +
1224 + // Clean up and return
1225 + freez((void *)rows);
1226 +
1227 + return;
1228 + }
1229 +
1230 + // Sort if requested
1231 + if (sort_idx >= 0) {
1232 + // Simple bubble sort implementation
1233 + for (size_t i = 0; i < row_idx; i++) {
1234 + for (size_t j = i + 1; j < row_idx; j++) {
1235 + bool should_swap = false;
1236 +
1237 + struct json_object *val_i = json_object_array_get_idx(rows[i], sort_idx);
1238 + struct json_object *val_j = json_object_array_get_idx(rows[j], sort_idx);
1239 +
1240 + // Handle null values
1241 + if (!val_i && !val_j) {
1242 + should_swap = false;
1243 + } else if (!val_i) {
1244 + should_swap = !data->request.sort.descending;
1245 + } else if (!val_j) {
1246 + should_swap = data->request.sort.descending;
1247 + } else {
1248 + // Try to match based on apparent type
1249 + // Check if either value is a number type
1250 + if (json_object_is_type(val_i, json_type_int) ||
1251 + json_object_is_type(val_i, json_type_double) ||
1252 + json_object_is_type(val_j, json_type_int) ||
1253 + json_object_is_type(val_j, json_type_double)) {
1254 +
1255 + // Let json-c do the type conversion
1256 + double i_val = json_object_get_double(val_i);
1257 + double j_val = json_object_get_double(val_j);
1258 + should_swap = data->request.sort.descending ? (i_val < j_val) : (i_val > j_val);
1259 +
1260 + } else if (json_object_is_type(val_i, json_type_boolean) ||
1261 + json_object_is_type(val_j, json_type_boolean)) {
1262 +
1263 + // Let json-c do the type conversion
1264 + bool i_val = json_object_get_boolean(val_i);
1265 + bool j_val = json_object_get_boolean(val_j);
1266 + should_swap = data->request.sort.descending ? (i_val && !j_val) : (!i_val && j_val);
1267 +
1268 + } else {
1269 + // Default to string comparison for everything else
1270 + int cmp = strcmp(json_object_get_string(val_i), json_object_get_string(val_j));
1271 + should_swap = data->request.sort.descending ? (cmp < 0) : (cmp > 0);
1272 + }
1273 + }
1274 +
1275 + if (should_swap) {
1276 + struct json_object *temp = rows[i];
1277 + rows[i] = rows[j];
1278 + rows[j] = temp;
1279 + }
1280 + }
1281 + }
1282 + }
1283 +
1284 + // Apply row limit
1285 + size_t limit = row_idx;
1286 + if (data->request.limit > 0 && data->request.limit < limit && data->input.type == FN_TYPE_TABLE) {
1287 + // we don't limit history functions, only regular tables
1288 + // for history functions, we sent the limit to the backend
1289 + // so whatever it returns is what we show
1290 + // This is important, otherwise the cursor will be wrong!
1291 + limit = data->request.limit;
1292 + }
1293 +
1294 + // Create new filtered result
1295 + struct json_object *filtered_result = json_object_new_object();
1296 + struct json_object *filtered_data = json_object_new_array();
1297 + struct json_object *filtered_columns = json_object_new_object();
1298 +
1299 + // Copy only specific metadata fields from original
1300 + {
1301 + // Keep only status, type, update_every, has_history + data and columns
1302 + const char *keep_fields[] = {"status", "type", "update_every", "has_history"};
1303 + size_t keep_count = sizeof(keep_fields) / sizeof(keep_fields[0]);
1304 +
1305 + for (size_t i = 0; i < keep_count; i++) {
1306 + struct json_object *field_obj = NULL;
1307 +
1308 + if (json_object_object_get_ex(data->input.jobj, keep_fields[i], &field_obj)) {
1309 + json_object_object_add(filtered_result, keep_fields[i], json_object_get(field_obj));
1310 + }
1311 + }
1312 + }
1313 +
1314 + // Extract column transform information BEFORE creating filtered columns
1315 + COLUMN_TRANSFORM_INFO col_transforms[MAX_COLUMNS] = {{0}};
1316 + extract_column_transforms(columns_obj, column_indices, column_names, selected_count, col_transforms);
1317 +
1318 + // Create filtered data rows with transformation
1319 + for (size_t i = 0; i < limit; i++) {
1320 + struct json_object *row = rows[i];
1321 + struct json_object *new_row = json_object_new_array();
1322 +
1323 + // Extract only selected columns
1324 + for (size_t j = 0; j < selected_count; j++) {
1325 + int col_idx = column_indices[j];
1326 + struct json_object *val = json_object_array_get_idx(row, col_idx);
1327 +
1328 + // Try to transform the value
1329 + struct json_object *transformed = transform_value_for_mcp(val, col_transforms[j].type, col_transforms[j].transform);
1330 +
1331 + if (transformed) {
1332 + // Use the transformed value (we own it)
1333 + json_object_array_add(new_row, transformed);
1334 + } else if (val) {
1335 + // No transformation needed, use original with ref count increase
1336 + // json_object_array_add takes ownership, so we must increment ref count
1337 + json_object_array_add(new_row, json_object_get(val));
1338 + } else {
1339 + // NULL value
1340 + json_object_array_add(new_row, NULL);
1341 + }
1342 + }
1343 +
1344 + json_object_array_add(filtered_data, new_row);
1345 + }
1346 +
1347 + // Create filtered column definitions AFTER processing data
1348 + for (size_t i = 0; i < selected_count; i++) {
1349 + int col_idx = column_indices[i];
1350 + const char *col_name = column_names[col_idx];
1351 +
1352 + struct json_object *col_obj = NULL;
1353 + if (json_object_object_get_ex(columns_obj, col_name, &col_obj)) {
1354 + struct json_object *col_copy = create_filtered_column(col_obj, col_name);
1355 +
1356 + // Update index to match new position
1357 + json_object_object_add(col_copy, "index", json_object_new_int((int)i));
1358 +
1359 + // Check if this column was transformed to string
1360 + if (is_transformable_to_string(col_transforms[i].type, col_transforms[i].transform)) {
1361 + // Override type to string since we transformed the value
1362 + json_object_object_del(col_copy, "type");
1363 + json_object_object_add(col_copy, "type", json_object_new_string("string"));
1364 +
1365 + // Remove numeric-specific properties that don't make sense for strings
1366 + json_object_object_del(col_copy, "max");
1367 + json_object_object_del(col_copy, "min");
1368 + json_object_object_del(col_copy, "units");
1369 + }
1370 +
1371 + json_object_object_add(filtered_columns, col_name, col_copy);
1372 + }
1373 + }
1374 +
1375 + // Add filtered data and columns to result
1376 + json_object_object_add(filtered_result, "data", filtered_data);
1377 + json_object_object_add(filtered_result, "columns", filtered_columns);
1378 +
1379 + // Check if we found any rows
1380 + if (limit == 0 && data->request.conditions.count > 0) {
1381 + // No rows matched the conditions
1382 + json_object_put(filtered_result);
1383 + data->output.status = MCP_TABLE_ERROR_NO_MATCHES;
1384 + } else {
1385 + // Set status flag if we used wildcard search and found results
1386 + if (data->request.conditions.has_missing_columns && row_idx > 0) {
1387 + data->output.status = MCP_TABLE_INFO_MISSING_COLUMNS_FOUND_RESULTS;
1388 + }
1389 +
1390 + // Add nextCursor for pagination if applicable (only for successful results)
1391 + if (data->pagination.enabled && data->input.rows > 0) {
1392 + usec_t next_cursor_timestamp = calculate_next_cursor_from_input(data);
1393 + if (next_cursor_timestamp > 0) {
1394 + CLEAN_BUFFER *cursor_str = buffer_create(0, NULL);
1395 + buffer_sprintf(cursor_str, "%" PRIu64, next_cursor_timestamp);
1396 + json_object_object_add(filtered_result, "nextCursor", json_object_new_string(buffer_tostring(cursor_str)));
1397 + }
1398 + }
1399 +
1400 + // Convert to string and store result
1401 + const char *filtered_json = json_object_to_json_string_ext(filtered_result, JSON_C_TO_STRING_PRETTY);
1402 + buffer_strcat(data->output.result, filtered_json);
1403 +
1404 + // Update actual counts from filtered result
1405 + data->output.rows = limit;
1406 + data->output.columns = selected_count;
1407 +
1408 + // Free the filtered result
1409 + json_object_put(filtered_result);
1410 + }
1411 +
1412 + // Clean up
1413 + freez((void *)rows);
1414 +
1415 +}
1416 +
1417 +// Parse conditions without column information (early parsing during request)
1418 +static MCP_RETURN_CODE mcp_parse_conditions_early(CONDITION_ARRAY *condition_array, struct json_object *conditions_json, BUFFER *error_buffer) {
1419 + if (!condition_array || !conditions_json)
1420 + return MCP_RC_ERROR;
1421 +
1422 + // Initialize the condition array
1423 + memset(condition_array, 0, sizeof(CONDITION_ARRAY));
1424 +
1425 + if (!json_object_is_type(conditions_json, json_type_array))
1426 + return MCP_RC_INVALID_PARAMS;
1427 +
1428 + size_t conditions_count = json_object_array_length(conditions_json);
1429 + if (conditions_count == 0)
1430 + return MCP_RC_OK; // Empty array is valid
1431 +
1432 + if (conditions_count > MAX_CONDITIONS) {
1433 + if (error_buffer)
1434 + buffer_sprintf(error_buffer, "Too many conditions. Maximum is %d.", MAX_CONDITIONS);
1435 + return MCP_RC_INVALID_PARAMS;
1436 + }
1437 +
1438 + for (size_t i = 0; i < conditions_count; i++) {
1439 + struct json_object *condition = json_object_array_get_idx(conditions_json, i);
1440 +
1441 + // Each condition should be an array of [column, operator, value]
1442 + if (!condition || !json_object_is_type(condition, json_type_array) ||
1443 + json_object_array_length(condition) != 3) {
1444 + if (error_buffer)
1445 + buffer_sprintf(error_buffer, "Invalid condition format at index %zu. Expected [column, operator, value]", i);
1446 + mcp_functions_free_condition_patterns(condition_array);
1447 + return MCP_RC_INVALID_PARAMS;
1448 + }
1449 +
1450 + struct json_object *col_name_obj = json_object_array_get_idx(condition, 0);
1451 + struct json_object *operator_obj = json_object_array_get_idx(condition, 1);
1452 + struct json_object *value_obj = json_object_array_get_idx(condition, 2);
1453 +
1454 + if (!col_name_obj || !json_object_is_type(col_name_obj, json_type_string) ||
1455 + !operator_obj || !json_object_is_type(operator_obj, json_type_string) ||
1456 + !value_obj) {
1457 + if (error_buffer)
1458 + buffer_sprintf(error_buffer, "Invalid condition element types at index %zu. Expected [string, string, any]", i);
1459 + mcp_functions_free_condition_patterns(condition_array);
1460 + return MCP_RC_INVALID_PARAMS;
1461 + }
1462 +
1463 + CONDITION *curr = &condition_array->items[condition_array->count];
1464 +
1465 + // Get column name and operator
1466 + curr->column_name = json_object_get_string(col_name_obj);
1467 + const char *op_str = json_object_get_string(operator_obj);
1468 +
1469 + curr->op = mcp_functions_string_to_operator(op_str);
1470 + if (curr->op == OP_UNKNOWN) {
1471 + if (error_buffer)
1472 + buffer_sprintf(error_buffer, "Invalid operator '%s' at index %zu. Valid operators are: ==, !=, <>, <, <=, >, >=, match, not match",
1473 + op_str, i);
1474 + mcp_functions_free_condition_patterns(condition_array);
1475 + return MCP_RC_INVALID_PARAMS;
1476 + }
1477 +
1478 + // Parse and store the value based on its type
1479 + if (json_object_is_type(value_obj, json_type_null)) {
1480 + curr->v_type = COND_VALUE_NULL;
1481 + } else if (json_object_is_type(value_obj, json_type_boolean)) {
1482 + curr->v_type = COND_VALUE_BOOLEAN;
1483 + curr->v_bool = json_object_get_boolean(value_obj);
1484 + } else if (json_object_is_type(value_obj, json_type_int) || json_object_is_type(value_obj, json_type_double)) {
1485 + curr->v_type = COND_VALUE_NUMBER;
1486 + curr->v_num = json_object_get_double(value_obj);
1487 + } else {
1488 + // Everything else is treated as string
1489 + curr->v_type = COND_VALUE_STRING;
1490 + curr->v_str = json_object_get_string(value_obj);
1491 + }
1492 +
1493 + // Pre-compile patterns for MATCH operators
1494 + if (curr->op == OP_MATCH || curr->op == OP_NOT_MATCH) {
1495 + const char *pattern_str = NULL;
1496 + if (curr->v_type == COND_VALUE_STRING) {
1497 + pattern_str = curr->v_str;
1498 + } else {
1499 + // For non-string types, use json-c's string conversion
1500 + pattern_str = json_object_get_string(value_obj);
1501 + }
1502 + if (pattern_str) {
1503 + curr->pattern = string_to_simple_pattern_nocase_substring(pattern_str);
1504 + }
1505 + }
1506 +
1507 + // Mark column index as unknown for now (-1)
1508 + curr->column_index = -1;
1509 +
1510 + condition_array->count++;
1511 + }
1512 +
1513 + return MCP_RC_OK;
1514 +}
1515 +
1516 +// Build the function name with GET parameters appended
1517 +static void build_function_name_with_params(BUFFER *dest, const char *function_name, struct json_object *selections, MCP_FUNCTION_DATA *data, MCP_FUNCTION_REGISTRY_ENTRY *entry) {
1518 + buffer_strcat(dest, function_name);
1519 +
1520 + // Add time-based parameters if supported and specified
1521 + if (entry->has_timeframe) {
1522 + buffer_sprintf(dest, " after:%ld", data->request.after);
1523 + buffer_sprintf(dest, " before:%ld", data->request.before);
1524 + }
1525 +
1526 + if (entry->pagination.enabled && data->request.anchor > 0) {
1527 + buffer_sprintf(dest, " %s:%llu", string2str(entry->pagination.key), (unsigned long long)data->request.anchor);
1528 + }
1529 +
1530 + if (entry->has_last && data->request.last > 0) {
1531 + buffer_sprintf(dest, " last:%zu", data->request.last);
1532 + }
1533 +
1534 + if (entry->has_direction && data->request.direction && *data->request.direction) {
1535 + buffer_sprintf(dest, " direction:%s", data->request.direction);
1536 + }
1537 +
1538 + if (entry->has_query && data->request.query && *data->request.query) {
1539 + buffer_sprintf(dest, " query:%s", data->request.query);
1540 + }
1541 +
1542 + if (entry->has_data_only) {
1543 + buffer_sprintf(dest, " data_only:true");
1544 + }
1545 +
1546 + if (entry->has_slice) {
1547 + buffer_sprintf(dest, " slice:true");
1548 + }
1549 +
1550 + // Add selections parameters
1551 + if (selections && json_object_is_type(selections, json_type_object)) {
1552 + struct json_object_iterator it = json_object_iter_begin(selections);
1553 + struct json_object_iterator itEnd = json_object_iter_end(selections);
1554 +
1555 + while (!json_object_iter_equal(&it, &itEnd)) {
1556 + const char *key = json_object_iter_peek_name(&it);
1557 + struct json_object *val = json_object_iter_peek_value(&it);
1558 +
1559 + if (!val) {
1560 + json_object_iter_next(&it);
1561 + continue;
1562 + }
1563 +
1564 + buffer_sprintf(dest, " %s:", key);
1565 +
1566 + if (json_object_is_type(val, json_type_string)) {
1567 + // Single string value
1568 + buffer_strcat(dest, json_object_get_string(val));
1569 + } else if (json_object_is_type(val, json_type_array)) {
1570 + // Array of values
1571 + size_t array_len = json_object_array_length(val);
1572 + for (size_t i = 0; i < array_len; i++) {
1573 + if (i > 0) buffer_strcat(dest, ",");
1574 + struct json_object *item = json_object_array_get_idx(val, i);
1575 + if (item && json_object_is_type(item, json_type_string)) {
1576 + buffer_strcat(dest, json_object_get_string(item));
1577 + }
1578 + }
1579 + }
1580 +
1581 + json_object_iter_next(&it);
1582 + }
1583 + }
1584 +}
1585 +
1586 +// Calculate nextCursor from original input data for MCP-compliant pagination
1587 +// Returns the next cursor timestamp, or 0 if not applicable
1588 +static usec_t calculate_next_cursor_from_input(MCP_FUNCTION_DATA *data) {
1589 + if (!data->pagination.enabled || data->input.rows == 0 || !data->input.jobj) {
1590 + return 0;
1591 + }
1592 +
1593 + // Only handle timestamp_usec units for now
1594 + if (data->pagination.units != MCP_PAGINATION_UNITS_TIMESTAMP_USEC) {
1595 + return 0;
1596 + }
1597 +
1598 + // Get the data array from original input
1599 + struct json_object *data_array;
1600 + if (!json_object_object_get_ex(data->input.jobj, "data", &data_array) ||
1601 + !json_object_is_type(data_array, json_type_array)) {
1602 + return 0;
1603 + }
1604 +
1605 + // Get the columns object to find the timestamp column
1606 + struct json_object *columns_obj;
1607 + if (!json_object_object_get_ex(data->input.jobj, "columns", &columns_obj)) {
1608 + return 0;
1609 + }
1610 +
1611 + // Find the timestamp column index - handle both array and object formats
1612 + int timestamp_column_index = -1;
1613 + const char *timestamp_column_name = string2str(data->pagination.column);
1614 +
1615 + if (json_object_is_type(columns_obj, json_type_array)) {
1616 + // Array format: ["timestamp", "field1", "field2"]
1617 + size_t columns_count = json_object_array_length(columns_obj);
1618 + for (size_t i = 0; i < columns_count; i++) {
1619 + struct json_object *col_obj = json_object_array_get_idx(columns_obj, i);
1620 + if (!col_obj) continue;
1621 +
1622 + if (json_object_is_type(col_obj, json_type_string)) {
1623 + const char *col_name = json_object_get_string(col_obj);
1624 + if (strcmp(col_name, timestamp_column_name) == 0) {
1625 + timestamp_column_index = (int)i;
1626 + break;
1627 + }
1628 + }
1629 + }
1630 + } else if (json_object_is_type(columns_obj, json_type_object)) {
1631 + // Object format: {"timestamp": {...}, "field1": {...}}
1632 + // Need to find the index by iterating through keys in order
1633 + struct json_object_iterator it = json_object_iter_begin(columns_obj);
1634 + struct json_object_iterator it_end = json_object_iter_end(columns_obj);
1635 + int index = 0;
1636 +
1637 + while (!json_object_iter_equal(&it, &it_end)) {
1638 + const char *key = json_object_iter_peek_name(&it);
1639 + if (key && strcmp(key, timestamp_column_name) == 0) {
1640 + timestamp_column_index = index;
1641 + break;
1642 + }
1643 + json_object_iter_next(&it);
1644 + index++;
1645 + }
1646 + } else {
1647 + return 0;
1648 + }
1649 +
1650 + if (timestamp_column_index == -1) {
1651 + return 0;
1652 + }
1653 +
1654 + // Extract timestamps from all rows in original data
1655 + size_t rows_count = json_object_array_length(data_array);
1656 + if (rows_count == 0) {
1657 + return 0;
1658 + }
1659 +
1660 + usec_t min_timestamp = UINT64_MAX;
1661 + usec_t max_timestamp = 0;
1662 + bool found_any = false;
1663 +
1664 + for (size_t i = 0; i < rows_count; i++) {
1665 + struct json_object *row_obj = json_object_array_get_idx(data_array, i);
1666 + if (!row_obj || !json_object_is_type(row_obj, json_type_array)) {
1667 + continue;
1668 + }
1669 +
1670 + size_t row_length = json_object_array_length(row_obj);
1671 + if ((size_t)timestamp_column_index >= row_length) {
1672 + continue;
1673 + }
1674 +
1675 + struct json_object *timestamp_obj = json_object_array_get_idx(row_obj, timestamp_column_index);
1676 + if (!timestamp_obj) {
1677 + continue;
1678 + }
1679 +
1680 + usec_t timestamp;
1681 + if (json_object_is_type(timestamp_obj, json_type_int)) {
1682 + timestamp = (usec_t)json_object_get_int64(timestamp_obj);
1683 + } else if (json_object_is_type(timestamp_obj, json_type_string)) {
1684 + const char *timestamp_str = json_object_get_string(timestamp_obj);
1685 + timestamp = str2ull(timestamp_str, NULL);
1686 + } else {
1687 + continue;
1688 + }
1689 +
1690 + if (timestamp < min_timestamp) {
1691 + min_timestamp = timestamp;
1692 + }
1693 + if (timestamp > max_timestamp) {
1694 + max_timestamp = timestamp;
1695 + }
1696 + found_any = true;
1697 + }
1698 +
1699 + if (!found_any) {
1700 + return 0;
1701 + }
1702 +
1703 + // Determine direction - default to forward if not specified
1704 + bool is_backward = (data->request.direction && strcmp(data->request.direction, "backward") == 0);
1705 +
1706 + // Calculate nextCursor based on direction:
1707 + // - Forward: use max timestamp + 1
1708 + // - Backward: use min timestamp - 1
1709 + usec_t next_cursor;
1710 + if (is_backward) {
1711 + next_cursor = (min_timestamp > 0) ? min_timestamp - 1 : 0;
1712 + } else {
1713 + next_cursor = (max_timestamp < UINT64_MAX) ? max_timestamp + 1 : 0;
1714 + }
1715 +
1716 + return next_cursor;
1717 +}
1718 +
1719 +// Build POST payload for v3+ functions
1720 +// Format: { "after": timestamp, "before": timestamp, "last": N, "data_only": true, "selections": { "key1": ["value1", "value2"] } }
1721 +static BUFFER *build_post_payload_with_selections(struct json_object *selections, MCP_FUNCTION_DATA *data, MCP_FUNCTION_REGISTRY_ENTRY *entry) {
1722 + BUFFER *payload = buffer_create(0, NULL);
1723 + buffer_json_initialize(payload, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
1724 +
1725 + // Add time-based parameters if supported and specified
1726 + if (entry->has_timeframe && data->request.after != 0) {
1727 + buffer_json_member_add_time_t(payload, "after", data->request.after);
1728 + }
1729 +
1730 + if (entry->has_timeframe && data->request.before != 0) {
1731 + buffer_json_member_add_time_t(payload, "before", data->request.before);
1732 + }
1733 +
1734 + if (entry->pagination.enabled && data->request.anchor > 0) {
1735 + buffer_json_member_add_uint64(payload, string2str(entry->pagination.key), data->request.anchor);
1736 + }
1737 +
1738 + if (entry->has_last && data->request.last > 0) {
1739 + buffer_json_member_add_uint64(payload, "last", data->request.last);
1740 + }
1741 +
1742 + if (entry->has_direction && data->request.direction) {
1743 + buffer_json_member_add_string(payload, "direction", data->request.direction);
1744 + }
1745 +
1746 + if (entry->has_query && data->request.query) {
1747 + buffer_json_member_add_string(payload, "query", data->request.query);
1748 + }
1749 +
1750 + if (entry->has_data_only) {
1751 + buffer_json_member_add_boolean(payload, "data_only", true);
1752 + }
1753 +
1754 + if (entry->has_slice) {
1755 + buffer_json_member_add_boolean(payload, "slice", true);
1756 + }
1757 +
1758 + // Build selections object - start with provided selections and add condition-based selections for has_history functions
1759 + bool selections_added = false;
1760 +
1761 + // For has_history functions, add selections based on conditions with == or match operators
1762 + if (entry->has_history && data->request.conditions.count > 0) {
1763 + buffer_json_member_add_object(payload, "selections");
1764 + selections_added = true;
1765 +
1766 + for (size_t i = 0; i < data->request.conditions.count; i++) {
1767 + CONDITION *cond = &data->request.conditions.items[i];
1768 +
1769 + // Skip wildcard searches and unsupported operators (these were validated earlier)
1770 + if (cond->column_index == -1 || (cond->op != OP_EQUALS && cond->op != OP_MATCH)) {
1771 + continue;
1772 + }
1773 +
1774 + const char *column_name = cond->column_name;
1775 +
1776 + if (cond->op == OP_EQUALS && cond->v_type == COND_VALUE_STRING) {
1777 + // Single value equality: key == value -> "key": ["value"]
1778 + buffer_json_member_add_array(payload, column_name);
1779 + buffer_json_add_array_item_string(payload, cond->v_str);
1780 + buffer_json_array_close(payload);
1781 + }
1782 + else if (cond->op == OP_MATCH && cond->v_type == COND_VALUE_STRING) {
1783 + // Pattern match without wildcards: key match a|b|c -> "key": ["a", "b", "c"]
1784 + const char *pattern_str = cond->v_str;
1785 +
1786 + // Parse the pipe-separated values
1787 + buffer_json_member_add_array(payload, column_name);
1788 +
1789 + CLEAN_BUFFER *temp_value = buffer_create(0, NULL);
1790 + const char *current = pattern_str;
1791 + const char *pipe_pos;
1792 +
1793 + while ((pipe_pos = strchr(current, '|')) != NULL) {
1794 + // Extract value between current and pipe_pos
1795 + buffer_flush(temp_value);
1796 + buffer_strncat(temp_value, current, pipe_pos - current);
1797 +
1798 + // Add to array if non-empty
1799 + if (buffer_strlen(temp_value) > 0) {
1800 + buffer_json_add_array_item_string(payload, buffer_tostring(temp_value));
1801 + }
1802 +
1803 + current = pipe_pos + 1;
1804 + }
1805 +
1806 + // Add the last value after the final pipe (or the only value if no pipes)
1807 + if (*current) {
1808 + buffer_json_add_array_item_string(payload, current);
1809 + }
1810 +
1811 + buffer_json_array_close(payload);
1812 + }
1813 + }
1814 + }
1815 +
1816 + // Add user-provided selections if provided
1817 + if (selections && json_object_is_type(selections, json_type_object)) {
1818 + if (!selections_added) {
1819 + buffer_json_member_add_object(payload, "selections");
1820 + selections_added = true;
1821 + }
1822 +
1823 + struct json_object_iterator it = json_object_iter_begin(selections);
1824 + struct json_object_iterator itEnd = json_object_iter_end(selections);
1825 +
1826 + while (!json_object_iter_equal(&it, &itEnd)) {
1827 + const char *key = json_object_iter_peek_name(&it);
1828 + struct json_object *val = json_object_iter_peek_value(&it);
1829 +
1830 + if (!val) {
1831 + json_object_iter_next(&it);
1832 + continue;
1833 + }
1834 +
1835 + if (json_object_is_type(val, json_type_string)) {
1836 + // Single string value - convert to array for consistency
1837 + buffer_json_member_add_array(payload, key);
1838 + buffer_json_add_array_item_string(payload, json_object_get_string(val));
1839 + buffer_json_array_close(payload);
1840 + } else if (json_object_is_type(val, json_type_array)) {
1841 + // Array of values
1842 + buffer_json_member_add_array(payload, key);
1843 +
1844 + size_t array_len = json_object_array_length(val);
1845 + for (size_t i = 0; i < array_len; i++) {
1846 + struct json_object *item = json_object_array_get_idx(val, i);
1847 + if (item && json_object_is_type(item, json_type_string)) {
1848 + buffer_json_add_array_item_string(payload, json_object_get_string(item));
1849 + }
1850 + }
1851 +
1852 + buffer_json_array_close(payload);
1853 + }
1854 +
1855 + json_object_iter_next(&it);
1856 + }
1857 + }
1858 +
1859 + if (selections_added) {
1860 + buffer_json_object_close(payload); // close "selections"
1861 + }
1862 +
1863 + buffer_json_finalize(payload);
1864 +
1865 + return payload;
1866 +}
1867 +
1868 +// Generate helpful message about required parameters
1869 +static void generate_required_params_message(BUFFER *message, MCP_FUNCTION_REGISTRY_ENTRY *entry,
1870 + bool missing_timeframe, BUFFER *missing_params) {
1871 + bool has_missing_params = (buffer_strlen(missing_params) > 0);
1872 +
1873 + if (!missing_timeframe && !has_missing_params) {
1874 + return; // Nothing to show
1875 + }
1876 +
1877 + buffer_sprintf(message,
1878 + "This function has some required parameters.\n"
1879 + "Please repeat the request adding the following:\n\n");
1880 +
1881 + // Add timeframe requirements ONLY if timeframe is missing
1882 + if (missing_timeframe && entry->has_timeframe) {
1883 + buffer_strcat(message,
1884 + "TIMEFRAME PARAMETERS (Required):\n"
1885 + "- 'after': Start time (timestamp in seconds or RFC3339 datetime string)\n"
1886 + "- 'before': End time (timestamp in seconds or RFC3339 datetime string)\n");
1887 +
1888 + // Add optional time-based parameters that are commonly used with history functions
1889 + if (entry->has_direction) {
1890 + buffer_strcat(message, "- 'direction': Query direction ('forward' or 'backward')\n");
1891 + }
1892 + if (entry->has_last) {
1893 + buffer_strcat(message, "- 'limit': Maximum number of entries to return\n");
1894 + }
1895 +
1896 + buffer_strcat(message, "\n");
1897 + }
1898 +
1899 + // Only show sections for parameters that are actually missing
1900 + if (has_missing_params) {
1901 + size_t selection_count = 0;
1902 + for (size_t i = 0; i < entry->required_params_count; i++) {
1903 + MCP_FUNCTION_PARAM *param = &entry->required_params[i];
1904 + const char *param_id = string2str(param->id);
1905 +
1906 + // Check if this parameter is in the missing list
1907 + if (strstr(buffer_tostring(missing_params), param_id)) {
1908 + selection_count++;
1909 + buffer_sprintf(message, "SELECTION %zu:\n", selection_count);
1910 + buffer_sprintf(message, "Description: %s\n", string2str(param->help));
1911 + buffer_sprintf(message, "key: '%s'\n", param_id);
1912 +
1913 + if (param->type == MCP_REQUIRED_PARAMS_TYPE_SELECT) {
1914 + buffer_strcat(message, "With one of the following values:\n");
1915 + } else {
1916 + buffer_strcat(message, "With one or more of the following values:\n");
1917 + }
1918 +
1919 + for (size_t j = 0; j < param->options_count; j++) {
1920 + const char *id = string2str(param->options[j].id);
1921 + const char *name = string2str(param->options[j].name);
1922 + const char *info = string2str(param->options[j].info);
1923 +
1924 + // Only show name if it's different from id
1925 + bool show_name = (name && *name && strcmp(id, name) != 0);
1926 +
1927 + if (info && *info) {
1928 + if (show_name) {
1929 + buffer_sprintf(message, " - '%s': %s (%s)\n", id, name, info);
1930 + } else {
1931 + buffer_sprintf(message, " - '%s' (%s)\n", id, info);
1932 + }
1933 + } else {
1934 + if (show_name) {
1935 + buffer_sprintf(message, " - '%s': %s\n", id, name);
1936 + } else {
1937 + buffer_sprintf(message, " - '%s'\n", id);
1938 + }
1939 + }
1940 + }
1941 +
1942 + buffer_strcat(message, "\n");
1943 + }
1944 + }
1945 + }
1946 +
1947 + // Only show example if there are missing parameters
1948 + if (missing_timeframe || has_missing_params) {
1949 + buffer_strcat(message, "\nExample:\n\n```json\n{\n");
1950 +
1951 + // Add timeframe parameters to example ONLY if they're missing
1952 + if (missing_timeframe && entry->has_timeframe) {
1953 + buffer_strcat(message, " \"after\": 1648627200,\n");
1954 + buffer_strcat(message, " \"before\": 1648630800,\n");
1955 +
1956 + // Add optional time-based parameters to the example
1957 + if (entry->has_direction) {
1958 + buffer_strcat(message, " \"direction\": \"backward\",\n");
1959 + }
1960 + if (entry->has_last) {
1961 + buffer_strcat(message, " \"limit\": 100,\n");
1962 + }
1963 + }
1964 +
1965 + // Add selections ONLY for missing required params
1966 + if (has_missing_params) {
1967 + buffer_strcat(message, " \"selections\": {\n");
1968 +
1969 + bool first_param = true;
1970 + for (size_t i = 0; i < entry->required_params_count; i++) {
1971 + MCP_FUNCTION_PARAM *param = &entry->required_params[i];
1972 + const char *param_id = string2str(param->id);
1973 +
1974 + // Only include this parameter if it's missing
1975 + if (strstr(buffer_tostring(missing_params), param_id)) {
1976 + if (!first_param) buffer_strcat(message, ",\n");
1977 + first_param = false;
1978 +
1979 + if (param->type == MCP_REQUIRED_PARAMS_TYPE_SELECT) {
1980 + // For select type, show single value (not array)
1981 + if (param->options_count > 0) {
1982 + buffer_sprintf(message, " \"%s\": \"%s\"",
1983 + param_id,
1984 + string2str(param->options[0].id));
1985 + } else {
1986 + buffer_sprintf(message, " \"%s\": \"value\"", param_id);
1987 + }
1988 + } else {
1989 + // For multiselect type, show array
1990 + buffer_sprintf(message, " \"%s\": [", param_id);
1991 + if (param->options_count > 0) {
1992 + // Show first few options as examples
1993 + size_t examples = param->options_count > 2 ? 2 : param->options_count;
1994 + for (size_t j = 0; j < examples; j++) {
1995 + if (j > 0) buffer_strcat(message, ", ");
1996 + buffer_sprintf(message, "\"%s\"", string2str(param->options[j].id));
1997 + }
1998 + } else {
1999 + buffer_strcat(message, "\"value1\", \"value2\"");
2000 + }
2001 + buffer_strcat(message, "]");
2002 + }
2003 + }
2004 + }
2005 +
2006 + buffer_strcat(message, "\n }");
2007 + }
2008 +
2009 + buffer_strcat(message, "\n}\n```");
2010 + }
2011 +}
2012 +
2013 +// Parse the request parameters and populate the MCP_FUNCTION_DATA structure
2014 +static MCP_RETURN_CODE mcp_parse_function_request(MCP_FUNCTION_DATA *data, MCP_CLIENT *mcpc, struct json_object *params) {
2015 + if (!data || !mcpc || !params)
2016 + return MCP_RC_ERROR;
2017 +
2018 + // Set request context
2019 + data->request.mcpc = mcpc;
2020 + data->request.params = params;
2021 + data->request.auth = mcpc->user_auth;
2022 +
2023 + // Parse required parameter: node
2024 + struct json_object *obj = NULL;
2025 + if (json_object_object_get_ex(params, "node", &obj) &&
2026 + json_object_is_type(obj, json_type_string)) {
2027 + data->request.node = json_object_get_string(obj);
2028 + }
2029 +
2030 + if (!data->request.node || !*data->request.node) {
2031 + buffer_sprintf(mcpc->error, "Missing required parameter 'node'");
2032 + return MCP_RC_BAD_REQUEST;
2033 + }
2034 +
2035 + // Parse required parameter: function
2036 + if (json_object_object_get_ex(params, "function", &obj) &&
2037 + json_object_is_type(obj, json_type_string)) {
2038 + data->request.function = json_object_get_string(obj);
2039 + }
2040 +
2041 + if (!data->request.function || !*data->request.function) {
2042 + buffer_sprintf(mcpc->error, "Missing required parameter 'function'");
2043 + return MCP_RC_BAD_REQUEST;
2044 + }
2045 +
2046 + // Parse timeout parameter
2047 + data->request.timeout = (time_t)mcp_params_extract_timeout(params, "timeout", 60, 1, 3600, mcpc->error);
2048 + if (buffer_strlen(mcpc->error) > 0) {
2049 + return MCP_RC_BAD_REQUEST;
2050 + }
2051 +
2052 + // Parse optional filtering parameters
2053 +
2054 + // Parse time-based parameters
2055 + data->request.after = mcp_params_parse_time(params, "after", 0);
2056 + data->request.before = mcp_params_parse_time(params, "before", 0);
2057 +
2058 + // Check if timeframe parameters are required but missing (will be validated later with registry entry)
2059 +
2060 + // Parse cursor parameter
2061 + data->request.cursor = mcp_params_extract_string(params, "cursor", NULL);
2062 +
2063 + // Convert cursor to internal anchor timestamp
2064 + data->request.anchor = 0; // Will be set if cursor is valid
2065 + if (data->request.cursor && *data->request.cursor) {
2066 + // For now, assume cursor is a timestamp_usec string
2067 + char *endptr;
2068 + usec_t cursor_timestamp = strtoull(data->request.cursor, &endptr, 10);
2069 + if (*endptr == '\0' && cursor_timestamp > 0) {
2070 + data->request.anchor = cursor_timestamp;
2071 + }
2072 + }
2073 + if (buffer_strlen(mcpc->error) > 0) {
2074 + return MCP_RC_BAD_REQUEST;
2075 + }
2076 +
2077 + data->request.last = (size_t)mcp_params_extract_size(params, "limit", 0, 0, SIZE_MAX, mcpc->error);
2078 + if (buffer_strlen(mcpc->error) > 0) {
2079 + return MCP_RC_BAD_REQUEST;
2080 + }
2081 +
2082 + // direction parameter
2083 + if (json_object_object_get_ex(params, "direction", &obj) &&
2084 + json_object_is_type(obj, json_type_string)) {
2085 + const char *direction = json_object_get_string(obj);
2086 + if (direction && (strcmp(direction, "forward") == 0 || strcmp(direction, "backward") == 0)) {
2087 + data->request.direction = direction;
2088 + } else if (direction) {
2089 + buffer_sprintf(mcpc->error, "Invalid direction: '%s'. Valid options are 'forward' or 'backward'.", direction);
2090 + return MCP_RC_BAD_REQUEST;
2091 + }
2092 + }
2093 +
2094 + // query parameter for full-text search
2095 + if (json_object_object_get_ex(params, "q", &obj) &&
2096 + json_object_is_type(obj, json_type_string)) {
2097 + const char *query = json_object_get_string(obj);
2098 + if (query && *query) {
2099 + data->request.query = query;
2100 + }
2101 + }
2102 +
2103 +
2104 + // columns array - parse it early
2105 + if (json_object_object_get_ex(params, "columns", &obj) &&
2106 + json_object_is_type(obj, json_type_array) &&
2107 + json_object_array_length(obj) > 0) {
2108 + size_t count = json_object_array_length(obj);
2109 + if (count > MAX_SELECTED_COLUMNS) {
2110 + buffer_sprintf(mcpc->error, "Too many columns requested. Maximum is %d.", MAX_SELECTED_COLUMNS);
2111 + return MCP_RC_BAD_REQUEST;
2112 + }
2113 +
2114 + data->request.columns.count = 0;
2115 + for (size_t i = 0; i < count; i++) {
2116 + struct json_object *col_obj = json_object_array_get_idx(obj, i);
2117 + if (col_obj && json_object_is_type(col_obj, json_type_string)) {
2118 + data->request.columns.array[data->request.columns.count++] = json_object_get_string(col_obj);
2119 + }
2120 + }
2121 + }
2122 +
2123 + // sort_column
2124 + if (json_object_object_get_ex(params, "sort_column", &obj) &&
2125 + json_object_is_type(obj, json_type_string)) {
2126 + const char *value = json_object_get_string(obj);
2127 + if (value && *value) {
2128 + data->request.sort.column = value;
2129 + }
2130 + }
2131 +
2132 + // sort_order and parse it to boolean
2133 + data->request.sort.descending = true; // default to DESC
2134 + if (json_object_object_get_ex(params, "sort_order", &obj) &&
2135 + json_object_is_type(obj, json_type_string)) {
2136 + const char *sort_order = json_object_get_string(obj);
2137 + if (sort_order) {
2138 + if (strcasecmp(sort_order, "asc") == 0) {
2139 + data->request.sort.descending = false;
2140 + } else if (strcasecmp(sort_order, "desc") != 0) {
2141 + // Invalid sort order
2142 + buffer_sprintf(mcpc->error, "Invalid sort_order: '%s'. Valid options are 'asc' or 'desc'.", sort_order);
2143 + return MCP_RC_BAD_REQUEST;
2144 + }
2145 + }
2146 + }
2147 +
2148 + // limit
2149 + data->request.limit = 0; // 0 means no limit
2150 + if (json_object_object_get_ex(params, "limit", &obj) &&
2151 + json_object_is_type(obj, json_type_int)) {
2152 + int limit = json_object_get_int(obj);
2153 + if (limit > 0) {
2154 + data->request.limit = (size_t)limit;
2155 + }
2156 + }
2157 +
2158 + // conditions array - parse it early
2159 + if (json_object_object_get_ex(params, "conditions", &obj) &&
2160 + json_object_is_type(obj, json_type_array) &&
2161 + json_object_array_length(obj) > 0) {
2162 + // Parse conditions early
2163 + MCP_RETURN_CODE rc = mcp_parse_conditions_early(&data->request.conditions, obj, mcpc->error);
2164 + if (rc != MCP_RC_OK) {
2165 + return rc;
2166 + }
2167 + }
2168 +
2169 + // Convert query parameter to a full-text search condition
2170 + if (data->request.query && *data->request.query) {
2171 + // Check if we have room for one more condition
2172 + if (data->request.conditions.count >= MAX_CONDITIONS) {
2173 + buffer_sprintf(mcpc->error, "Too many search criteria. Cannot add full-text search query on top of %zu existing conditions. Maximum is %d.",
2174 + data->request.conditions.count, MAX_CONDITIONS);
2175 + return MCP_RC_BAD_REQUEST;
2176 + }
2177 +
2178 + // Add a wildcard search condition for the query
2179 + CONDITION *query_condition = &data->request.conditions.items[data->request.conditions.count];
2180 + query_condition->column_name = "*";
2181 + query_condition->column_index = -1; // Will be set properly during resolve
2182 + query_condition->op = OP_MATCH;
2183 + query_condition->v_type = COND_VALUE_STRING;
2184 + query_condition->v_str = data->request.query;
2185 +
2186 + // Create pattern using substring function (no need to add wildcards manually)
2187 + query_condition->pattern = string_to_simple_pattern_nocase_substring(data->request.query);
2188 +
2189 + data->request.conditions.count++;
2190 + }
2191 +
2192 + // Find the host
2193 + data->request.host = rrdhost_find_by_hostname(data->request.node);
2194 + if (!data->request.host) {
2195 + data->request.host = rrdhost_find_by_guid(data->request.node);
2196 + if (!data->request.host) {
2197 + data->request.host = rrdhost_find_by_node_id(data->request.node);
2198 + }
2199 + }
2200 +
2201 + if (!data->request.host) {
2202 + buffer_sprintf(mcpc->error, "Node not found: %s", data->request.node);
2203 + return MCP_RC_NOT_FOUND;
2204 + }
2205 +
2206 + // Generate transaction UUID
2207 + uuid_generate(data->request.transaction_uuid);
2208 + uuid_unparse_lower(data->request.transaction_uuid, data->request.transaction);
2209 +
2210 + return MCP_RC_OK;
2211 +}
2212 +
2213 +// Execute the function and populate the input section of data
2214 +static MCP_RETURN_CODE mcp_function_run(MCP_FUNCTION_DATA *data, BUFFER *payload) {
2215 + if (!data || !data->request.host || !data->request.function)
2216 + return MCP_RC_ERROR;
2217 +
2218 + // Create source buffer from user_auth
2219 + CLEAN_BUFFER *source = buffer_create(0, NULL);
2220 + user_auth_to_source_buffer(data->request.auth, source);
2221 + buffer_strcat(source, ",modelcontextprotocol");
2222 +
2223 + // Create result buffer (will be owned by data->input.json)
2224 + BUFFER *result_buffer = buffer_create(0, NULL);
2225 +
2226 + // Execute the function
2227 + int ret = rrd_function_run(
2228 + data->request.host,
2229 + result_buffer,
2230 + (int)data->request.timeout,
2231 + data->request.auth->access,
2232 + data->request.function,
2233 + true,
2234 + data->request.transaction,
2235 + NULL,
2236 + NULL,
2237 + NULL,
2238 + NULL,
2239 + NULL,
2240 + NULL,
2241 + payload,
2242 + buffer_tostring(source),
2243 + false
2244 + );
2245 +
2246 + if (ret != HTTP_RESP_OK) {
2247 + buffer_sprintf(data->request.mcpc->error,
2248 + "Failed to execute function '%s' on node '%s', "
2249 + "http error code %d (%s):\n"
2250 + "```json\n%s\n```",
2251 + data->request.function, data->request.node, ret,
2252 + http_response_code2string(ret),
2253 + buffer_tostring(result_buffer));
2254 + buffer_free(result_buffer);
2255 + return MCP_RC_ERROR;
2256 + }
2257 +
2258 + // Store the result in data->input
2259 + data->input.json = result_buffer;
2260 + data->input.jobj = json_tokener_parse(buffer_tostring(result_buffer));
2261 +
2262 + // Analyze the response type
2263 + int status = 0;
2264 + data->input.type = mcp_functions_analyze_response(data->input.jobj, &status);
2265 +
2266 + return MCP_RC_OK;
2267 +}
2268 +
2269 +
2270 +// Process table response
2271 +static MCP_RETURN_CODE mcp_functions_process_table(MCP_FUNCTION_DATA *data, MCP_REQUEST_ID id) {
2272 + const size_t max_size_threshold = 20UL * 1024;
2273 +
2274 + // Initialize success response
2275 + mcp_init_success_result(data->request.mcpc, id);
2276 +
2277 + // Start building content array for the result
2278 + buffer_json_member_add_array(data->request.mcpc->result, "content");
2279 +
2280 + if (!data->input.jobj) {
2281 + // Not valid JSON - return raw output with message
2282 + data->output.status = MCP_TABLE_NOT_JSON;
2283 + buffer_strcat(data->output.result, buffer_tostring(data->input.json));
2284 + add_table_messages_to_mcp_result(data, NULL);
2285 + }
2286 + else if (data->input.type == FN_TYPE_NOT_TABLE && data->input.type != FN_TYPE_TABLE_WITH_HISTORY) {
2287 + // Not a processable table format
2288 + data->output.status = MCP_TABLE_NOT_PROCESSABLE;
2289 + buffer_strcat(data->output.result, buffer_tostring(data->input.json));
2290 + add_table_messages_to_mcp_result(data, NULL);
2291 + }
2292 + else {
2293 + // It's a processable table - get data and columns
2294 + struct json_object *data_obj = NULL;
2295 + struct json_object *columns_obj = NULL;
2296 +
2297 + if (!json_object_object_get_ex(data->input.jobj, "data", &data_obj) ||
2298 + !json_object_object_get_ex(data->input.jobj, "columns", &columns_obj)) {
2299 + // Missing required fields, treat as not processable
2300 + data->output.status = MCP_TABLE_NOT_PROCESSABLE;
2301 + buffer_strcat(data->output.result, buffer_tostring(data->input.json));
2302 + add_table_messages_to_mcp_result(data, NULL);
2303 + }
2304 + else {
2305 + // Check if data is empty
2306 + size_t original_row_count = json_object_array_length(data_obj);
2307 +
2308 + if (original_row_count == 0) {
2309 + // Empty result
2310 + data->output.status = MCP_TABLE_EMPTY_RESULT;
2311 + buffer_strcat(data->output.result, buffer_tostring(data->input.json));
2312 + add_table_messages_to_mcp_result(data, columns_obj);
2313 + }
2314 + else {
2315 + // Process the table with filtering
2316 + mcp_process_table_result(data, max_size_threshold);
2317 +
2318 + // Check for errors or special conditions
2319 + if (data->output.status != MCP_TABLE_OK &&
2320 + data->output.status != MCP_TABLE_RESPONSE_TOO_BIG &&
2321 + data->output.status != MCP_TABLE_INFO_MISSING_COLUMNS_FOUND_RESULTS) {
2322 + // Handle errors
2323 + add_table_messages_to_mcp_result(data, columns_obj);
2324 + }
2325 + else {
2326 + // Check if we need to add any informational messages
2327 + if (data->output.status == MCP_TABLE_INFO_MISSING_COLUMNS_FOUND_RESULTS) {
2328 + add_table_messages_to_mcp_result(data, columns_obj);
2329 + data->output.status = MCP_TABLE_OK;
2330 + }
2331 + else if (data->output.status == MCP_TABLE_RESPONSE_TOO_BIG) {
2332 + // The table was already processed with limit=1 due to size
2333 + // Add the guidance message
2334 + add_table_messages_to_mcp_result(data, columns_obj);
2335 + }
2336 +
2337 + // Add the final result
2338 + buffer_json_add_array_item_object(data->request.mcpc->result);
2339 + {
2340 + buffer_json_member_add_string(data->request.mcpc->result, "type", "text");
2341 + buffer_json_member_add_string(data->request.mcpc->result, "text", buffer_tostring(data->output.result));
2342 + }
2343 + buffer_json_object_close(data->request.mcpc->result);
2344 + }
2345 + }
2346 + }
2347 + }
2348 +
2349 + buffer_json_array_close(data->request.mcpc->result); // Close content array
2350 +
2351 + // Add nextCursor for pagination if applicable (only for successful results)
2352 + if (data->pagination.enabled && data->input.rows > 0 && data->output.status == MCP_TABLE_OK) {
2353 + usec_t next_cursor_timestamp = calculate_next_cursor_from_input(data);
2354 + if (next_cursor_timestamp > 0) {
2355 + CLEAN_BUFFER *cursor_str = buffer_create(0, NULL);
2356 + buffer_sprintf(cursor_str, "%" PRIu64, next_cursor_timestamp);
2357 + buffer_json_member_add_string(data->request.mcpc->result, "nextCursor", buffer_tostring(cursor_str));
2358 + }
2359 + }
2360 +
2361 + buffer_json_object_close(data->request.mcpc->result); // Close result object
2362 + buffer_json_finalize(data->request.mcpc->result); // Finalize the JSON
2363 +
2364 + return MCP_RC_OK;
2365 +}
2366 +
2367 +// Check all requirements and violations collectively for a function execution
2368 +// Returns true if all checks pass, false if there are violations
2369 +// If false is returned, mcpc->error and mcpc->result contain appropriate error messages
2370 +static bool check_requirements_and_violations(MCP_FUNCTION_DATA *data,
2371 + MCP_FUNCTION_REGISTRY_ENTRY *registry_entry,
2372 + struct json_object *selections,
2373 + MCP_REQUEST_ID id) {
2374 + MCP_CLIENT *mcpc = data->request.mcpc;
2375 +
2376 + // 1. Set booleans for each invalid parameter/violation
2377 + bool invalid_timeframe_on_non_history = false;
2378 + bool missing_timeframe_on_history = false;
2379 + bool invalid_condition_operators = false;
2380 + bool invalid_wildcard_patterns = false;
2381 + bool invalid_column_sorting = false;
2382 + bool missing_required_params = false;
2383 + bool invalid_selections_on_non_required = false;
2384 +
2385 + // Arrays to track specific invalid items
2386 + CLEAN_BUFFER *invalid_conditions = buffer_create(0, NULL);
2387 + CLEAN_BUFFER *missing_params = buffer_create(0, NULL);
2388 +
2389 + // Check 1: Functions with has_history=false should not receive time parameters
2390 + if (!registry_entry->has_history) {
2391 + if (data->request.after > 0 || data->request.before > 0 ||
2392 + data->request.anchor > 0 || data->request.direction ||
2393 + data->request.last > 0) {
2394 + invalid_timeframe_on_non_history = true;
2395 + }
2396 + }
2397 +
2398 + // Check 2: Functions with has_history=true should have required parameters
2399 + if (registry_entry->has_history) {
2400 + if (data->request.after == 0) {
2401 + missing_timeframe_on_history = true;
2402 + }
2403 +
2404 + // Check 3: Functions with has_history=true should not use unsupported operations
2405 + if (data->request.conditions.count > 0) {
2406 + for (size_t i = 0; i < data->request.conditions.count; i++) {
2407 + CONDITION *cond = &data->request.conditions.items[i];
2408 +
2409 + // Check for unsupported operators
2410 + if (cond->op != OP_EQUALS && cond->op != OP_MATCH) {
2411 + invalid_condition_operators = true;
2412 + if (buffer_strlen(invalid_conditions) > 0)
2413 + buffer_strcat(invalid_conditions, ", ");
2414 + buffer_sprintf(invalid_conditions, "'%s' (uses %s operator)",
2415 + cond->column_name,
2416 + cond->op == OP_NOT_EQUALS ? "!=" :
2417 + cond->op == OP_LESS ? "<" :
2418 + cond->op == OP_LESS_EQUALS ? "<=" :
2419 + cond->op == OP_GREATER ? ">" :
2420 + cond->op == OP_GREATER_EQUALS ? ">=" :
2421 + cond->op == OP_NOT_MATCH ? "not match" : "unknown");
2422 + continue;
2423 + }
2424 +
2425 + // Check for pattern matching with wildcards (except for "*" column which is full-text search)
2426 + if (cond->op == OP_MATCH && cond->pattern && strcmp(cond->column_name, "*") != 0) {
2427 + const char *pattern_str = cond->v_str;
2428 + if (pattern_str && (strchr(pattern_str, '*') || strchr(pattern_str, '?'))) {
2429 + invalid_wildcard_patterns = true;
2430 + if (buffer_strlen(invalid_conditions) > 0)
2431 + buffer_strcat(invalid_conditions, ", ");
2432 + buffer_sprintf(invalid_conditions, "'%s' (uses wildcards in pattern '%s')",
2433 + cond->column_name, pattern_str);
2434 + }
2435 + }
2436 + }
2437 + }
2438 +
2439 + // Check 4: Functions with has_history=true should not use column sorting
2440 + if (data->request.sort.column) {
2441 + invalid_column_sorting = true;
2442 + }
2443 + }
2444 +
2445 + // Check 5: Required parameters validation
2446 + if (registry_entry->required_params_count > 0) {
2447 + if (!selections) {
2448 + missing_required_params = true;
2449 + for (size_t i = 0; i < registry_entry->required_params_count; i++) {
2450 + const char *param_id = string2str(registry_entry->required_params[i].id);
2451 + if (buffer_strlen(missing_params) > 0)
2452 + buffer_strcat(missing_params, ", ");
2453 + buffer_strcat(missing_params, param_id);
2454 + }
2455 + } else {
2456 + // Check if all required parameters have non-empty values
2457 + for (size_t i = 0; i < registry_entry->required_params_count; i++) {
2458 + const char *param_id = string2str(registry_entry->required_params[i].id);
2459 + MCP_FUNCTION_PARAM *param = &registry_entry->required_params[i];
2460 + struct json_object *param_value = NULL;
2461 + bool param_invalid = false;
2462 +
2463 + if (!json_object_object_get_ex(selections, param_id, &param_value)) {
2464 + param_invalid = true;
2465 + } else {
2466 + // Validate based on parameter type
2467 + if (param->type == MCP_REQUIRED_PARAMS_TYPE_SELECT) {
2468 + // For select type, accept either string or array with single element
2469 + if (json_object_is_type(param_value, json_type_string)) {
2470 + const char *str_val = json_object_get_string(param_value);
2471 + if (!str_val || !*str_val) {
2472 + param_invalid = true;
2473 + }
2474 + } else if (json_object_is_type(param_value, json_type_array)) {
2475 + if (json_object_array_length(param_value) == 0) {
2476 + param_invalid = true;
2477 + }
2478 + } else {
2479 + param_invalid = true;
2480 + }
2481 + } else {
2482 + // For multiselect type, require array with at least one element
2483 + if (!json_object_is_type(param_value, json_type_array) ||
2484 + json_object_array_length(param_value) == 0) {
2485 + param_invalid = true;
2486 + }
2487 + }
2488 + }
2489 +
2490 + if (param_invalid) {
2491 + missing_required_params = true;
2492 + if (buffer_strlen(missing_params) > 0)
2493 + buffer_strcat(missing_params, ", ");
2494 + buffer_strcat(missing_params, param_id);
2495 + }
2496 + }
2497 + }
2498 + } else {
2499 + // Function has no required parameters - selections should not be provided
2500 + if (selections && json_object_is_type(selections, json_type_object) &&
2501 + json_object_object_length(selections) > 0) {
2502 + invalid_selections_on_non_required = true;
2503 + }
2504 + }
2505 +
2506 + // 2. Check if we have any violations
2507 + bool has_violations = invalid_timeframe_on_non_history || missing_timeframe_on_history ||
2508 + invalid_condition_operators || invalid_wildcard_patterns ||
2509 + invalid_column_sorting || missing_required_params ||
2510 + invalid_selections_on_non_required;
2511 +
2512 + if (!has_violations) {
2513 + return true;
2514 + }
2515 +
2516 + // 3. Generate error response with summary and detailed messages
2517 + mcp_init_success_result(mcpc, id);
2518 + buffer_json_member_add_array(mcpc->result, "content");
2519 + buffer_json_add_array_item_object(mcpc->result);
2520 + {
2521 + buffer_json_member_add_string(mcpc->result, "type", "text");
2522 +
2523 + CLEAN_BUFFER *message = buffer_create(0, NULL);
2524 +
2525 + // Summary of what's wrong
2526 + buffer_strcat(message, "FUNCTION EXECUTION FAILED\n\n");
2527 + buffer_strcat(message, "Summary of issues:\n");
2528 +
2529 + int issue_count = 0;
2530 + if (invalid_timeframe_on_non_history) {
2531 + buffer_sprintf(message, "%d. Invalid timeframe parameters for non-history function\n", ++issue_count);
2532 + }
2533 + if (missing_timeframe_on_history) {
2534 + buffer_sprintf(message, "%d. Missing required timeframe parameters\n", ++issue_count);
2535 + }
2536 + if (invalid_condition_operators) {
2537 + buffer_sprintf(message, "%d. Invalid condition operators used\n", ++issue_count);
2538 + }
2539 + if (invalid_wildcard_patterns) {
2540 + buffer_sprintf(message, "%d. Invalid wildcard patterns in conditions\n", ++issue_count);
2541 + }
2542 + if (invalid_column_sorting) {
2543 + buffer_sprintf(message, "%d. Invalid column sorting for history function\n", ++issue_count);
2544 + }
2545 + if (missing_required_params) {
2546 + buffer_sprintf(message, "%d. Missing or invalid required parameters\n", ++issue_count);
2547 + }
2548 + if (invalid_selections_on_non_required) {
2549 + buffer_sprintf(message, "%d. Invalid selections parameter for function without requirements\n", ++issue_count);
2550 + }
2551 +
2552 + // Detailed explanations
2553 + buffer_strcat(message, "\nDetailed explanations:\n\n");
2554 +
2555 + if (invalid_timeframe_on_non_history) {
2556 + buffer_strcat(message, "❌ TIMEFRAME PARAMETERS NOT SUPPORTED\n");
2557 + buffer_strcat(message, " Problem: This function does not support time-based parameters\n");
2558 + buffer_strcat(message, " Invalid parameters: after, before, cursor, direction, limit\n");
2559 + buffer_strcat(message, " Solution: Remove all timeframe parameters from your request\n\n");
2560 + }
2561 +
2562 + if (missing_timeframe_on_history) {
2563 + buffer_strcat(message, "❌ MISSING TIMEFRAME PARAMETERS\n");
2564 + buffer_strcat(message, " Problem: History functions require the 'after' parameter\n");
2565 + buffer_strcat(message, " Required: 'after' parameter must be specified (cannot be 0)\n");
2566 + buffer_strcat(message, " Time relationship: 'before' is relative to now, 'after' is relative to 'before'\n");
2567 + buffer_strcat(message, " Solution: Add after parameter, e.g., \"after\": -3600 (1 hour before 'before')\n\n");
2568 + }
2569 +
2570 +
2571 + if (invalid_condition_operators) {
2572 + buffer_strcat(message, "❌ INVALID CONDITION OPERATORS\n");
2573 + buffer_sprintf(message, " Problem: Invalid operators used in conditions for: %s\n", buffer_tostring(invalid_conditions));
2574 + buffer_strcat(message, " Allowed: Only '==' (equals) and 'match' (for value lists)\n");
2575 + buffer_strcat(message, " Solution: Change operators to '==' or use 'match' with pipe-separated values\n\n");
2576 + }
2577 +
2578 + if (invalid_wildcard_patterns) {
2579 + buffer_strcat(message, "❌ INVALID WILDCARD PATTERNS\n");
2580 + buffer_sprintf(message, " Problem: Wildcard patterns not supported in conditions for: %s\n", buffer_tostring(invalid_conditions));
2581 + buffer_strcat(message, " Solution: Use exact values or pipe-separated lists (e.g., 'value1|value2|value3')\n\n");
2582 + }
2583 +
2584 + if (invalid_column_sorting) {
2585 + buffer_strcat(message, "❌ INVALID COLUMN SORTING\n");
2586 + buffer_strcat(message, " Problem: Column-based sorting not supported for history functions\n");
2587 + buffer_strcat(message, " Solution: Remove 'sort_column' parameter and use 'direction' for time-based sorting\n\n");
2588 + }
2589 +
2590 + if (missing_required_params) {
2591 + buffer_strcat(message, "❌ MISSING OR INVALID REQUIRED PARAMETERS\n");
2592 + buffer_sprintf(message, " Problem: Required parameters missing or invalid: %s\n", buffer_tostring(missing_params));
2593 + buffer_strcat(message, " Solution: Provide all required parameters with valid values\n\n");
2594 +
2595 + // Include the helpful parameter message
2596 + generate_required_params_message(message, registry_entry, missing_timeframe_on_history, missing_params);
2597 + }
2598 +
2599 + if (invalid_selections_on_non_required) {
2600 + buffer_strcat(message, "❌ INVALID SELECTIONS PARAMETER\n");
2601 + buffer_strcat(message, " Problem: This function does not accept the 'selections' parameter\n");
2602 + buffer_strcat(message, " Solution: Remove the 'selections' parameter from your request\n\n");
2603 + }
2604 +
2605 + buffer_json_member_add_string(mcpc->result, "text", buffer_tostring(message));
2606 + }
2607 + buffer_json_object_close(mcpc->result);
2608 + buffer_json_array_close(mcpc->result);
2609 + buffer_json_object_close(mcpc->result);
2610 + buffer_json_finalize(mcpc->result);
2611 + return false;
2612 +}
2613 +
2614 +MCP_RETURN_CODE mcp_tool_execute_function_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id)
2615 +{
2616 + if (!mcpc || id == 0 || !params)
2617 + return MCP_RC_ERROR;
2618 +
2619 + // Create and initialize function data structure
2620 + MCP_FUNCTION_DATA data;
2621 + mcp_functions_data_init(&data);
2622 +
2623 + // Parse the request
2624 + MCP_RETURN_CODE rc = mcp_parse_function_request(&data, mcpc, params);
2625 + if (rc != MCP_RC_OK) {
2626 + mcp_functions_data_cleanup(&data);
2627 + return rc;
2628 + }
2629 +
2630 + // Get function registry entry
2631 + CLEAN_BUFFER *registry_error = buffer_create(0, NULL);
2632 + MCP_FUNCTION_REGISTRY_ENTRY *registry_entry = mcp_functions_registry_get(data.request.host, data.request.function, registry_error);
2633 +
2634 + if (!registry_entry) {
2635 + buffer_sprintf(mcpc->error, "Failed to get function info: %s", buffer_tostring(registry_error));
2636 + mcp_functions_data_cleanup(&data);
2637 + return MCP_RC_ERROR;
2638 + }
2639 +
2640 + // Check if function requires parameters
2641 + struct json_object *selections = NULL;
2642 + if (json_object_object_get_ex(params, "selections", &selections)) {
2643 + // Selections provided - validate they are an object
2644 + if (!json_object_is_type(selections, json_type_object)) {
2645 + buffer_strcat(mcpc->error, "The 'selections' parameter must be an object with key-value pairs where values are arrays of strings");
2646 + mcp_functions_registry_release(registry_entry);
2647 + mcp_functions_data_cleanup(&data);
2648 + return MCP_RC_BAD_REQUEST;
2649 + }
2650 + }
2651 +
2652 + // Check all requirements and violations collectively
2653 + if (!check_requirements_and_violations(&data, registry_entry, selections, id)) {
2654 + mcp_functions_registry_release(registry_entry);
2655 + mcp_functions_data_cleanup(&data);
2656 + return MCP_RC_OK; // We've already set up the response in the validation function
2657 + }
2658 +
2659 + // Build the actual function name to execute and POST payload if needed
2660 + CLEAN_BUFFER *actual_function = buffer_create(0, NULL);
2661 + CLEAN_BUFFER *post_payload = NULL;
2662 +
2663 + if (registry_entry->supports_post) {
2664 + // v3+ function with POST support - create POST payload (even if no selections)
2665 + buffer_strcat(actual_function, data.request.function);
2666 + post_payload = build_post_payload_with_selections(selections, &data, registry_entry);
2667 + } else if (selections) {
2668 + // GET function with parameters - append to function name
2669 + build_function_name_with_params(actual_function, data.request.function, selections, &data, registry_entry);
2670 + } else {
2671 + // No parameters needed or provided
2672 + buffer_strcat(actual_function, data.request.function);
2673 + }
2674 +
2675 + // Update the function name in data
2676 + data.request.function = buffer_tostring(actual_function);
2677 +
2678 + // Copy pagination settings to avoid keeping registry_entry locked
2679 + data.pagination.enabled = registry_entry->pagination.enabled;
2680 + data.pagination.units = registry_entry->pagination.units;
2681 + data.pagination.column = string_dup(registry_entry->pagination.column);
2682 +
2683 + // Release registry entry - we're done with it now
2684 + mcp_functions_registry_release(registry_entry);
2685 + registry_entry = NULL; // Prevent accidental access
2686 +
2687 + // Execute the function
2688 + rc = mcp_function_run(&data, post_payload);
2689 + if (rc != MCP_RC_OK) {
2690 + mcp_functions_data_cleanup(&data);
2691 + return rc;
2692 + }
2693 +
2694 + // Process the response (all function types use table processing now)
2695 + rc = mcp_functions_process_table(&data, id);
2696 +
2697 + // Cleanup
2698 + mcp_functions_data_cleanup(&data);
2699 +
2700 + return rc;
2701 +}
src/web/mcp/mcp-tools-execute-function.h new
+14
@@ -0,0 +1,14 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_H
4 +#define NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_H
5 +
6 +#include "mcp-tools.h"
7 +
8 +// Schema definition function - provides JSON schema for this tool
9 +void mcp_tool_execute_function_schema(BUFFER *buffer);
10 +
11 +// Execution function - performs the function execution
12 +MCP_RETURN_CODE mcp_tool_execute_function_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
13 +
14 +#endif // NETDATA_MCP_TOOLS_EXECUTE_FUNCTION_H
\ No newline at end of file
src/web/mcp/mcp-tools-list-metadata.c new
+532
@@ -0,0 +1,532 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-tools-list-metadata.h"
4 +#include "mcp-tools.h"
5 +#include "mcp-params.h"
6 +#include "database/contexts/rrdcontext.h"
7 +
8 +// Static configuration for all list tools
9 +static const MCP_LIST_TOOL_CONFIG mcp_list_tools[] = {
10 + {
11 + .name = MCP_TOOL_LIST_METRICS,
12 + .title = "List available metrics",
13 + .description = "Search and list available metrics to query, across some or all nodes, for any time-frame",
14 + .output_type = MCP_LIST_OUTPUT_METRICS,
15 + .mode = CONTEXTS_V2_CONTEXTS,
16 + .options = 0, // Just MCP will be added
17 + .params = {
18 + .has_q = true,
19 + .has_metrics = true,
20 + .has_nodes = true,
21 + .has_time_range = true,
22 + .has_cardinality_limit = true,
23 + .nodes_as_array = true, // list_metrics uses array for nodes
24 + },
25 + },
26 + {
27 + .name = MCP_TOOL_GET_METRICS_DETAILS,
28 + .title = "Get metrics details",
29 + .description = "Get retention and cardinality information about specific metrics",
30 + .output_type = MCP_LIST_OUTPUT_METRICS,
31 + .mode = CONTEXTS_V2_CONTEXTS,
32 + .options = CONTEXTS_OPTION_TITLES | CONTEXTS_OPTION_INSTANCES |
33 + CONTEXTS_OPTION_DIMENSIONS | CONTEXTS_OPTION_LABELS |
34 + CONTEXTS_OPTION_RETENTION | CONTEXTS_OPTION_LIVENESS |
35 + CONTEXTS_OPTION_FAMILY | CONTEXTS_OPTION_UNITS,
36 + .params = {
37 + .has_metrics = true,
38 + .has_nodes = true,
39 + .has_time_range = true,
40 + .has_cardinality_limit = true,
41 + .metrics_required = true,
42 + .nodes_as_array = true, // get_metrics_details uses array for nodes
43 + .metrics_as_array = true, // get_metrics_details uses array for metrics
44 + },
45 + },
46 + {
47 + .name = MCP_TOOL_LIST_NODES,
48 + .title = "List monitored nodes",
49 + .description = "Search for and list monitored nodes by hostname patterns. Use the 'nodes' parameter to search for specific nodes instead of retrieving all nodes",
50 + .output_type = MCP_LIST_OUTPUT_NODES,
51 + .mode = CONTEXTS_V2_NODES,
52 + .options = 0, // Just MCP will be added
53 + .params = {
54 + .has_nodes = true,
55 + .has_metrics = true, // Filters nodes that collect these metrics
56 + .has_time_range = true,
57 + .has_cardinality_limit = true,
58 + .metrics_as_array = true, // list_nodes uses an array for metrics filter
59 + },
60 + },
61 + {
62 + .name = MCP_TOOL_LIST_FUNCTIONS,
63 + .title = "List available functions",
64 + .description = "List available Netdata functions that can be executed on specific nodes",
65 + .output_type = MCP_LIST_OUTPUT_FUNCTIONS,
66 + .mode = CONTEXTS_V2_FUNCTIONS,
67 + .options = 0,
68 + .params = {
69 + .has_nodes = true,
70 + .has_time_range = false, // Functions are live, not historical
71 + .has_cardinality_limit = false, // Functions list is small, no limit needed
72 + .nodes_required = true, // Must specify which nodes to query
73 + .nodes_as_array = true, // list_functions uses array for nodes
74 + },
75 + },
76 + {
77 + .name = MCP_TOOL_GET_NODES_DETAILS,
78 + .title = "Get detailed information about monitored nodes",
79 + .description = "Gets comprehensive node information including hardware specs, OS details, capabilities, health status, available functions, streaming and monitoring configuration",
80 + .output_type = MCP_LIST_OUTPUT_NODES,
81 + .mode = CONTEXTS_V2_NODES | CONTEXTS_V2_NODES_INFO | CONTEXTS_V2_NODE_INSTANCES,
82 + .options = 0, // Just MCP will be added
83 + .params = {
84 + .has_nodes = true,
85 + .has_metrics = true, // Filters nodes that collect these metrics
86 + .has_time_range = true,
87 + .has_cardinality_limit = true,
88 + .nodes_required = true, // Must specify nodes due to large output
89 + .nodes_as_array = true, // get_nodes_details uses array for nodes
90 + .metrics_as_array = true, // get_nodes_details uses array for metrics
91 + },
92 + },
93 + {
94 + .name = MCP_TOOL_LIST_RAISED_ALERTS,
95 + .title = "List raised alerts",
96 + .description = "List currently active alerts (WARNING and CRITICAL status)",
97 + .output_type = MCP_LIST_OUTPUT_ALERTS,
98 + .mode = CONTEXTS_V2_ALERTS,
99 + .options = CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_VALUES,
100 + .params = {
101 + .has_nodes = true,
102 + .has_metrics = true, // Filter by context pattern
103 + .has_alert_pattern = true,
104 + .has_time_range = false, // Raised alerts are current, not historical
105 + .has_cardinality_limit = true,
106 + .nodes_as_array = true,
107 + .metrics_as_array = true, // Metrics should be an array for alerts
108 + },
109 + .defaults = {
110 + .alert_status = CONTEXT_ALERT_RAISED, // Only raised alerts
111 + .cardinality_limit = 200,
112 + },
113 + },
114 + {
115 + .name = MCP_TOOL_LIST_ALL_ALERTS,
116 + .title = "List all alerts",
117 + .description = "List all currently running alerts",
118 + .output_type = MCP_LIST_OUTPUT_ALERTS,
119 + .mode = CONTEXTS_V2_ALERTS,
120 + .options = CONTEXTS_OPTION_SUMMARY,
121 + .params = {
122 + .has_nodes = true,
123 + .has_metrics = true, // Filter by context pattern
124 + .has_alert_pattern = true,
125 + .has_time_range = true,
126 + .has_cardinality_limit = true,
127 + .nodes_as_array = true,
128 + .metrics_as_array = true, // Metrics should be an array for alerts
129 + },
130 + .defaults = {
131 + .alert_status = CONTEXTS_ALERT_STATUSES, // All statuses
132 + .cardinality_limit = 200,
133 + },
134 + },
135 +};
136 +
137 +// Get tool configuration by name
138 +const MCP_LIST_TOOL_CONFIG *mcp_get_list_tool_config(const char *name) {
139 + for (size_t i = 0; i < sizeof(mcp_list_tools) / sizeof(mcp_list_tools[0]); i++) {
140 + if (strcmp(mcp_list_tools[i].name, name) == 0) {
141 + return &mcp_list_tools[i];
142 + }
143 + }
144 + return NULL;
145 +}
146 +
147 +// Unified schema generation
148 +void mcp_unified_list_tool_schema(BUFFER *buffer, const MCP_LIST_TOOL_CONFIG *config) {
149 + if (!buffer || !config) return;
150 +
151 + // Determine output type name
152 + const char *output_name;
153 + switch (config->output_type) {
154 + case MCP_LIST_OUTPUT_NODES:
155 + output_name = "nodes";
156 + break;
157 + case MCP_LIST_OUTPUT_METRICS:
158 + output_name = "metrics";
159 + break;
160 + case MCP_LIST_OUTPUT_FUNCTIONS:
161 + output_name = "functions";
162 + break;
163 + case MCP_LIST_OUTPUT_ALERTS:
164 + output_name = "alerts";
165 + break;
166 + default:
167 + output_name = "items";
168 + break;
169 + }
170 +
171 + // Buffers for composing strings
172 + char title[256];
173 + char description[1024];
174 +
175 + // Tool input schema
176 + buffer_json_member_add_object(buffer, "inputSchema");
177 + buffer_json_member_add_string(buffer, "type", "object");
178 + buffer_json_member_add_string(buffer, "title", config->title);
179 +
180 + // Properties
181 + buffer_json_member_add_object(buffer, "properties");
182 +
183 + // Add a metrics pattern if supported
184 + if (config->params.has_metrics) {
185 + if (config->params.metrics_as_array) {
186 + // Use an array for metrics
187 + if (config->output_type != MCP_LIST_OUTPUT_METRICS) {
188 + // This is a node/function/alert query - metrics acts as a filter
189 + if (config->output_type == MCP_LIST_OUTPUT_ALERTS) {
190 + // For alerts, metrics refers to contexts
191 + mcp_schema_add_array_param(buffer, "metrics",
192 + config->params.metrics_required ? "Specify the contexts to filter by" : "Filter by contexts",
193 + config->params.metrics_required ?
194 + "Array of specific context names to filter alerts by. This parameter is required. "
195 + "Each context must be an exact match - no wildcards or patterns allowed. "
196 + "Use '" MCP_TOOL_LIST_METRICS "' to discover available contexts. "
197 + "Examples: [\"system.cpu\", \"disk.space\"], [\"mysql.queries\", \"redis.memory\"]" :
198 + "Array of specific context names to filter alerts by. "
199 + "Each context must be an exact match - no wildcards or patterns allowed. "
200 + "Use '" MCP_TOOL_LIST_METRICS "' to discover available contexts. "
201 + "If not specified, alerts from all contexts are included. "
202 + "Examples: [\"system.cpu\", \"disk.space\"], [\"mysql.queries\", \"redis.memory\"]");
203 + } else {
204 + // For nodes/functions, metrics is a filter
205 + mcp_schema_add_array_param(buffer, "metrics",
206 + config->params.metrics_required ? "Specify the metrics to filter by" : "Filter by metrics",
207 + config->params.metrics_required ?
208 + "Array of specific metric names to filter by. This parameter is required. "
209 + "Each metric must be an exact match - no wildcards or patterns allowed. "
210 + "Use '" MCP_TOOL_LIST_METRICS "' to discover available metrics. "
211 + "Examples: [\"system.cpu\", \"system.load\"], [\"disk.io\", \"disk.space\"]" :
212 + "Array of specific metric names to filter by. "
213 + "Each metric must be an exact match - no wildcards or patterns allowed. "
214 + "Use '" MCP_TOOL_LIST_METRICS "' to discover available metrics. "
215 + "If not specified, all metrics are included. "
216 + "Examples: [\"system.cpu\", \"system.load\"], [\"disk.io\", \"disk.space\"]");
217 + }
218 + } else {
219 + // This is a metrics query
220 + mcp_schema_add_array_param(buffer, "metrics",
221 + config->params.metrics_required ? "Specify the metrics" : "Filter metrics",
222 + config->params.metrics_required ?
223 + "Array of specific metric names to retrieve details for. This parameter is required. "
224 + "Each metric must be an exact match - no wildcards or patterns allowed. "
225 + "Examples: [\"system.cpu\", \"system.load\", \"system.ram\"]" :
226 + "Array of specific metric names to filter. "
227 + "Each metric must be an exact match - no wildcards or patterns allowed. "
228 + "If not specified, all metrics are included. "
229 + "Examples: [\"system.cpu\", \"system.load\", \"system.ram\"]");
230 + }
231 + } else {
232 + // Use string pattern for metrics
233 + buffer_json_member_add_object(buffer, "metrics");
234 + buffer_json_member_add_string(buffer, "type", "string");
235 +
236 + // Compose title and description based on context
237 + if (config->output_type != MCP_LIST_OUTPUT_METRICS) {
238 + // This is a node/function query - metrics acts as a filter
239 + snprintfz(title, sizeof(title), "%s metrics",
240 + config->params.metrics_required ? "Specify the" : "Filter");
241 + snprintfz(description, sizeof(description),
242 + "Filter %s to only those collecting these metrics. "
243 + "Use pipe (|) to separate multiple patterns. Supports wildcards. "
244 + "Examples: 'system.*', '*cpu*|*memory*', 'disk.*|net.*'",
245 + output_name);
246 + } else {
247 + // This is a metrics query
248 + snprintfz(title, sizeof(title), "%s metrics",
249 + config->params.metrics_required ? "Specify the" : "Filter");
250 + if (config->params.metrics_required) {
251 + snprintfz(description, sizeof(description),
252 + "Pipe-separated list of metric names. "
253 + "Example: 'system.cpu|system.load|system.ram'");
254 + } else {
255 + snprintfz(description, sizeof(description),
256 + "Pattern matching on metric names. Use pipe (|) to separate multiple patterns. "
257 + "Supports wildcards. Examples: 'system.*', '*cpu*|*memory*', 'disk.*|net.*|system.*'");
258 + }
259 + }
260 +
261 + buffer_json_member_add_string(buffer, "title", title);
262 + buffer_json_member_add_string(buffer, "description", description);
263 + if (!config->params.metrics_required) {
264 + buffer_json_member_add_string(buffer, "default", "*");
265 + }
266 + buffer_json_object_close(buffer); // metrics
267 + }
268 + }
269 +
270 + // Add full-text search if supported
271 + if (config->params.has_q) {
272 + buffer_json_member_add_object(buffer, "q");
273 + {
274 + buffer_json_member_add_string(buffer, "type", "string");
275 + buffer_json_member_add_string(buffer, "title", "Full-text search on metrics metadata");
276 + buffer_json_member_add_string(buffer, "description",
277 + "Filter metrics by searching across all their metadata (names, titles, instances, dimensions, labels). "
278 + "Use pipe (|) to separate multiple search terms. Examples: 'memory|pressure', 'cpu|load|system'");
279 + }
280 + buffer_json_object_close(buffer); // q
281 + }
282 +
283 + // Add a nodes' pattern if supported
284 + if (config->params.has_nodes) {
285 + if (config->params.nodes_as_array) {
286 + // Use an array for nodes
287 + mcp_schema_add_array_param(buffer, "nodes",
288 + config->params.nodes_required ? "Specify the nodes" : "Filter by nodes",
289 + config->params.nodes_required ?
290 + "Array of specific node names to query. This parameter is required because this tool produces detailed output. "
291 + "Each node must be an exact match - no wildcards or patterns allowed. "
292 + "Use '" MCP_TOOL_LIST_NODES "' to discover available nodes. "
293 + "Examples: [\"node1\", \"node2\"], [\"web-server-01\", \"db-server-01\"]" :
294 + "Array of specific node names to filter by. "
295 + "Each node must be an exact match - no wildcards or patterns allowed. "
296 + "Use '" MCP_TOOL_LIST_NODES "' to discover available nodes. "
297 + "If not specified, all nodes are included. "
298 + "Examples: [\"node1\", \"node2\"], [\"web-server-01\", \"db-server-01\"]");
299 + } else {
300 + // Use string pattern for nodes
301 + buffer_json_member_add_object(buffer, "nodes");
302 + buffer_json_member_add_string(buffer, "type", "string");
303 +
304 + // Compose title and description based on context
305 + snprintfz(title, sizeof(title), "%s nodes",
306 + config->params.nodes_required ? "Specify the" : "Filter");
307 +
308 + if (config->params.nodes_required) {
309 + snprintfz(description, sizeof(description),
310 + "Specify which nodes to query. This parameter is required because this tool produces detailed output. "
311 + "Use pipe (|) to separate multiple patterns. Examples: 'node1|node2', '*web*|*db*', 'prod-*'");
312 + } else if (config->output_type == MCP_LIST_OUTPUT_NODES || config->output_type == MCP_LIST_OUTPUT_FUNCTIONS) {
313 + // This is a node/function query - direct filtering
314 + snprintfz(description, sizeof(description),
315 + "Search for nodes by hostname patterns. This is the primary way to find specific nodes without retrieving the full list. "
316 + "Use pipe (|) to separate multiple patterns. Wildcards (*) are supported for flexible matching. "
317 + "Examples: 'node1|node2' (exact names), '*web*' (contains 'web'), 'prod-*' (starts with 'prod-'), '*db*|*cache*' (contains 'db' or 'cache')");
318 + } else {
319 + // This is a metrics query - 'nodes' acts as a filter
320 + snprintfz(description, sizeof(description),
321 + "Filter %s to only those collected by these nodes. "
322 + "Use pipe (|) to separate multiple patterns. "
323 + "Examples: 'node1|node2', '*web*|*db*', 'prod-*|staging-*'",
324 + output_name);
325 + }
326 +
327 + buffer_json_member_add_string(buffer, "title", title);
328 + buffer_json_member_add_string(buffer, "description", description);
329 + if (!config->params.nodes_required) {
330 + buffer_json_member_add_string(buffer, "default", "*");
331 + }
332 + buffer_json_object_close(buffer); // nodes
333 + }
334 + }
335 +
336 + // Add time range parameters if supported
337 + if (config->params.has_time_range) {
338 + // Build description prefix for metadata queries
339 + char description_prefix[256];
340 + snprintfz(description_prefix, sizeof(description_prefix),
341 + "%s with data collected", output_name ? output_name : "results");
342 + mcp_schema_add_time_params(buffer, description_prefix, false);
343 + }
344 +
345 + // Add cardinality limit if supported
346 + if (config->params.has_cardinality_limit) {
347 + size_t default_cardinality = config->defaults.cardinality_limit ?: MCP_METADATA_CARDINALITY_LIMIT;
348 + mcp_schema_add_cardinality_limit(buffer,
349 + "Maximum number of items to return per category (dimensions, instances, labels, etc.). "
350 + "Prevents response explosion. When exceeded, the response will indicate how many items were omitted.",
351 + default_cardinality,
352 + 1, // minimum
353 + MAX(default_cardinality, MCP_METADATA_CARDINALITY_LIMIT_MAX));
354 + }
355 +
356 + // Add alert-specific parameters
357 + if (config->params.has_alert_pattern) {
358 + // Use string pattern for alerts
359 + buffer_json_member_add_object(buffer, "alerts");
360 + buffer_json_member_add_string(buffer, "type", "string");
361 + buffer_json_member_add_string(buffer, "title", "Filter alerts");
362 + buffer_json_member_add_string(buffer, "description",
363 + "Pattern matching on alert names. Use pipe (|) to separate multiple patterns. "
364 + "Supports wildcards. Examples: 'disk_*', '*cpu*|*memory*', 'health.*'");
365 + buffer_json_member_add_string(buffer, "default", "*");
366 + buffer_json_object_close(buffer); // alerts
367 + }
368 +
369 + buffer_json_object_close(buffer); // properties
370 +
371 + // Required fields
372 + if (config->params.metrics_required || config->params.nodes_required) {
373 + buffer_json_member_add_array(buffer, "required");
374 + if (config->params.metrics_required) {
375 + buffer_json_add_array_item_string(buffer, "metrics");
376 + }
377 + if (config->params.nodes_required) {
378 + buffer_json_add_array_item_string(buffer, "nodes");
379 + }
380 + buffer_json_array_close(buffer);
381 + }
382 +
383 + buffer_json_object_close(buffer); // inputSchema
384 +}
385 +
386 +// Removed extract_string_param and extract_size_param - now using mcp-params functions
387 +
388 +// Unified execution
389 +MCP_RETURN_CODE mcp_unified_list_tool_execute(MCP_CLIENT *mcpc, const MCP_LIST_TOOL_CONFIG *config,
390 + struct json_object *params, MCP_REQUEST_ID id)
391 +{
392 + if (!mcpc || !config || id == 0)
393 + return MCP_RC_ERROR;
394 +
395 + // Extract parameters based on configuration
396 + const char *q = config->params.has_q ? mcp_params_extract_string(params, "q", NULL) : NULL;
397 +
398 + // Handle metrics - either as array or string pattern
399 + const char *metrics_pattern = NULL;
400 + CLEAN_BUFFER *metrics_buffer = NULL;
401 +
402 + if (config->params.has_metrics) {
403 + if (config->params.metrics_as_array) {
404 + // Parse metrics as an array
405 + metrics_buffer = mcp_params_parse_array_to_pattern(params, "metrics", false, false, MCP_TOOL_LIST_METRICS, mcpc->error);
406 + if (buffer_strlen(mcpc->error) > 0) {
407 + return MCP_RC_BAD_REQUEST;
408 + }
409 + metrics_pattern = buffer_tostring(metrics_buffer);
410 + } else {
411 + // Parse metrics as a string pattern
412 + metrics_pattern = mcp_params_extract_string(params, "metrics", NULL);
413 + }
414 + if(metrics_pattern && !*metrics_pattern) {
415 + metrics_pattern = NULL; // Treat empty string as no metrics specified
416 + }
417 + }
418 +
419 + // Handle nodes - either as array or string pattern
420 + const char *nodes_pattern = NULL;
421 + CLEAN_BUFFER *nodes_buffer = NULL;
422 +
423 + if (config->params.has_nodes) {
424 + if (config->params.nodes_as_array) {
425 + // Parse nodes as array
426 + nodes_buffer = mcp_params_parse_array_to_pattern(params, "nodes", false, false, MCP_TOOL_LIST_NODES, mcpc->error);
427 + if (buffer_strlen(mcpc->error) > 0) {
428 + return MCP_RC_BAD_REQUEST;
429 + }
430 + nodes_pattern = buffer_tostring(nodes_buffer);
431 + } else {
432 + // Parse nodes as a string pattern
433 + nodes_pattern = mcp_params_extract_string(params, "nodes", NULL);
434 + }
435 + if(nodes_pattern && !*nodes_pattern) {
436 + nodes_pattern = NULL; // Treat empty string as no nodes specified
437 + }
438 + }
439 +
440 + // Check required parameters
441 + if (config->params.metrics_required && !metrics_pattern) {
442 + buffer_sprintf(mcpc->error, "Missing required parameter 'metrics'. Use '" MCP_TOOL_LIST_METRICS "' to discover available metrics.");
443 + return MCP_RC_ERROR;
444 + }
445 +
446 + if (config->params.nodes_required && !nodes_pattern) {
447 + buffer_sprintf(mcpc->error, "Missing required parameter 'nodes'. Use '" MCP_TOOL_LIST_NODES "' to discover available nodes.");
448 + return MCP_RC_ERROR;
449 + }
450 +
451 + // Extract time parameters (only if the tool supports them)
452 + time_t after = 0;
453 + time_t before = 0;
454 + if (config->params.has_time_range) {
455 + after = mcp_params_parse_time(params, "after", MCP_DEFAULT_AFTER_TIME);
456 + before = mcp_params_parse_time(params, "before", MCP_DEFAULT_BEFORE_TIME);
457 + }
458 +
459 + // Extract cardinality limit if supported
460 + size_t cardinality_limit = 0;
461 + if (config->params.has_cardinality_limit) {
462 + size_t default_cardinality = config->defaults.cardinality_limit ?: MCP_METADATA_CARDINALITY_LIMIT;
463 + cardinality_limit = mcp_params_extract_size(params, "cardinality_limit", default_cardinality, 1, 500, mcpc->error);
464 + if (buffer_strlen(mcpc->error) > 0) {
465 + return MCP_RC_BAD_REQUEST;
466 + }
467 + }
468 +
469 + // Extract alert-specific parameters
470 + const char *alert_pattern = NULL;
471 +
472 + if (config->params.has_alert_pattern) {
473 + // Parse alerts as a string pattern
474 + alert_pattern = mcp_params_extract_string(params, "alerts", NULL);
475 + if(alert_pattern && !*alert_pattern) {
476 + alert_pattern = NULL; // Treat empty string as no alerts specified
477 + }
478 + }
479 +
480 + CLEAN_BUFFER *t = buffer_create(0, NULL);
481 +
482 + struct api_v2_contexts_request req = {
483 + .scope_contexts = metrics_pattern,
484 + .scope_nodes = nodes_pattern,
485 + .contexts = metrics_pattern,
486 + .nodes = nodes_pattern,
487 + .q = q,
488 + .after = after,
489 + .before = before,
490 + .cardinality_limit = cardinality_limit,
491 + .options = config->options | CONTEXTS_OPTION_MCP | CONTEXTS_OPTION_RFC3339 | CONTEXTS_OPTION_JSON_LONG_KEYS | CONTEXTS_OPTION_MINIFY,
492 + .alerts = {
493 + .alert = alert_pattern,
494 + .status = config->defaults.alert_status,
495 + },
496 + };
497 +
498 + // Determine mode - add SEARCH if q is provided
499 + CONTEXTS_V2_MODE mode = config->mode;
500 + if (config->params.has_q && q) {
501 + mode = CONTEXTS_V2_SEARCH;
502 + req.options |= CONTEXTS_OPTION_FAMILY | CONTEXTS_OPTION_UNITS | CONTEXTS_OPTION_TITLES |
503 + CONTEXTS_OPTION_LABELS | CONTEXTS_OPTION_INSTANCES | CONTEXTS_OPTION_DIMENSIONS;
504 + }
505 +
506 + int code = rrdcontext_to_json_v2(t, &req, mode);
507 + if (code != HTTP_RESP_OK) {
508 + buffer_sprintf(mcpc->error, "Failed to fetch %s, query returned http error code %d", config->name, code);
509 + return MCP_RC_ERROR;
510 + }
511 +
512 + // Initialize success response
513 + mcp_init_success_result(mcpc, id);
514 + {
515 + // Start building a content array for the result
516 + buffer_json_member_add_array(mcpc->result, "content");
517 + {
518 + // Return text content for LLM compatibility
519 + buffer_json_add_array_item_object(mcpc->result);
520 + {
521 + buffer_json_member_add_string(mcpc->result, "type", "text");
522 + buffer_json_member_add_string(mcpc->result, "text", buffer_tostring(t));
523 + }
524 + buffer_json_object_close(mcpc->result); // Close text content
525 + }
526 + buffer_json_array_close(mcpc->result); // Close content array
527 + }
528 + buffer_json_object_close(mcpc->result); // Close result object
529 + buffer_json_finalize(mcpc->result); // Finalize the JSON
530 +
531 + return MCP_RC_OK;
532 +}
\ No newline at end of file
src/web/mcp/mcp-tools-list-metadata.h new
+67
@@ -0,0 +1,67 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_LIST_METADATA_H
4 +#define NETDATA_MCP_TOOLS_LIST_METADATA_H
5 +
6 +#include "mcp.h"
7 +#include "web/api/maps/contexts_options.h"
8 +#include "web/api/maps/contexts_alert_statuses.h"
9 +#include "database/contexts/rrdcontext.h"
10 +
11 +// Tool output types - what kind of data the tool returns
12 +typedef enum {
13 + MCP_LIST_OUTPUT_NODES, // Tool returns nodes
14 + MCP_LIST_OUTPUT_METRICS, // Tool returns metrics (contexts)
15 + MCP_LIST_OUTPUT_FUNCTIONS, // Tool returns functions
16 + MCP_LIST_OUTPUT_ALERTS, // Tool returns alerts
17 +} MCP_LIST_OUTPUT_TYPE;
18 +
19 +// Configuration structure for unified list tools
20 +typedef struct mcp_list_tool_config {
21 + const char *name;
22 + const char *title;
23 + const char *description;
24 +
25 + // What type of output this tool returns
26 + MCP_LIST_OUTPUT_TYPE output_type;
27 +
28 + // Mode for rrdcontext_to_json_v2
29 + CONTEXTS_V2_MODE mode;
30 +
31 + // Additional options (MCP is always added)
32 + CONTEXTS_OPTIONS options;
33 +
34 + // Parameters configuration
35 + struct {
36 + bool has_q; // Full-text search
37 + bool has_metrics; // Metrics pattern
38 + bool has_nodes; // Nodes pattern
39 + bool has_instances; // Instances pattern
40 + bool has_dimensions; // Dimensions pattern
41 + bool has_time_range; // Has after/before time parameters
42 + bool has_cardinality_limit; // Has cardinality limit parameter
43 + bool has_alert_pattern; // Has alert name pattern parameter
44 +
45 + bool metrics_required; // Is metrics parameter required?
46 + bool nodes_required; // Is nodes parameter required?
47 + bool nodes_as_array; // Should nodes be an array instead of pattern?
48 + bool metrics_as_array; // Should metrics be an array instead of pattern?
49 + } params;
50 +
51 + // Tool-specific defaults (0 means use global default)
52 + struct {
53 + size_t cardinality_limit;
54 + uint32_t alert_status; // Default alert status filter (CONTEXTS_ALERT_STATUS)
55 + } defaults;
56 +
57 +} MCP_LIST_TOOL_CONFIG;
58 +
59 +// Get tool configuration by name
60 +const MCP_LIST_TOOL_CONFIG *mcp_get_list_tool_config(const char *name);
61 +
62 +// Unified functions
63 +void mcp_unified_list_tool_schema(BUFFER *buffer, const MCP_LIST_TOOL_CONFIG *config);
64 +MCP_RETURN_CODE mcp_unified_list_tool_execute(MCP_CLIENT *mcpc, const MCP_LIST_TOOL_CONFIG *config,
65 + struct json_object *params, MCP_REQUEST_ID id);
66 +
67 +#endif //NETDATA_MCP_TOOLS_LIST_METADATA_H
\ No newline at end of file
src/web/mcp/mcp-tools-query-metrics.c new
+704
@@ -0,0 +1,704 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +/**
4 + * MCP Metrics Query Tool
5 + *
6 + * This tool allows querying metrics data via the Model Context Protocol.
7 + * It provides an interface to the data query engine similar to the API v2 data endpoint.
8 + *
9 + * Query Process:
10 + * 1. The query engine first determines all unique time-series to query by filtering based on context, nodes,
11 + * time-frame, and other supplied filters.
12 + *
13 + * 2. It then queries each time-series, automatically applying over-time-aggregation. For example, if the
14 + * database has 1000 points for a time series and you request 10 points, the query engine reduces the
15 + * 1000 points to 10 using the time_group aggregation function (average, max, min, etc.).
16 + *
17 + * 3. After time aggregation, the query engine applies the group_by aggregation across metrics.
18 + * For example, if querying disk I/O for 10 disks from 2 nodes with 2 dimensions each (read/write),
19 + * you have 40 unique time-series. With group_by=dimension, the engine would:
20 + * - Aggregate all 20 'read' dimensions (from all disks across all nodes) into a single 'read' dimension
21 + * - Aggregate all 20 'write' dimensions (from all disks across all nodes) into a single 'write' dimension
22 + * - Use the specified aggregation function (sum, min, max, average) for this cross-metric aggregation
23 + *
24 + * 4. The result will contain only the grouped dimensions, but with rich metadata:
25 + * - Each data point contains: timestamp, aggregated value, anomaly rate, and quality flags
26 + * - Quality flags indicate whether original data had gaps or counter overflows
27 + *
28 + * 5. When 'jsonwrap' is included in options, the response includes comprehensive statistics about all
29 + * facets of the query, providing aggregated min, max, average, anomaly rate, and volume contribution
30 + * percentages per node, instance, dimension, and label.
31 + */
32 +
33 +#include "mcp-tools-query-metrics.h"
34 +#include "mcp-tools.h"
35 +#include "mcp-params.h"
36 +#include "web/api/formatters/rrd2json.h"
37 +
38 +
39 +// JSON schema for the metrics query tool
40 +void mcp_tool_query_metrics_schema(BUFFER *buffer) {
41 + // Tool input schema
42 + buffer_json_member_add_object(buffer, "inputSchema");
43 + buffer_json_member_add_string(buffer, "type", "object");
44 + buffer_json_member_add_string(buffer, "title", "Query Metrics Data");
45 +
46 + // Properties
47 + buffer_json_member_add_object(buffer, "properties");
48 +
49 + // Selection parameters
50 + mcp_schema_add_string_param(
51 + buffer, "metric", "Metric Name",
52 + "The exact metric (context) to query.\n"
53 + "Use the '" MCP_TOOL_LIST_METRICS "' tool to discover available metrics.",
54 + NULL, true);
55 +
56 + mcp_schema_add_array_param(
57 + buffer, "dimensions",
58 + "Dimensions Filter",
59 + "Array of dimensions to include in the query.\n"
60 + "Examples: [\"read\", \"write\"] or [\"in\", \"out\"] or [\"used\", \"free\", \"cached\"]\n"
61 + "Use the '" MCP_TOOL_GET_METRICS_DETAILS "' tool to discover the available dimensions for a metric.");
62 +
63 + mcp_schema_add_labels_object(
64 + buffer, "Labels Filter",
65 + "Query only the instances with the given labels. "
66 + "Example: {\"disk_type\": [\"ssd\", \"nvme\"], \"mount_point\": [\"/\"]}\n"
67 + "Values in the same array are ORed, different keys are ANDed. "
68 + "Use the '" MCP_TOOL_GET_METRICS_DETAILS "' tool to discover available labels and values for a metric.");
69 +
70 + mcp_schema_add_array_param(
71 + buffer, "instances",
72 + "Instances Filter",
73 + "Query only the given instances.\n"
74 + "Use the '" MCP_TOOL_GET_METRICS_DETAILS "' tool to discover available instances for a metric.\n"
75 + "If no instances are specified, all instances of the metric are queried.\n"
76 + "Example: [\"instance1\", \"instance2\", \"instance3\"]\n."
77 + "IMPORTANT: when you have a choice, prefer to filter by labels instead of instances, because many monitored "
78 + "components may change instance names over time.");
79 +
80 + mcp_schema_add_array_param(
81 + buffer, "nodes",
82 + "Nodes Filter",
83 + "Array of nodes to include in the query.\n"
84 + "If no nodes are specified, all nodes having data for the given metrics in the specified time-frame will be queried.\n"
85 + "Examples: [\"node1\", \"node2\", \"node3\"]\n"
86 + "Use the '" MCP_TOOL_LIST_NODES "' tool to discover the available nodes.");
87 +
88 + // Add cardinality limit
89 + mcp_schema_add_cardinality_limit(buffer,
90 + "Limit the response cardinality (number of dimensions, instances, labels, etc.). "
91 + "When the limit is exceeded, the response will indicate how many items were omitted.",
92 + MCP_DATA_CARDINALITY_LIMIT,
93 + 1, // minimum
94 + MAX(MCP_DATA_CARDINALITY_LIMIT, MCP_DATA_CARDINALITY_LIMIT_MAX));
95 +
96 + // Time parameters
97 + mcp_schema_add_time_params(buffer, "query window", true);
98 +
99 + buffer_json_member_add_object(buffer, "points");
100 + {
101 + buffer_json_member_add_string(buffer, "type", "number");
102 + buffer_json_member_add_string(buffer, "title", "Data Points");
103 + buffer_json_member_add_string(buffer, "description", "Number of data points to return.");
104 + buffer_json_member_add_uint64(buffer, "default", 60);
105 + }
106 + buffer_json_object_close(buffer); // points
107 +
108 + buffer_json_member_add_object(buffer, "timeout");
109 + {
110 + buffer_json_member_add_string(buffer, "type", "number");
111 + buffer_json_member_add_string(buffer, "title", "Timeout");
112 + buffer_json_member_add_string(buffer, "description", "Query timeout in seconds.");
113 + buffer_json_member_add_uint64(buffer, "default", 60);
114 + }
115 + buffer_json_object_close(buffer); // timeout
116 +
117 + buffer_json_member_add_object(buffer, "options");
118 + {
119 + buffer_json_member_add_string(buffer, "type", "string");
120 + buffer_json_member_add_string(buffer, "title", "Query Options");
121 + buffer_json_member_add_string(
122 + buffer, "description",
123 + "Space-separated list of additional query options:\n"
124 + "'percentage': Return values as percentages of total\n"
125 + "'absolute' or 'absolute-sum': Return absolute values for stacked charts\n"
126 + "'display-absolute': Convert percentage values to absolute before application of grouping functions\n"
127 + "'all-dimensions': Include all dimensions, even those with just zero values\n"
128 + "Example: 'absolute percentage'");
129 + }
130 + buffer_json_object_close(buffer); // options
131 +
132 + // Time grouping
133 + buffer_json_member_add_object(buffer, "time_group");
134 + {
135 + buffer_json_member_add_string(buffer, "type", "string");
136 + buffer_json_member_add_string(buffer, "title", "Time Grouping Method");
137 + buffer_json_member_add_string(buffer, "description", "Method to group data points over time. The 'extremes' method returns the maximum value for positive numbers and the minimum value for negative numbers, which is particularly useful for showing the highest peaks in both directions on charts.");
138 + buffer_json_member_add_string(buffer, "default", "average");
139 +
140 + // Define enum of possible values
141 + buffer_json_member_add_array(buffer, "enum");
142 + buffer_json_add_array_item_string(buffer, "average"); // "avg" and "mean" are aliases
143 + buffer_json_add_array_item_string(buffer, "min");
144 + buffer_json_add_array_item_string(buffer, "max");
145 + buffer_json_add_array_item_string(buffer, "sum");
146 + buffer_json_add_array_item_string(buffer, "incremental-sum"); // "incremental_sum" is an alias
147 + buffer_json_add_array_item_string(buffer, "median");
148 + buffer_json_add_array_item_string(buffer, "trimmed-mean");
149 + buffer_json_add_array_item_string(buffer, "trimmed-median");
150 + buffer_json_add_array_item_string(buffer, "percentile"); // requires time_group_options parameter
151 + buffer_json_add_array_item_string(buffer, "stddev"); // standard deviation
152 + buffer_json_add_array_item_string(buffer, "coefficient-of-variation"); // relative standard deviation (cv)
153 + buffer_json_add_array_item_string(buffer, "ema"); // exponential moving average (alias "ses" or "ewma")
154 + buffer_json_add_array_item_string(buffer, "des"); // double exponential smoothing
155 + buffer_json_add_array_item_string(buffer, "countif"); // requires time_group_options parameter
156 + buffer_json_add_array_item_string(buffer, "extremes"); // for each time frame, returns max for positive values and min for negative values
157 + buffer_json_array_close(buffer);
158 + }
159 + buffer_json_object_close(buffer); // time_group
160 +
161 + buffer_json_member_add_object(buffer, "time_group_options");
162 + {
163 + buffer_json_member_add_string(buffer, "type", "string");
164 + buffer_json_member_add_string(buffer, "title", "Time Group Options");
165 + buffer_json_member_add_string(
166 + buffer, "description",
167 + "Additional options for time grouping.\n"
168 + "For 'percentile', specify a percentage (0-100).\n"
169 + "For 'countif', specify a comparison operator and value (e.g., '>0', '=0', '!=0', '<=10').");
170 + }
171 + buffer_json_object_close(buffer); // time_group_options
172 +
173 + // Tier selection
174 + buffer_json_member_add_object(buffer, "tier");
175 + {
176 + buffer_json_member_add_string(buffer, "type", "number");
177 + buffer_json_member_add_string(buffer, "title", "Storage Tier");
178 + buffer_json_member_add_string(
179 + buffer, "description",
180 + "Storage tier to query from.\n"
181 + "If not specified, Netdata will automatically pick the best tier based on the time-frame and points requested.\n"
182 + "CAUTION: specifying a high-resolution tier (like 0) over long time-frames (like days) may consume significant system resources.");
183 + }
184 + buffer_json_object_close(buffer); // tier
185 +
186 + // Group by parameters
187 + buffer_json_member_add_object(buffer, "group_by");
188 + {
189 + buffer_json_member_add_string(buffer, "type", "array");
190 + buffer_json_member_add_string(buffer, "title", "Group By");
191 + buffer_json_member_add_string(
192 + buffer, "description",
193 + "Specifies how to group metrics across different time-series.\n"
194 + "- 'dimension': Groups by dimension name across all instances/nodes. Example: for disks it provides the aggregate of reads and writes across all disks of all nodes.\n"
195 + "- 'instance': Groups by instance across all nodes. Example: for disks, it provides the aggregate per disk name (sda, sdb, etc), aggregating their reads and writes, across all nodes.\n"
196 + "- 'node': Groups by node. Example: for disks, it provides one metric per node, aggregating reads and writes across all its disks.\n"
197 + "- 'label': Groups by the given label key (use the parameter 'group_by_label' to set the key). Example: for disks, aggregate over key 'disk_type' to get an group all 'physical', 'virtual' and 'partition' separately.\n"
198 + "Multiple groupings can be combined. Example: '[\"dimension\", \"label\"]'.");
199 + buffer_json_member_add_array(buffer, "default");
200 + buffer_json_add_array_item_string(buffer, "dimension");
201 + buffer_json_array_close(buffer);
202 +
203 + // Define items schema with enum values
204 + buffer_json_member_add_object(buffer, "items");
205 + {
206 + buffer_json_member_add_string(buffer, "type", "string");
207 + buffer_json_member_add_array(buffer, "enum");
208 + buffer_json_add_array_item_string(buffer, "dimension");
209 + buffer_json_add_array_item_string(buffer, "instance");
210 + buffer_json_add_array_item_string(buffer, "node");
211 + buffer_json_add_array_item_string(buffer, "label");
212 +
213 + // we don't offer these to MCP clients.
214 + // buffer_json_add_array_item_string(buffer, "context");
215 + // buffer_json_add_array_item_string(buffer, "units");
216 + buffer_json_array_close(buffer);
217 + }
218 + buffer_json_object_close(buffer); // items
219 + }
220 + buffer_json_object_close(buffer); // group_by
221 +
222 + mcp_schema_add_string_param(
223 + buffer, "group_by_label",
224 + "Group By Label",
225 + "When 'group_by' includes 'label', this parameter specifies the label key to group by.\n"
226 + "Example: if metrics have an 'interface_type' label with values like 'real' or 'virtual', "
227 + "setting 'group_by_label' to 'interface_type' would aggregate metrics separately for physical and virtual network interfaces.",
228 + NULL, false);
229 +
230 + buffer_json_member_add_object(buffer, "aggregation");
231 + {
232 + buffer_json_member_add_string(buffer, "type", "string");
233 + buffer_json_member_add_string(buffer, "title", "Aggregation Method");
234 + buffer_json_member_add_string(
235 + buffer, "description",
236 + "Method to use when aggregating grouped metrics.\n"
237 + "- 'sum': Sum of all grouped metrics (useful for additive metrics like bytes transferred, operations, etc.)\n"
238 + "- 'min': Minimum value among all grouped metrics (useful for finding best performance metrics)\n"
239 + "- 'max': Maximum value among all grouped metrics (useful for finding worst performance metrics, peak resource usage)\n"
240 + "- 'extremes': When values are both positive and negative, shows the maximum value for positive metrics and the minimum value for negative metrics\n"
241 + "- 'average': Average of all grouped metrics (CAUTION: When 'group_by' doesn't include 'dimension', this averages different metric types together - e.g., CPU user + system + idle - which is rarely meaningful)\n"
242 + "- 'percentage': Expresses each grouped metric as a percentage of its group's total (useful for seeing proportional contributions)\n");
243 +
244 + // Define enum of possible values
245 + buffer_json_member_add_array(buffer, "enum");
246 + buffer_json_add_array_item_string(buffer, "sum");
247 + buffer_json_add_array_item_string(buffer, "min");
248 + buffer_json_add_array_item_string(buffer, "max");
249 + buffer_json_add_array_item_string(buffer, "extremes");
250 + buffer_json_add_array_item_string(buffer, "average");
251 + buffer_json_add_array_item_string(buffer, "percentage");
252 + buffer_json_array_close(buffer);
253 + }
254 + buffer_json_object_close(buffer); // aggregation
255 +
256 + buffer_json_object_close(buffer); // properties
257 +
258 + // Required fields
259 + buffer_json_member_add_array(buffer, "required");
260 + buffer_json_add_array_item_string(buffer, "metric");
261 + buffer_json_add_array_item_string(buffer, "dimensions");
262 + buffer_json_add_array_item_string(buffer, "after");
263 + buffer_json_add_array_item_string(buffer, "before");
264 + buffer_json_add_array_item_string(buffer, "points");
265 + buffer_json_add_array_item_string(buffer, "time_group");
266 + buffer_json_add_array_item_string(buffer, "group_by");
267 + buffer_json_add_array_item_string(buffer, "aggregation");
268 + buffer_json_add_array_item_string(buffer, "cardinality_limit");
269 + buffer_json_array_close(buffer);
270 +
271 + buffer_json_object_close(buffer); // inputSchema
272 +}
273 +
274 +// Structure to hold interruption data
275 +typedef struct {
276 + MCP_CLIENT *mcpc;
277 + MCP_REQUEST_ID id;
278 +} mcp_query_interrupt_data;
279 +
280 +// Interrupt callback for query execution
281 +static bool mcp_query_interrupt_callback(void *data) {
282 + mcp_query_interrupt_data *int_data = (mcp_query_interrupt_data *)data;
283 +
284 + // Check if the MCP client is still valid and connected
285 + if (!int_data || !int_data->mcpc)
286 + return false;
287 +
288 + // Real implementations might check for client disconnection or timeout
289 + // Here we're just returning false to indicate "no interrupt"
290 + return false;
291 +}
292 +
293 +// Removed extract_string_param and extract_size_param - now using mcp-params functions
294 +
295 +// Execute the metrics query
296 +MCP_RETURN_CODE mcp_tool_query_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
297 + if (!mcpc || id == 0)
298 + return MCP_RC_ERROR;
299 +
300 + buffer_flush(mcpc->result);
301 +
302 + usec_t received_ut = now_monotonic_usec();
303 +
304 + // Extract and validate context parameter
305 + const char *context = mcp_params_extract_string(params, "metric", NULL);
306 +
307 + // Validate required parameters with detailed error messages
308 + if (!context || !*context) {
309 + buffer_sprintf(mcpc->error, "Missing required parameter 'metric'. Use the '" MCP_TOOL_LIST_METRICS "' tool to discover available metrics/contexts.");
310 + return MCP_RC_BAD_REQUEST;
311 + }
312 +
313 + // Check if context contains patterns
314 + if (simple_pattern_contains_wildcards(context, SIMPLE_PATTERN_DEFAULT_WEB_SEPARATORS)) {
315 + buffer_sprintf(mcpc->error, "The 'context' parameter must be an exact context name, not a pattern. "
316 + "Wildcards or pattern separators are not supported. "
317 + "Use the " MCP_TOOL_LIST_METRICS " tool to discover exact context names.");
318 + return MCP_RC_BAD_REQUEST;
319 + }
320 +
321 + // Check if all required parameters are provided
322 + struct json_object *after_obj = NULL, *before_obj = NULL, *points_obj = NULL, *time_group_obj = NULL;
323 + struct json_object *group_by_obj = NULL, *aggregation_obj = NULL, *dimensions_obj = NULL;
324 +
325 + if (!json_object_object_get_ex(params, "dimensions", &dimensions_obj) || !dimensions_obj) {
326 + buffer_sprintf(mcpc->error, "Missing required parameter 'dimensions'. Use the '" MCP_TOOL_LIST_METRICS "' to get the list of dimensions for this metric/context.");
327 + return MCP_RC_BAD_REQUEST;
328 + }
329 +
330 + if (!json_object_object_get_ex(params, "after", &after_obj) || !after_obj) {
331 + buffer_sprintf(mcpc->error, "Missing required parameter 'after'. This parameter defines the start time for your query (Unix epoch timestamp in seconds, or negative value relative to 'before', or RFC3339 datetime string).");
332 + return MCP_RC_BAD_REQUEST;
333 + }
334 +
335 + if (!json_object_object_get_ex(params, "before", &before_obj) || !before_obj) {
336 + buffer_sprintf(mcpc->error, "Missing required parameter 'before'. This parameter defines the end time for your query (Unix epoch timestamp in seconds, or negative value relative to now, or RFC3339 datetime string).");
337 + return MCP_RC_BAD_REQUEST;
338 + }
339 +
340 + if (!json_object_object_get_ex(params, "points", &points_obj) || !points_obj) {
341 + buffer_sprintf(mcpc->error, "Missing required parameter 'points'. This parameter defines how many data points to return in your result set (e.g., 60 for minute-level granularity in an hour).");
342 + return MCP_RC_BAD_REQUEST;
343 + }
344 +
345 + if (!json_object_object_get_ex(params, "time_group", &time_group_obj) || !time_group_obj) {
346 + buffer_sprintf(mcpc->error, "Missing required parameter 'time_group'. This parameter defines how to aggregate data points over time (e.g., 'average', 'min', 'max', 'sum').");
347 + return MCP_RC_BAD_REQUEST;
348 + }
349 +
350 + if (!json_object_object_get_ex(params, "group_by", &group_by_obj) || !group_by_obj) {
351 + buffer_sprintf(mcpc->error, "Missing required parameter 'group_by'. This parameter defines how to group metrics (e.g., 'dimension', 'instance', 'node', or combinations like 'dimension,node').");
352 + return MCP_RC_BAD_REQUEST;
353 + }
354 +
355 + if (!json_object_object_get_ex(params, "aggregation", &aggregation_obj) || !aggregation_obj) {
356 + buffer_sprintf(mcpc->error, "Missing required parameter 'aggregation'. This parameter defines the function to use when aggregating metrics (e.g., 'sum', 'min', 'max', 'average').");
357 + return MCP_RC_BAD_REQUEST;
358 + }
359 +
360 + struct json_object *cardinality_limit_obj = NULL;
361 + if (!json_object_object_get_ex(params, "cardinality_limit", &cardinality_limit_obj) || !cardinality_limit_obj) {
362 + buffer_sprintf(mcpc->error, "Missing required parameter 'cardinality_limit'. This parameter limits the number of items returned to keep response sizes manageable (default: %d).", MCP_DATA_CARDINALITY_LIMIT);
363 + return MCP_RC_BAD_REQUEST;
364 + }
365 +
366 + // Get time_group value to check if it's a percentile or countif
367 + const char *time_group_str = NULL;
368 + if (json_object_is_type(time_group_obj, json_type_string)) {
369 + time_group_str = json_object_get_string(time_group_obj);
370 +
371 + // Check if time_group_options is required based on time_group
372 + if (time_group_str && (
373 + strcmp(time_group_str, "percentile") == 0 ||
374 + strcmp(time_group_str, "countif") == 0)) {
375 +
376 + struct json_object *time_group_options_obj = NULL;
377 + if (!json_object_object_get_ex(params, "time_group_options", &time_group_options_obj) || !time_group_options_obj) {
378 + if (strcmp(time_group_str, "percentile") == 0) {
379 + buffer_sprintf(mcpc->error, "Missing required parameter 'time_group_options' when using time_group='percentile'. You must specify a percentage value between 0-100 (e.g., '95' for 95th percentile).");
380 + } else {
381 + buffer_sprintf(mcpc->error, "Missing required parameter 'time_group_options' when using time_group='countif'. You must specify a comparison operator and value (e.g., '>0', '=0', '!=0', '<=10').");
382 + }
383 + return MCP_RC_BAD_REQUEST;
384 + }
385 + }
386 + }
387 +
388 + // Handle nodes array parameter
389 + CLEAN_BUFFER *nodes_buffer = NULL;
390 +
391 + nodes_buffer = mcp_params_parse_array_to_pattern(params, "nodes", false, false, MCP_TOOL_LIST_NODES, mcpc->error);
392 + if (buffer_strlen(mcpc->error) > 0) {
393 + return MCP_RC_BAD_REQUEST;
394 + }
395 +
396 + // Handle instances array parameter
397 + CLEAN_BUFFER *instances_buffer = NULL;
398 +
399 + instances_buffer = mcp_params_parse_array_to_pattern(params, "instances", false, false, MCP_TOOL_GET_METRICS_DETAILS, mcpc->error);
400 + if (buffer_strlen(mcpc->error) > 0) {
401 + return MCP_RC_BAD_REQUEST;
402 + }
403 +
404 + // Handle dimensions array parameter
405 + CLEAN_BUFFER *dimensions_buffer = NULL;
406 +
407 + dimensions_buffer = mcp_params_parse_array_to_pattern(params, "dimensions", true, false, MCP_TOOL_GET_METRICS_DETAILS, mcpc->error);
408 + if (buffer_strlen(mcpc->error) > 0) {
409 + buffer_strcat(mcpc->error, ". You must explicitly list every dimension you want to query. "
410 + "Use the '" MCP_TOOL_GET_METRICS_DETAILS "' tool to discover available dimensions for the context.");
411 + return MCP_RC_BAD_REQUEST;
412 + }
413 + // Handle labels - expects a structured object only
414 + CLEAN_BUFFER *labels_buffer = NULL;
415 +
416 + labels_buffer = mcp_params_parse_labels_object(params, MCP_TOOL_GET_METRICS_DETAILS, mcpc->error);
417 + if (buffer_strlen(mcpc->error) > 0) {
418 + return MCP_RC_BAD_REQUEST;
419 + }
420 +
421 + // Removed alerts parameter - not used in query_metrics
422 +
423 + // Time parameters
424 + time_t after = mcp_params_parse_time(params, "after", MCP_DEFAULT_AFTER_TIME);
425 + time_t before = mcp_params_parse_time(params, "before", MCP_DEFAULT_BEFORE_TIME);
426 +
427 + // Validate time range
428 + if (after == 0 && before == 0) {
429 + buffer_sprintf(mcpc->error, "Invalid time range: both 'after' and 'before' cannot be zero. Use negative values for relative times (e.g., after=-3600, before=-0 for the last hour) or specific timestamps for absolute times.");
430 + return MCP_RC_BAD_REQUEST;
431 + }
432 +
433 + // Check if after is later than before (when both are absolute timestamps)
434 + if (after > 0 && before > 0 && after >= before) {
435 + buffer_sprintf(mcpc->error, "Invalid time range: 'after' (%lld) must be earlier than 'before' (%lld). The query time range must be at least 1 second.", (long long)after, (long long)before);
436 + return MCP_RC_BAD_REQUEST;
437 + }
438 +
439 + // No need to check aggregation_obj here - we already have a default in group_by struct
440 +
441 + // Other parameters
442 + size_t points = mcp_params_extract_size(params, "points", 0, 0, SIZE_MAX, mcpc->error);
443 + if (buffer_strlen(mcpc->error) > 0) {
444 + return MCP_RC_BAD_REQUEST;
445 + }
446 +
447 + size_t cardinality_limit = mcp_params_extract_size(params, "cardinality_limit", MCP_DATA_CARDINALITY_LIMIT, 1, 1000, mcpc->error);
448 + if (buffer_strlen(mcpc->error) > 0) {
449 + return MCP_RC_BAD_REQUEST;
450 + }
451 +
452 + // Check if points is more than 1000
453 + if (points < 1) {
454 + buffer_sprintf(mcpc->error,
455 + "Too few data points requested: %zu. The minimum allowed is 1 point.",
456 + points);
457 + return MCP_RC_BAD_REQUEST;
458 + }
459 +
460 + // Check if points is more than 1000
461 + if (points > 1000) {
462 + buffer_sprintf(mcpc->error,
463 + "Too many data points requested: %zu. The maximum allowed is 1000 points. Please reduce the 'points' parameter value to 1000 or less.\n"
464 + "This limit helps reduce response size and save context space when used with AI assistants.",
465 + points);
466 + return MCP_RC_BAD_REQUEST;
467 + }
468 +
469 + long timeout = mcp_params_extract_timeout(params, "timeout", 0, 0, 3600, mcpc->error);
470 + if (buffer_strlen(mcpc->error) > 0) {
471 + return MCP_RC_BAD_REQUEST;
472 + }
473 +
474 + const char *options_str = mcp_params_extract_string(params, "options", NULL);
475 + RRDR_OPTIONS options = 0;
476 + if (options_str && *options_str)
477 + options |= rrdr_options_parse(options_str);
478 +
479 + // Time grouping
480 + RRDR_TIME_GROUPING time_group = RRDR_GROUPING_AVERAGE;
481 + if (time_group_str && *time_group_str)
482 + time_group = time_grouping_parse(time_group_str, RRDR_GROUPING_AVERAGE);
483 +
484 + const char *time_group_options = mcp_params_extract_string(params, "time_group_options", NULL);
485 +
486 + // Tier selection (give an invalid default to now the caller added a tier to the query)
487 + size_t tier = mcp_params_extract_size(params, "tier", nd_profile.storage_tiers + 1, 0, SIZE_MAX, mcpc->error);
488 + if (buffer_strlen(mcpc->error) > 0) {
489 + return MCP_RC_BAD_REQUEST;
490 + }
491 + if (tier < nd_profile.storage_tiers)
492 + options |= RRDR_OPTION_SELECTED_TIER;
493 + else
494 + tier = 0;
495 +
496 + // Group by parameters (simplified - in real implementation handle multiple passes)
497 + struct group_by_pass group_by[MAX_QUERY_GROUP_BY_PASSES] = {
498 + {
499 + .group_by = RRDR_GROUP_BY_NONE,
500 + .group_by_label = NULL,
501 + .aggregation = RRDR_GROUP_BY_FUNCTION_AVERAGE,
502 + },
503 + };
504 +
505 + // Handle group_by array parameter and convert to comma-separated string
506 + CLEAN_BUFFER *group_by_buffer = NULL;
507 + const char *group_by_str = NULL;
508 +
509 + group_by_buffer = mcp_params_parse_array_to_pattern(params, "group_by", true, false, MCP_TOOL_GET_METRICS_DETAILS, mcpc->error);
510 + if (buffer_strlen(mcpc->error) > 0) {
511 + return MCP_RC_BAD_REQUEST;
512 + }
513 +
514 + if (group_by_buffer && buffer_strlen(group_by_buffer) > 0) {
515 + group_by_str = buffer_tostring(group_by_buffer);
516 + group_by[0].group_by = group_by_parse(group_by_str);
517 + }
518 +
519 + const char *group_by_label = mcp_params_extract_string(params, "group_by_label", NULL);
520 + if (group_by_label && *group_by_label) {
521 + group_by[0].group_by_label = (char *)group_by_label;
522 + group_by[0].group_by |= RRDR_GROUP_BY_LABEL;
523 + }
524 +
525 + const char *aggregation_str = mcp_params_extract_string(params, "aggregation", NULL);
526 + if (aggregation_str && *aggregation_str)
527 + group_by[0].aggregation = group_by_aggregate_function_parse(aggregation_str);
528 +
529 + // Create interrupt callback data
530 + mcp_query_interrupt_data interrupt_data = {
531 + .mcpc = mcpc,
532 + .id = id
533 + };
534 +
535 + // Prepare a query target request
536 + QUERY_TARGET_REQUEST qtr = {
537 + .version = 3,
538 + .scope_nodes = buffer_tostring(nodes_buffer), // Use nodes as scope_nodes
539 + .scope_contexts = context, // Use the single context as scope_contexts
540 + .scope_instances = buffer_tostring(instances_buffer), // Use instances as scope_instances for MCP
541 + .scope_labels = buffer_tostring(labels_buffer), // Use labels as scope_labels for MCP
542 + .scope_dimensions = buffer_tostring(dimensions_buffer), // Use dimensions as scope_dimensions for MCP
543 + .after = after,
544 + .before = before,
545 + .host = NULL,
546 + .st = NULL,
547 + .nodes = NULL, // Don't use the 'nodes' parameter here (we use scope_nodes)
548 + .contexts = NULL, // Don't use the 'contexts' parameter here (we use scope_contexts)
549 + .instances = NULL, // Don't use the 'instances' parameter here (we use scope_instances)
550 + .dimensions = NULL, // Don't use the 'dimensions' parameter here (we use scope_dimensions)
551 + .alerts = NULL,
552 + .timeout_ms = (int)(timeout * MSEC_PER_SEC),
553 + .points = points,
554 + .format = DATASOURCE_JSON2,
555 + .options = options |
556 + RRDR_OPTION_ABSOLUTE | RRDR_OPTION_JSON_WRAP | RRDR_OPTION_RETURN_JWAR |
557 + RRDR_OPTION_VIRTUAL_POINTS | RRDR_OPTION_NOT_ALIGNED | RRDR_OPTION_NONZERO |
558 + RRDR_OPTION_MINIFY | RRDR_OPTION_MINIMAL_STATS | RRDR_OPTION_LONG_JSON_KEYS |
559 + RRDR_OPTION_MCP_INFO | RRDR_OPTION_RFC3339,
560 + .time_group_method = time_group,
561 + .time_group_options = time_group_options,
562 + .resampling_time = 0,
563 + .tier = tier,
564 + .chart_label_key = NULL,
565 + .labels = NULL, // Don't use labels parameter here (we use scope_labels)
566 + .query_source = QUERY_SOURCE_API_DATA,
567 + .priority = STORAGE_PRIORITY_NORMAL,
568 + .received_ut = received_ut,
569 + .cardinality_limit = cardinality_limit,
570 +
571 + .interrupt_callback = mcp_query_interrupt_callback,
572 + .interrupt_callback_data = &interrupt_data,
573 +
574 + .transaction = NULL, // No transaction for MCP
575 + };
576 +
577 + // Copy group_by structures
578 + for (size_t g = 0; g < MAX_QUERY_GROUP_BY_PASSES; g++)
579 + qtr.group_by[g] = group_by[g];
580 +
581 + // Create a query target
582 + QUERY_TARGET *qt = query_target_create(&qtr);
583 + if (!qt) {
584 + buffer_sprintf(mcpc->error, "Failed to prepare the query.");
585 + return MCP_RC_INTERNAL_ERROR;
586 + }
587 +
588 + // Create a temporary buffer for the query result
589 + CLEAN_BUFFER *tmp_buffer = buffer_create(0, NULL);
590 +
591 + // Prepare onewayalloc for query execution
592 + ONEWAYALLOC *owa = onewayalloc_create(0);
593 +
594 + // Execute the query and get the data
595 + time_t latest_timestamp = 0;
596 + int ret = data_query_execute(owa, tmp_buffer, qt, &latest_timestamp);
597 +
598 + // Clean up
599 + query_target_release(qt);
600 + onewayalloc_destroy(owa);
601 +
602 + if (ret != HTTP_RESP_OK) {
603 + buffer_flush(mcpc->result);
604 + const char *error_desc = "unknown error";
605 +
606 + // Map common HTTP error codes to more descriptive messages
607 + switch (ret) {
608 + case HTTP_RESP_BAD_REQUEST:
609 + error_desc = "bad request parameters";
610 + break;
611 + case HTTP_RESP_NOT_FOUND:
612 + error_desc = "metric/context not found";
613 + break;
614 + case HTTP_RESP_GATEWAY_TIMEOUT:
615 + case HTTP_RESP_SERVICE_UNAVAILABLE:
616 + error_desc = "timeout or service unavailable";
617 + break;
618 + case HTTP_RESP_INTERNAL_SERVER_ERROR:
619 + error_desc = "internal server error";
620 + break;
621 + default:
622 + break;
623 + }
624 +
625 + buffer_sprintf(mcpc->error, "Failed to execute query: %s (http error code: %d). The context '%s' might not exist, or no data is available for the specified time range.",
626 + error_desc, ret, context);
627 + return MCP_RC_INTERNAL_ERROR;
628 + }
629 +
630 + // Check if instance filtering or grouping is used
631 + bool using_instances = (instances_buffer && buffer_strlen(instances_buffer) > 0) ||
632 + (group_by[0].group_by & RRDR_GROUP_BY_INSTANCE);
633 +
634 + // Return the raw query engine response as-is
635 + mcp_init_success_result(mcpc, id);
636 + {
637 + buffer_json_member_add_array(mcpc->result, "content");
638 + {
639 + // Main result content
640 + buffer_json_add_array_item_object(mcpc->result);
641 + {
642 + buffer_json_member_add_string(mcpc->result, "type", "text");
643 + buffer_json_member_add_string(mcpc->result, "text", buffer_tostring(tmp_buffer));
644 + }
645 + buffer_json_object_close(mcpc->result);
646 +
647 + // Add a warning about potentially misleading aggregation
648 + bool warn_aggregation = false;
649 + // Only warn if using average without dimension grouping AND multiple dimensions selected
650 + int dimensions_count = (int)json_object_array_length(dimensions_obj);
651 + if (dimensions_count > 1 &&
652 + group_by[0].aggregation == RRDR_GROUP_BY_FUNCTION_AVERAGE &&
653 + !(group_by[0].group_by & RRDR_GROUP_BY_DIMENSION)) {
654 + warn_aggregation = true;
655 + }
656 +
657 + if (warn_aggregation) {
658 + buffer_json_add_array_item_object(mcpc->result);
659 + {
660 + buffer_json_member_add_string(mcpc->result, "type", "text");
661 + buffer_json_member_add_string(mcpc->result, "text",
662 + "⚠️ WARNING: Potentially Misleading Aggregation\n\n"
663 + "You are using 'average' aggregation without including 'dimension' in group_by. "
664 + "This means different metric types are being averaged together, which rarely produces meaningful results.\n\n"
665 + "For example:\n"
666 + "- For CPU metrics: averaging user, system, idle, wait states together\n"
667 + "- For network metrics: averaging in/out traffic together\n"
668 + "- For disk I/O: averaging reads and writes together\n\n"
669 + "Check the 'aggregated' field in view.dimensions to see how many time-series were combined. "
670 + "Values greater than 1 indicate multiple different metrics were averaged together.\n\n"
671 + "Consider using:\n"
672 + "- 'sum' aggregation for additive metrics\n"
673 + "- Include 'dimension' in group_by (e.g., 'instance,dimension')\n"
674 + "- Review the summary section to understand what's being aggregated");
675 + }
676 + buffer_json_object_close(mcpc->result);
677 + }
678 +
679 + // Add an instance usage warning if applicable
680 + if (using_instances) {
681 + buffer_json_add_array_item_object(mcpc->result);
682 + {
683 + buffer_json_member_add_string(mcpc->result, "type", "text");
684 + buffer_json_member_add_string(mcpc->result, "text",
685 + "⚠️ Instance Usage Notice: Instance filtering/grouping behavior varies by collector type:\n\n"
686 + "- **Stable instances** (systemd services, cgroups): Instance names are typically stable and match their labels. "
687 + "Filtering by instance works reliably.\n\n"
688 + "- **Dynamic instances** (Kubernetes pods, containers, processes): Instance names often contain random IDs or session identifiers. "
689 + "Each restart creates a new instance. For these, filtering/grouping by labels is recommended to see the complete picture across all instances.\n\n"
690 + "- **Detecting restarts**: Grouping by labels and examining instance counts can reveal restart patterns - "
691 + "multiple instances with the same labels but different names often indicate restarts or scaling events.\n\n"
692 + "Best practice: Check if your target system uses stable or dynamic instances. When in doubt, group by labels for comprehensive data, "
693 + "then examine instance patterns for additional insights.");
694 + }
695 + buffer_json_object_close(mcpc->result);
696 + }
697 + }
698 + buffer_json_array_close(mcpc->result); // Close content array
699 + }
700 + buffer_json_object_close(mcpc->result); // Close result object
701 + buffer_json_finalize(mcpc->result); // Finalize the JSON
702 +
703 + return MCP_RC_OK;
704 +}
\ No newline at end of file
src/web/mcp/mcp-tools-query-metrics.h new
+14
@@ -0,0 +1,14 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_METRICS_QUERY_H
4 +#define NETDATA_MCP_TOOLS_METRICS_QUERY_H
5 +
6 +#include "mcp-tools.h"
7 +
8 +// Schema definition function - provides JSON schema for this tool
9 +void mcp_tool_query_metrics_schema(BUFFER *buffer);
10 +
11 +// Execution function - performs the metrics query
12 +MCP_RETURN_CODE mcp_tool_query_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
13 +
14 +#endif // NETDATA_MCP_TOOLS_METRICS_QUERY_H
\ No newline at end of file
src/web/mcp/mcp-tools-weights.c new
+389
@@ -0,0 +1,389 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#include "mcp-tools-weights.h"
4 +#include "mcp-params.h"
5 +#include "web/api/web_api.h"
6 +#include "web/api/queries/weights.h"
7 +#include "web/api/queries/query.h"
8 +
9 +// Common function to execute the 'weights' request
10 +static MCP_RETURN_CODE execute_weights_request(
11 + MCP_CLIENT *mcpc,
12 + struct json_object *params,
13 + MCP_REQUEST_ID id,
14 + WEIGHTS_METHOD method,
15 + const char *default_time_group
16 +) {
17 + // Extract time parameters using common parsing functions
18 + time_t after = mcp_params_parse_time(params, "after", MCP_DEFAULT_AFTER_TIME);
19 + time_t before = mcp_params_parse_time(params, "before", MCP_DEFAULT_BEFORE_TIME);
20 + time_t baseline_after = 0;
21 + time_t baseline_before = 0;
22 +
23 + // For correlation methods (KS2, VOLUME), parse baseline times
24 + if (method == WEIGHTS_METHOD_MC_KS2 || method == WEIGHTS_METHOD_MC_VOLUME) {
25 + baseline_after = mcp_params_parse_time(params, "baseline_after", 0);
26 + baseline_before = mcp_params_parse_time(params, "baseline_before", 0);
27 +
28 + // If baseline not specified, auto-calculate as 4x the query window before the query window
29 + if (baseline_after == 0 && baseline_before == 0) {
30 + time_t window = before - after;
31 + baseline_before = after;
32 + baseline_after = baseline_before - (window * 4);
33 + }
34 + }
35 +
36 + // Parse filter parameters using common parsing functions
37 + // Use CLEAN_BUFFER for automatic cleanup
38 + CLEAN_BUFFER *metrics_buffer = NULL;
39 + CLEAN_BUFFER *nodes_buffer = NULL;
40 + CLEAN_BUFFER *instances_buffer = NULL;
41 + CLEAN_BUFFER *dimensions_buffer = NULL;
42 + CLEAN_BUFFER *labels_buffer = NULL;
43 +
44 + // Parse metrics as an array
45 + metrics_buffer = mcp_params_parse_array_to_pattern(params, "metrics", false, false, MCP_TOOL_LIST_METRICS, mcpc->error);
46 + if (buffer_strlen(mcpc->error) > 0) {
47 + return MCP_RC_BAD_REQUEST;
48 + }
49 +
50 + // Parse nodes as array
51 + nodes_buffer = mcp_params_parse_array_to_pattern(params, "nodes", false, false, MCP_TOOL_LIST_NODES, mcpc->error);
52 + if (buffer_strlen(mcpc->error) > 0) {
53 + return MCP_RC_BAD_REQUEST;
54 + }
55 +
56 + // Parse instances as an array
57 + instances_buffer = mcp_params_parse_array_to_pattern(params, "instances", false, false, MCP_TOOL_GET_METRICS_DETAILS, mcpc->error);
58 + if (buffer_strlen(mcpc->error) > 0) {
59 + return MCP_RC_BAD_REQUEST;
60 + }
61 +
62 + // Parse dimensions as array
63 + dimensions_buffer = mcp_params_parse_array_to_pattern(params, "dimensions", false, false, MCP_TOOL_GET_METRICS_DETAILS, mcpc->error);
64 + if (buffer_strlen(mcpc->error) > 0) {
65 + return MCP_RC_BAD_REQUEST;
66 + }
67 +
68 + // Parse labels as object
69 + labels_buffer = mcp_params_parse_labels_object(params, MCP_TOOL_GET_METRICS_DETAILS, mcpc->error);
70 + if (buffer_strlen(mcpc->error) > 0) {
71 + return MCP_RC_BAD_REQUEST;
72 + }
73 +
74 + // Get cardinality limit
75 + struct json_object *obj;
76 + size_t cardinality_limit = 50; // Default for MCP
77 + if (json_object_object_get_ex(params, "cardinality_limit", &obj) && json_object_is_type(obj, json_type_int))
78 + cardinality_limit = json_object_get_int(obj);
79 +
80 + // Extract timeout parameter
81 + int timeout = mcp_params_extract_timeout(params, "timeout", 120, 1, 3600, mcpc->error);
82 + if (buffer_strlen(mcpc->error) > 0) {
83 + return MCP_RC_BAD_REQUEST;
84 + }
85 +
86 + // Set time_group parameter based on the method
87 + const char *time_group_options = default_time_group;
88 + RRDR_TIME_GROUPING time_group_method = RRDR_GROUPING_AVERAGE;
89 +
90 + // For find_unstable_metrics (WEIGHTS_METHOD_VALUE with cv default), always use CV
91 + if (method == WEIGHTS_METHOD_VALUE && default_time_group && strcmp(default_time_group, "cv") == 0) {
92 + time_group_method = RRDR_GROUPING_CV;
93 + time_group_options = "cv";
94 + } else if (default_time_group) {
95 + // Use the default time grouping specified by the tool
96 + time_group_method = time_grouping_parse(default_time_group, RRDR_GROUPING_AVERAGE);
97 + }
98 +
99 + // Set options
100 + RRDR_OPTIONS options = RRDR_OPTION_NOT_ALIGNED | RRDR_OPTION_NULL2ZERO | RRDR_OPTION_ABSOLUTE | RRDR_OPTION_NONZERO;
101 +
102 + // Build the 'weights' request structure
103 + QUERY_WEIGHTS_REQUEST qwr = {
104 + .version = 2,
105 + .host = NULL, // Will query all hosts
106 + .scope_nodes = buffer_tostring(nodes_buffer),
107 + .scope_contexts = buffer_tostring(metrics_buffer),
108 + .scope_instances = buffer_tostring(instances_buffer),
109 + .scope_labels = buffer_tostring(labels_buffer),
110 + .scope_dimensions = buffer_tostring(dimensions_buffer),
111 + .nodes = NULL,
112 + // exclude netdata internal metrics
113 + // exclude system interrupts and CPU interrupts, while are fragile
114 + .contexts = "!netdata.*|!system.interrupts|!system.intr|!cpu.interrupts|*",
115 + .instances = NULL,
116 + .dimensions = NULL,
117 + .labels = NULL,
118 + .alerts = NULL,
119 + .group_by = {
120 + .group_by = RRDR_GROUP_BY_NONE,
121 + .group_by_label = NULL,
122 + .aggregation = RRDR_GROUP_BY_FUNCTION_AVERAGE,
123 + },
124 + .method = method,
125 + .format = WEIGHTS_FORMAT_MCP,
126 + .time_group_method = time_group_method,
127 + .time_group_options = time_group_options,
128 + .baseline_after = baseline_after,
129 + .baseline_before = baseline_before,
130 + .after = after,
131 + .before = before,
132 + .points = 500, // Default points for weights
133 + .options = options,
134 + .tier = 0,
135 + .timeout_ms = (int)(timeout * 1000), // Convert seconds to milliseconds
136 + .cardinality_limit = cardinality_limit,
137 + .interrupt_callback = NULL,
138 + .interrupt_callback_data = NULL,
139 + .transaction = NULL,
140 + };
141 +
142 + // Create a temporary buffer for the 'weights' API response
143 + CLEAN_BUFFER *tmp_buffer = buffer_create(0, NULL);
144 +
145 + // Call the weights API function with the temporary buffer
146 + int http_code = web_api_v12_weights(tmp_buffer, &qwr);
147 +
148 + // Handle response
149 + if (http_code != HTTP_RESP_OK) {
150 + buffer_flush(mcpc->error);
151 +
152 + switch (http_code) {
153 + case HTTP_RESP_BAD_REQUEST:
154 + buffer_sprintf(mcpc->error, "Invalid request parameters");
155 + return MCP_RC_BAD_REQUEST;
156 +
157 + case HTTP_RESP_NOT_FOUND:
158 + buffer_sprintf(mcpc->error, "No results found");
159 + return MCP_RC_NOT_FOUND;
160 +
161 + case HTTP_RESP_GATEWAY_TIMEOUT:
162 + buffer_sprintf(mcpc->error, "Request timed out");
163 + return MCP_RC_ERROR;
164 +
165 + default:
166 + buffer_sprintf(mcpc->error, "Internal error (HTTP %d)", http_code);
167 + return MCP_RC_INTERNAL_ERROR;
168 + }
169 + }
170 +
171 + // Initialize response
172 + mcp_init_success_result(mcpc, id);
173 +
174 + // Wrap the response in MCP JSON-RPC format
175 + buffer_json_member_add_array(mcpc->result, "content");
176 + {
177 + buffer_json_add_array_item_object(mcpc->result);
178 + {
179 + buffer_json_member_add_string(mcpc->result, "type", "text");
180 + buffer_json_member_add_string(mcpc->result, "text", buffer_tostring(tmp_buffer));
181 + }
182 + buffer_json_object_close(mcpc->result);
183 + }
184 + buffer_json_array_close(mcpc->result);
185 +
186 + // Close the result object and finalize JSON
187 + buffer_json_object_close(mcpc->result); // Close the "result" object
188 + buffer_json_finalize(mcpc->result);
189 +
190 + return MCP_RC_OK;
191 +}
192 +
193 +// Schema helper for common time window parameters
194 +static void add_weights_time_parameters(BUFFER *buffer, bool include_baseline, bool required) {
195 +
196 + // add 'after' and 'before' parameters
197 + mcp_schema_add_time_params(buffer, "metrics", required);
198 +
199 + if (include_baseline) {
200 + mcp_schema_add_time_param(
201 + buffer, "baseline_after",
202 + "Baseline start time",
203 + "Start time for the baseline period to compare against. If not specified, "
204 + "automatically set to 4x the query window before the query period.",
205 + "'baseline_before'",
206 + 0,
207 + required);
208 +
209 + mcp_schema_add_time_param(
210 + buffer, "baseline_before",
211 + "Baseline end time",
212 + "End time for the baseline period. If not specified, automatically set to "
213 + "the start of the query period (adjacent to 'after').",
214 + "'after'",
215 + 0,
216 + required);
217 + }
218 +}
219 +
220 +// Schema helper for common filter parameters
221 +static void add_weights_filter_parameters(BUFFER *buffer) {
222 + mcp_schema_add_array_param(
223 + buffer, "metrics",
224 + "Filter by metrics",
225 + "Array of metrics (contexts) to filter (e.g., ['system.cpu', 'disk.io', 'mysql.queries']). Use '" MCP_TOOL_LIST_METRICS "' to discover available metrics.");
226 +
227 + mcp_schema_add_array_param(
228 + buffer, "nodes",
229 + "Filter by nodes",
230 + "Array of nodes to filter (e.g., ['web-server-1', 'database-primary']). Use '" MCP_TOOL_LIST_NODES "' to discover available nodes.");
231 +
232 + mcp_schema_add_array_param(
233 + buffer, "instances",
234 + "Filter by instances",
235 + "Array of metric instances to filter (e.g., ['eth0', 'sda', 'production_db']). Use '" MCP_TOOL_GET_METRICS_DETAILS "' to discover instances for a metric.");
236 +
237 + mcp_schema_add_array_param(buffer, "dimensions",
238 + "Filter by dimensions",
239 + "Array of dimension names to filter (e.g., ['user', 'writes', 'slow_queries']). Use '" MCP_TOOL_GET_METRICS_DETAILS "' to discover dimensions for a metric.");
240 +
241 + mcp_schema_add_labels_object(buffer,
242 + "Filter by labels",
243 + "Filter using labels where each key maps to an array of exact values. "
244 + "Values in the same array are ORed, different keys are ANDed. "
245 + "Example: {\"disk_type\": [\"ssd\", \"nvme\"], \"mount_point\": [\"/\"]}\n"
246 + "Note: Wildcards are not supported. Use exact label keys and values only. "
247 + "Use '" MCP_TOOL_GET_METRICS_DETAILS "' to discover available labels.");
248 +}
249 +
250 +// find_correlated_metrics implementation
251 +MCP_RETURN_CODE mcp_tool_find_correlated_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
252 + // Parse method parameter
253 + WEIGHTS_METHOD method = WEIGHTS_METHOD_MC_VOLUME; // Default to volume as per schema
254 +
255 + struct json_object *obj;
256 + if (json_object_object_get_ex(params, "method", &obj) && json_object_is_type(obj, json_type_string)) {
257 + const char *method_str = json_object_get_string(obj);
258 + if (strcmp(method_str, "ks2") == 0)
259 + method = WEIGHTS_METHOD_MC_KS2;
260 + else if (strcmp(method_str, "volume") == 0)
261 + method = WEIGHTS_METHOD_MC_VOLUME;
262 + }
263 +
264 + return execute_weights_request(mcpc, params, id, method, NULL);
265 +}
266 +
267 +void mcp_tool_find_correlated_metrics_schema(BUFFER *buffer) {
268 + buffer_json_member_add_object(buffer, "inputSchema");
269 + buffer_json_member_add_string(buffer, "type", "object");
270 + buffer_json_member_add_string(buffer, "title", "Find metrics that changed during an incident");
271 +
272 + buffer_json_member_add_object(buffer, "properties");
273 +
274 + add_weights_time_parameters(buffer, true, true); // include_baseline=true, required=true
275 + add_weights_filter_parameters(buffer);
276 +
277 + buffer_json_member_add_object(buffer, "method");
278 + buffer_json_member_add_string(buffer, "type", "string");
279 + buffer_json_member_add_string(buffer, "title", "Correlation method");
280 + buffer_json_member_add_string(buffer, "description",
281 + "Algorithm to use:\n"
282 + "- 'ks2': Statistical distribution comparison (slow, but intelligent)\n"
283 + "- 'volume': Percentage change in averages (fast, works well for most cases)");
284 + buffer_json_member_add_array(buffer, "enum");
285 + buffer_json_add_array_item_string(buffer, "ks2");
286 + buffer_json_add_array_item_string(buffer, "volume");
287 + buffer_json_array_close(buffer);
288 + buffer_json_member_add_string(buffer, "default", "volume");
289 + buffer_json_object_close(buffer); // method
290 +
291 + mcp_schema_add_cardinality_limit(
292 + buffer,
293 + "Maximum number of results to return",
294 + MCP_WEIGHTS_CARDINALITY_LIMIT,
295 + 30, // minimum for weights
296 + MAX(MCP_WEIGHTS_CARDINALITY_LIMIT, MCP_WEIGHTS_CARDINALITY_LIMIT_MAX));
297 +
298 + // Timeout parameter
299 + mcp_schema_add_timeout(buffer, "timeout",
300 + "Query timeout",
301 + "Maximum time to wait for the query to complete (in seconds)",
302 + 120, 1, 3600, false);
303 +
304 + buffer_json_object_close(buffer); // properties
305 +
306 + buffer_json_member_add_array(buffer, "required");
307 + buffer_json_add_array_item_string(buffer, "after");
308 + buffer_json_add_array_item_string(buffer, "before");
309 + buffer_json_array_close(buffer);
310 +
311 + buffer_json_object_close(buffer); // inputSchema
312 +}
313 +
314 +// find_anomalous_metrics implementation
315 +MCP_RETURN_CODE mcp_tool_find_anomalous_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
316 + return execute_weights_request(mcpc, params, id, WEIGHTS_METHOD_ANOMALY_RATE, NULL);
317 +}
318 +
319 +void mcp_tool_find_anomalous_metrics_schema(BUFFER *buffer) {
320 + buffer_json_member_add_object(buffer, "inputSchema");
321 + buffer_json_member_add_string(buffer, "type", "object");
322 + buffer_json_member_add_string(buffer, "title", "Find metrics with highest anomaly rates");
323 +
324 + buffer_json_member_add_object(buffer, "properties");
325 +
326 + add_weights_time_parameters(buffer, false, true); // include_baseline=false, required=true
327 + add_weights_filter_parameters(buffer);
328 +
329 + mcp_schema_add_cardinality_limit(
330 + buffer,
331 + "Maximum number of results to return",
332 + MCP_WEIGHTS_CARDINALITY_LIMIT,
333 + 30, // minimum for weights
334 + MAX(MCP_WEIGHTS_CARDINALITY_LIMIT, MCP_WEIGHTS_CARDINALITY_LIMIT_MAX));
335 +
336 + // Timeout parameter
337 + mcp_schema_add_timeout(buffer, "timeout",
338 + "Query timeout",
339 + "Maximum time to wait for the query to complete (in seconds)",
340 + 120, 1, 3600, false);
341 +
342 + buffer_json_object_close(buffer); // properties
343 +
344 + buffer_json_member_add_array(buffer, "required");
345 + buffer_json_add_array_item_string(buffer, "after");
346 + buffer_json_add_array_item_string(buffer, "before");
347 + buffer_json_array_close(buffer);
348 +
349 + buffer_json_object_close(buffer); // inputSchema
350 +}
351 +
352 +// find_unstable_metrics implementation
353 +MCP_RETURN_CODE mcp_tool_find_unstable_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
354 + // Use coefficient of variation for finding unstable metrics
355 + return execute_weights_request(mcpc, params, id, WEIGHTS_METHOD_VALUE, "cv");
356 +}
357 +
358 +void mcp_tool_find_unstable_metrics_schema(BUFFER *buffer) {
359 + buffer_json_member_add_object(buffer, "inputSchema");
360 + buffer_json_member_add_string(buffer, "type", "object");
361 + buffer_json_member_add_string(buffer, "title", "Find metrics with high variability");
362 +
363 + buffer_json_member_add_object(buffer, "properties");
364 +
365 + add_weights_time_parameters(buffer, false, true); // include_baseline=false, required=true
366 + add_weights_filter_parameters(buffer);
367 +
368 + mcp_schema_add_cardinality_limit(
369 + buffer,
370 + "Maximum number of results to return",
371 + MCP_WEIGHTS_CARDINALITY_LIMIT,
372 + 30, // minimum for weights
373 + MAX(MCP_WEIGHTS_CARDINALITY_LIMIT, MCP_WEIGHTS_CARDINALITY_LIMIT_MAX));
374 +
375 + // Timeout parameter
376 + mcp_schema_add_timeout(buffer, "timeout",
377 + "Query timeout",
378 + "Maximum time to wait for the query to complete (in seconds)",
379 + 120, 1, 3600, false);
380 +
381 + buffer_json_object_close(buffer); // properties
382 +
383 + buffer_json_member_add_array(buffer, "required");
384 + buffer_json_add_array_item_string(buffer, "after");
385 + buffer_json_add_array_item_string(buffer, "before");
386 + buffer_json_array_close(buffer);
387 +
388 + buffer_json_object_close(buffer); // inputSchema
389 +}
src/web/mcp/mcp-tools-weights.h new
+20
@@ -0,0 +1,20 @@
1 +// SPDX-License-Identifier: GPL-3.0-or-later
2 +
3 +#ifndef NETDATA_MCP_TOOLS_WEIGHTS_H
4 +#define NETDATA_MCP_TOOLS_WEIGHTS_H
5 +
6 +#include "mcp.h"
7 +
8 +// Execute the find_correlated_metrics tool
9 +MCP_RETURN_CODE mcp_tool_find_correlated_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
10 +void mcp_tool_find_correlated_metrics_schema(BUFFER *buffer);
11 +
12 +// Execute the find_anomalous_metrics tool
13 +MCP_RETURN_CODE mcp_tool_find_anomalous_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
14 +void mcp_tool_find_anomalous_metrics_schema(BUFFER *buffer);
15 +
16 +// Execute the find_unstable_metrics tool
17 +MCP_RETURN_CODE mcp_tool_find_unstable_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
18 +void mcp_tool_find_unstable_metrics_schema(BUFFER *buffer);
19 +
20 +#endif // NETDATA_MCP_TOOLS_WEIGHTS_H
\ No newline at end of file
src/web/mcp/mcp-tools.c
+311 -149
@@ -9,21 +9,16 @@
9 * Tools are model-controlled actions - meaning the AI decides when and how to use them based on context.
10 * Each tool has a defined input schema that specifies required and optional parameters.
11 *
12 - * Key features of the tools namespace:
12 + * Standard methods in the MCP specification:
13 *
14 - * 1. Tool Discovery:
15 - * - Clients can list available tools (tools/list)
16 - * - Get detailed descriptions of specific tools (tools/describe)
17 - * - Understand what parameters a tool requires (through JSON Schema)
14 + * 1. tools/list - Lists available tools with their schemas and metadata
15 + * - Returns a list of all available tools
16 + * - Includes each tool's name, description, and input schema
17 *
19 - * 2. Tool Execution:
20 - * - Execute tools with specific parameters (tools/execute)
21 - * - Validate parameters without execution (tools/validate)
22 - * - Asynchronous execution is supported for long-running tools
23 - *
24 - * 3. Execution Management:
25 - * - Check execution status (tools/status)
26 - * - Cancel running executions (tools/cancel)
18 + * 2. tools/call - Executes a specific tool with provided parameters
19 + * - Takes a tool name and arguments
20 + * - Returns the result of the tool execution
21 + * - May include resource references, text, images, or other content types
22 *
23 * In the Netdata context, tools provide access to operations like:
24 * - Exploring metrics and their relationships
@@ -31,152 +26,332 @@
26 * - Finding correlations between metrics
27 * - Root cause analysis for anomalies
28 * - Summarizing system health
34 - *
35 - * Each tool execution is assigned a unique ID, allowing clients to track and manage executions.
29 */
30
31 #include "mcp-tools.h"
39 -#include "mcp-initialize.h"
32
41 -// Return a list of available tools (transport-agnostic)
42 -static MCP_RETURN_CODE mcp_tools_method_list(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
43 - if (!mcpc || id == 0) return MCP_RC_ERROR;
33 +// Include tool-specific header files
34 +#include "mcp-tools-list-metadata.h"
35 +#include "mcp-tools-execute-function.h"
36 +#include "mcp-tools-query-metrics.h"
37 +#include "mcp-tools-weights.h"
38 +#include "mcp-tools-alert-transitions.h"
39 +#include "mcp-tools-configured-alerts.h"
40
45 - // Initialize success response
46 - mcp_init_success_result(mcpc, id);
47 -
48 - // Create tools array
49 - buffer_json_member_add_array(mcpc->result, "tools");
50 -
51 - // Add explore_metrics tool
52 - buffer_json_add_array_item_object(mcpc->result);
53 - buffer_json_member_add_string(mcpc->result, "name", "explore_metrics");
54 - buffer_json_member_add_string(mcpc->result, "description",
55 - "Explore Netdata's time-series metrics with support for high-resolution data");
56 -
57 - // Add input schema for metrics tool
58 - buffer_json_member_add_object(mcpc->result, "inputSchema");
59 - buffer_json_member_add_string(mcpc->result, "type", "object");
60 - buffer_json_member_add_string(mcpc->result, "title", "MetricsQuery");
61 -
62 - // Properties
63 - buffer_json_member_add_object(mcpc->result, "properties");
64 -
65 - // Context property
66 - buffer_json_member_add_object(mcpc->result, "context");
67 - buffer_json_member_add_string(mcpc->result, "type", "string");
68 - buffer_json_member_add_string(mcpc->result, "title", "Context");
69 - buffer_json_object_close(mcpc->result); // Close context
70 -
71 - // After property
72 - buffer_json_member_add_object(mcpc->result, "after");
73 - buffer_json_member_add_string(mcpc->result, "type", "integer");
74 - buffer_json_member_add_string(mcpc->result, "title", "After");
75 - buffer_json_object_close(mcpc->result); // Close after
76 -
77 - // Before property
78 - buffer_json_member_add_object(mcpc->result, "before");
79 - buffer_json_member_add_string(mcpc->result, "type", "integer");
80 - buffer_json_member_add_string(mcpc->result, "title", "Before");
81 - buffer_json_object_close(mcpc->result); // Close before
82 -
83 - // Points property
84 - buffer_json_member_add_object(mcpc->result, "points");
85 - buffer_json_member_add_string(mcpc->result, "type", "integer");
86 - buffer_json_member_add_string(mcpc->result, "title", "Points");
87 - buffer_json_object_close(mcpc->result); // Close points
88 -
89 - // Group property
90 - buffer_json_member_add_object(mcpc->result, "group");
91 - buffer_json_member_add_string(mcpc->result, "type", "string");
92 - buffer_json_member_add_string(mcpc->result, "title", "Group");
93 - buffer_json_object_close(mcpc->result); // Close group
94 -
95 - buffer_json_object_close(mcpc->result); // Close properties
96 -
97 - // Required properties
98 - buffer_json_member_add_array(mcpc->result, "required");
99 - buffer_json_add_array_item_string(mcpc->result, "context");
100 - buffer_json_array_close(mcpc->result); // Close required
101 -
102 - buffer_json_object_close(mcpc->result); // Close inputSchema
103 - buffer_json_object_close(mcpc->result); // Close explore_metrics tool
104 -
105 - // Add explore_nodes tool
106 - buffer_json_add_array_item_object(mcpc->result);
107 - buffer_json_member_add_string(mcpc->result, "name", "explore_nodes");
108 - buffer_json_member_add_string(mcpc->result, "description",
109 - "Discover and explore all monitored nodes in your infrastructure");
110 -
111 - // Add input schema for nodes tool
112 - buffer_json_member_add_object(mcpc->result, "inputSchema");
113 - buffer_json_member_add_string(mcpc->result, "type", "object");
114 - buffer_json_member_add_string(mcpc->result, "title", "NodesQuery");
115 -
116 - // Properties
117 - buffer_json_member_add_object(mcpc->result, "properties");
118 -
119 - // Filter property
120 - buffer_json_member_add_object(mcpc->result, "filter");
121 - buffer_json_member_add_string(mcpc->result, "type", "string");
122 - buffer_json_member_add_string(mcpc->result, "title", "Filter");
123 - buffer_json_object_close(mcpc->result); // Close filter
124 -
125 - buffer_json_object_close(mcpc->result); // Close properties
126 - buffer_json_object_close(mcpc->result); // Close inputSchema
127 - buffer_json_object_close(mcpc->result); // Close explore_nodes tool
128 -
129 - buffer_json_array_close(mcpc->result); // Close tools array
130 - buffer_json_finalize(mcpc->result); // Finalize the JSON
41 +// Tool handler function prototypes
42 +typedef MCP_RETURN_CODE (*mcp_tool_execute_t)(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id);
43 +typedef void (*mcp_tool_schema_t)(BUFFER *schema_buffer);
44 +
45 +// Tool definition structure
46 +typedef struct {
47 + const char *name; // Tool name
48 + const char *description; // Tool description
49 + mcp_tool_execute_t execute_callback; // Tool execution callback
50 + mcp_tool_schema_t schema_callback; // Tool schema definition callback
51
132 - return MCP_RC_OK;
52 + // UI/UX annotations
53 + const char *title; // Human-readable title
54 + bool read_only_hint; // If true, the tool doesn't modify a state
55 + bool open_world_hint; // If true, the tool interacts with the external world
56 +} MCP_TOOL_DEF;
57 +
58 +// Wrapper functions for unified list tools
59 +static MCP_RETURN_CODE unified_list_tool_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id, const char *tool_name) {
60 + const MCP_LIST_TOOL_CONFIG *config = mcp_get_list_tool_config(tool_name);
61 + if (!config) {
62 + buffer_sprintf(mcpc->error, "Unknown list tool: %s", tool_name);
63 + return MCP_RC_INTERNAL_ERROR;
64 + }
65 + return mcp_unified_list_tool_execute(mcpc, config, params, id);
66 }
67
135 -// Stub implementations for other tools methods (transport-agnostic)
136 -static MCP_RETURN_CODE mcp_tools_method_execute(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
137 - buffer_sprintf(mcpc->error, "Method 'tools/execute' not implemented yet");
138 - return MCP_RC_NOT_IMPLEMENTED;
68 +static void unified_list_tool_schema(BUFFER *buffer, const char *tool_name) {
69 + const MCP_LIST_TOOL_CONFIG *config = mcp_get_list_tool_config(tool_name);
70 + if (!config) return;
71 + mcp_unified_list_tool_schema(buffer, config);
72 }
73
141 -static MCP_RETURN_CODE mcp_tools_method_cancel(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
142 - buffer_sprintf(mcpc->error, "Method 'tools/cancel' not implemented yet");
143 - return MCP_RC_NOT_IMPLEMENTED;
74 +// Specific wrappers for each list tool
75 +static MCP_RETURN_CODE list_metrics_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
76 + return unified_list_tool_execute(mcpc, params, id, MCP_TOOL_LIST_METRICS);
77 +}
78 +static void list_metrics_schema(BUFFER *buffer) {
79 + unified_list_tool_schema(buffer, MCP_TOOL_LIST_METRICS);
80 }
81
146 -static MCP_RETURN_CODE mcp_tools_method_status(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
147 - buffer_sprintf(mcpc->error, "Method 'tools/status' not implemented yet");
148 - return MCP_RC_NOT_IMPLEMENTED;
82 +static MCP_RETURN_CODE get_metrics_details_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
83 + return unified_list_tool_execute(mcpc, params, id, MCP_TOOL_GET_METRICS_DETAILS);
84 +}
85 +static void get_metrics_details_schema(BUFFER *buffer) {
86 + unified_list_tool_schema(buffer, MCP_TOOL_GET_METRICS_DETAILS);
87 +}
88 +
89 +static MCP_RETURN_CODE list_nodes_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
90 + return unified_list_tool_execute(mcpc, params, id, MCP_TOOL_LIST_NODES);
91 +}
92 +static void list_nodes_schema(BUFFER *buffer) {
93 + unified_list_tool_schema(buffer, MCP_TOOL_LIST_NODES);
94 +}
95 +
96 +static MCP_RETURN_CODE list_functions_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
97 + return unified_list_tool_execute(mcpc, params, id, MCP_TOOL_LIST_FUNCTIONS);
98 +}
99 +static void list_functions_schema(BUFFER *buffer) {
100 + unified_list_tool_schema(buffer, MCP_TOOL_LIST_FUNCTIONS);
101 }
102
151 -static MCP_RETURN_CODE mcp_tools_method_validate(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
152 - buffer_sprintf(mcpc->error, "Method 'tools/validate' not implemented yet");
153 - return MCP_RC_NOT_IMPLEMENTED;
103 +static MCP_RETURN_CODE get_nodes_details_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
104 + return unified_list_tool_execute(mcpc, params, id, MCP_TOOL_GET_NODES_DETAILS);
105 +}
106 +static void get_nodes_details_schema(BUFFER *buffer) {
107 + unified_list_tool_schema(buffer, MCP_TOOL_GET_NODES_DETAILS);
108 +}
109 +
110 +static MCP_RETURN_CODE list_raised_alerts_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
111 + return unified_list_tool_execute(mcpc, params, id, MCP_TOOL_LIST_RAISED_ALERTS);
112 +}
113 +static void list_raised_alerts_schema(BUFFER *buffer) {
114 + unified_list_tool_schema(buffer, MCP_TOOL_LIST_RAISED_ALERTS);
115 }
116
156 -static MCP_RETURN_CODE mcp_tools_method_describe(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id __maybe_unused) {
157 - buffer_sprintf(mcpc->error, "Method 'tools/describe' not implemented yet");
158 - return MCP_RC_NOT_IMPLEMENTED;
117 +static MCP_RETURN_CODE list_all_alerts_execute(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
118 + return unified_list_tool_execute(mcpc, params, id, MCP_TOOL_LIST_ALL_ALERTS);
119 }
120 +static void list_all_alerts_schema(BUFFER *buffer) {
121 + unified_list_tool_schema(buffer, MCP_TOOL_LIST_ALL_ALERTS);
122 +}
123 +
124 +
125 +// Static array of tool definitions
126 +static const MCP_TOOL_DEF mcp_tools[] = {
127 + // List tools (using unified implementation)
128 + {
129 + .name = MCP_TOOL_LIST_METRICS,
130 + .title = "Discover available metrics",
131 + .description = "Lists available metrics (contexts) with time-aware filtering. Returns metric names matching search patterns, filtered by nodes and time window. Supports full-text search across names, titles, instances, dimensions, and labels.\n",
132 + .execute_callback = list_metrics_execute,
133 + .schema_callback = list_metrics_schema,
134 + .read_only_hint = true,
135 + .open_world_hint = false
136 + },
137 + {
138 + .name = MCP_TOOL_GET_METRICS_DETAILS,
139 + .title = "Get detailed information about specific metrics",
140 + .description = "Gets comprehensive metadata for specific metrics. Returns titles, units, dimensions, instances, labels, and collection status.\n",
141 + .execute_callback = get_metrics_details_execute,
142 + .schema_callback = get_metrics_details_schema,
143 + .read_only_hint = true,
144 + .open_world_hint = false
145 + },
146 + {
147 + .name = MCP_TOOL_LIST_NODES,
148 + .title = "List all monitored nodes in the Netdata ecosystem",
149 + .description = "Lists all monitored nodes in the infrastructure. Returns node IDs, hostnames, connection status, and parent-child relationships. Use this to discover available nodes before querying metrics or executing functions.\n",
150 + .execute_callback = list_nodes_execute,
151 + .schema_callback = list_nodes_schema,
152 + .read_only_hint = true,
153 + .open_world_hint = false
154 + },
155 + {
156 + .name = MCP_TOOL_LIST_FUNCTIONS,
157 + .title = "List available functions",
158 + .description = "Lists all available Netdata functions that can be executed on nodes. Returns function names, descriptions, and execution requirements. Use this to discover what functions are available before executing them.\n",
159 + .execute_callback = list_functions_execute,
160 + .schema_callback = list_functions_schema,
161 + .read_only_hint = true,
162 + .open_world_hint = false
163 + },
164 + {
165 + .name = MCP_TOOL_GET_NODES_DETAILS,
166 + .title = "Get detailed information about monitored nodes",
167 + .description = "Gets comprehensive node information including hardware specs, OS details, capabilities, health status, available functions, and monitoring configuration. Essential for understanding node capabilities before executing functions.\n",
168 + .execute_callback = get_nodes_details_execute,
169 + .schema_callback = get_nodes_details_schema,
170 + .read_only_hint = true,
171 + .open_world_hint = false
172 + },
173 +
174 + // Non-list tools (keep their original implementation)
175 + {
176 + .name = MCP_TOOL_EXECUTE_FUNCTION,
177 + .title = "Execute a function on a specific node",
178 + .description = "Executes live data collection functions on nodes. Common functions: 'processes' (running processes), 'network-connections' (active connections), 'mount-points' (disk mounts), 'systemd-services' (service status). Returns tabular data with filtering and sorting options.\n",
179 + .execute_callback = mcp_tool_execute_function_execute,
180 + .schema_callback = mcp_tool_execute_function_schema,
181 + .read_only_hint = true, // Currently read-only (will change when dynamic config is added)
182 + .open_world_hint = true // Routes requests to remote nodes in the Netdata ecosystem
183 + },
184 + {
185 + .name = MCP_TOOL_QUERY_METRICS,
186 + .title = "Query metrics data",
187 + .description = "Queries time-series metrics data with powerful aggregation options. Specify context, time range, and grouping (by dimension, instance, node, or label). Returns data points with statistics and contribution analysis.\n",
188 + .execute_callback = mcp_tool_query_metrics_execute,
189 + .schema_callback = mcp_tool_query_metrics_schema,
190 + .read_only_hint = true,
191 + .open_world_hint = false
192 + },
193 +
194 + // Weights/correlation tools
195 + {
196 + .name = MCP_TOOL_FIND_CORRELATED_METRICS,
197 + .title = "Find metrics that changed during an incident",
198 + .description = "Finds metrics that changed significantly during an incident by comparing a problem time period with a normal baseline period. Essential for root cause analysis. IMPORTANT: For large infrastructures, use filters (metrics, nodes, instances, dimensions, or labels) to narrow the scope and avoid timeouts.",
199 + .execute_callback = mcp_tool_find_correlated_metrics_execute,
200 + .schema_callback = mcp_tool_find_correlated_metrics_schema,
201 + .read_only_hint = true,
202 + .open_world_hint = false
203 + },
204 + {
205 + .name = MCP_TOOL_FIND_ANOMALOUS_METRICS,
206 + .title = "Find metrics with highest anomaly rates",
207 + .description = "Finds metrics that were behaving anomalously according to Netdata's ML models. Returns metrics ranked by their anomaly rates (0 to 1, representing 0-100% of time anomalous). IMPORTANT: For large infrastructures, use filters (metrics, nodes, instances, dimensions, or labels) to narrow the scope and avoid timeouts.",
208 + .execute_callback = mcp_tool_find_anomalous_metrics_execute,
209 + .schema_callback = mcp_tool_find_anomalous_metrics_schema,
210 + .read_only_hint = true,
211 + .open_world_hint = false
212 + },
213 + {
214 + .name = MCP_TOOL_FIND_UNSTABLE_METRICS,
215 + .title = "Find metrics with high variability",
216 + .description = "Finds metrics with the highest variability using coefficient of variation (standard deviation as % of mean). Useful for identifying unstable or fluctuating metrics. IMPORTANT: For large infrastructures, use filters (metrics, nodes, instances, dimensions, or labels) to narrow the scope and avoid timeouts.",
217 + .execute_callback = mcp_tool_find_unstable_metrics_execute,
218 + .schema_callback = mcp_tool_find_unstable_metrics_schema,
219 + .read_only_hint = true,
220 + .open_world_hint = false
221 + },
222 +
223 + // Alert tools
224 + {
225 + .name = MCP_TOOL_LIST_RAISED_ALERTS,
226 + .title = "List raised alerts",
227 + .description = "List currently active alerts (WARNING and CRITICAL status) across all nodes",
228 + .execute_callback = list_raised_alerts_execute,
229 + .schema_callback = list_raised_alerts_schema,
230 + .read_only_hint = true,
231 + .open_world_hint = false
232 + },
233 + {
234 + .name = MCP_TOOL_LIST_ALL_ALERTS,
235 + .title = "List all alerts",
236 + .description = "List all alerts including cleared, undefined, and uninitialized alerts across all nodes",
237 + .execute_callback = list_all_alerts_execute,
238 + .schema_callback = list_all_alerts_schema,
239 + .read_only_hint = true,
240 + .open_world_hint = false
241 + },
242 + {
243 + .name = MCP_TOOL_LIST_ALERT_TRANSITIONS,
244 + .title = "List alert transitions",
245 + .description = "List recent alert state transitions showing how alerts changed over time",
246 + .execute_callback = mcp_tool_list_alert_transitions_execute,
247 + .schema_callback = mcp_tool_list_alert_transitions_schema,
248 + .read_only_hint = true,
249 + .open_world_hint = false
250 + },
251 +
252 + // Commented for the moment - probably dyncfg is a better way to do this
253 +// {
254 +// .name = MCP_TOOL_LIST_CONFIGURED_ALERTS,
255 +// .title = "List configured alert prototypes",
256 +// .description = "Lists all configured alert prototypes (templates) that define how alerts are created. Shows matching criteria, conditions, thresholds, and actions. Useful for understanding what Netdata is configured to monitor.",
257 +// .execute_callback = mcp_tool_list_configured_alerts_execute,
258 +// .schema_callback = mcp_tool_list_configured_alerts_schema,
259 +// .read_only_hint = true,
260 +// .open_world_hint = false
261 +// },
262
161 -static MCP_RETURN_CODE mcp_tools_method_getCapabilities(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, uint64_t id) {
162 - if (!mcpc || id == 0) return MCP_RC_ERROR;
263 + // Add more tools here
264
265 + // Terminator
266 + {
267 + .name = NULL
268 + }
269 +};
270 +
271 +// Return a list of available tools
272 +static MCP_RETURN_CODE mcp_tools_method_list(MCP_CLIENT *mcpc, struct json_object *params __maybe_unused, MCP_REQUEST_ID id) {
273 + if (!mcpc || id == 0)
274 + return MCP_RC_ERROR;
275 +
276 // Initialize success response
277 mcp_init_success_result(mcpc, id);
278
167 - // Add capabilities as result object properties
168 - buffer_json_member_add_boolean(mcpc->result, "listChanged", false);
169 - buffer_json_member_add_boolean(mcpc->result, "asyncExecution", true);
170 - buffer_json_member_add_boolean(mcpc->result, "batchExecution", true);
279 + // Create tool-array
280 + buffer_json_member_add_array(mcpc->result, "tools");
281
172 - // Close the result object
173 - buffer_json_finalize(mcpc->result);
282 + // Iterate through all defined tools and add them to the response
283 + for (size_t i = 0; mcp_tools[i].name != NULL; i++) {
284 + const MCP_TOOL_DEF *tool = &mcp_tools[i];
285 +
286 + // Add a tool object
287 + buffer_json_add_array_item_object(mcpc->result);
288 +
289 + // Add basic properties
290 + buffer_json_member_add_string(mcpc->result, "name", tool->name);
291 + buffer_json_member_add_string(mcpc->result, "description", tool->description);
292 +
293 + // Add schema using the tool's schema callback
294 + tool->schema_callback(mcpc->result);
295 +
296 + // Add annotations if available
297 + buffer_json_member_add_object(mcpc->result, "annotations");
298 + if (tool->title) {
299 + buffer_json_member_add_string(mcpc->result, "title", tool->title);
300 + }
301 + buffer_json_member_add_boolean(mcpc->result, "readOnlyHint", tool->read_only_hint);
302 + buffer_json_member_add_boolean(mcpc->result, "openWorldHint", tool->open_world_hint);
303 + buffer_json_object_close(mcpc->result); // Close annotations
304 +
305 + buffer_json_object_close(mcpc->result); // Close tool object
306 + }
307 +
308 + buffer_json_array_close(mcpc->result); // Close tools array
309 + buffer_json_finalize(mcpc->result); // Finalize the JSON
310
311 return MCP_RC_OK;
312 }
313
314 +// Main execute method that routes to specific tool handlers
315 +static MCP_RETURN_CODE mcp_tools_method_call(MCP_CLIENT *mcpc, struct json_object *params, MCP_REQUEST_ID id) {
316 + if (!mcpc || !params || id == 0)
317 + return MCP_RC_ERROR;
318 +
319 + // Extract tool name
320 + struct json_object *name_obj = NULL;
321 + if (!json_object_object_get_ex(params, "name", &name_obj)) {
322 + buffer_sprintf(mcpc->error, "Missing required parameter 'name'");
323 + return MCP_RC_BAD_REQUEST;
324 + }
325 +
326 + if (!json_object_is_type(name_obj, json_type_string)) {
327 + buffer_sprintf(mcpc->error, "Parameter 'name' must be a string");
328 + return MCP_RC_BAD_REQUEST;
329 + }
330 +
331 + const char *tool_name = json_object_get_string(name_obj);
332 +
333 + // Get arguments if present
334 + struct json_object *args_obj = NULL;
335 + json_object_object_get_ex(params, "arguments", &args_obj);
336 +
337 + // Search for the tool in our static array
338 + for (size_t i = 0; mcp_tools[i].name != NULL; i++) {
339 + if (strcmp(tool_name, mcp_tools[i].name) == 0) {
340 + // Found the tool, execute its callback
341 + return mcp_tools[i].execute_callback(mcpc, args_obj, id);
342 + }
343 + }
344 +
345 + // Tool not found
346 + buffer_sprintf(mcpc->error, "Unknown tool: %s", tool_name);
347 + return MCP_RC_BAD_REQUEST;
348 +}
349 +
350 +// The MCP specification only defines list and call methods for tools
351 +// Other methods are not part of the standard specification
352 +
353 // Tools namespace method dispatcher (transport-agnostic)
179 -MCP_RETURN_CODE mcp_tools_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id) {
354 +MCP_RETURN_CODE mcp_tools_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id) {
355 if (!mcpc || !method) return MCP_RC_INTERNAL_ERROR;
356
357 netdata_log_debug(D_MCP, "MCP tools method: %s", method);
@@ -188,29 +363,16 @@ MCP_RETURN_CODE mcp_tools_route(MCP_CLIENT *mcpc, const char *method, struct jso
363 MCP_RETURN_CODE rc;
364
365 if (strcmp(method, "list") == 0) {
366 + // List available tools - standard method in MCP specification
367 rc = mcp_tools_method_list(mcpc, params, id);
368 }
193 - else if (strcmp(method, "execute") == 0) {
194 - rc = mcp_tools_method_execute(mcpc, params, id);
195 - }
196 - else if (strcmp(method, "cancel") == 0) {
197 - rc = mcp_tools_method_cancel(mcpc, params, id);
198 - }
199 - else if (strcmp(method, "status") == 0) {
200 - rc = mcp_tools_method_status(mcpc, params, id);
201 - }
202 - else if (strcmp(method, "validate") == 0) {
203 - rc = mcp_tools_method_validate(mcpc, params, id);
204 - }
205 - else if (strcmp(method, "describe") == 0) {
206 - rc = mcp_tools_method_describe(mcpc, params, id);
207 - }
208 - else if (strcmp(method, "getCapabilities") == 0) {
209 - rc = mcp_tools_method_getCapabilities(mcpc, params, id);
369 + else if (strcmp(method, "call") == 0) {
370 + // Execute a tool; standard method in MCP specification
371 + rc = mcp_tools_method_call(mcpc, params, id);
372 }
373 else {
374 // Method not found in tools namespace
213 - buffer_sprintf(mcpc->error, "Method 'tools/%s' not implemented yet", method);
375 + buffer_sprintf(mcpc->error, "Method 'tools/%s' not supported. The MCP specification only defines 'list' and 'call' methods.", method);
376 rc = MCP_RC_NOT_IMPLEMENTED;
377 }
378
src/web/mcp/mcp-tools.h
+2 -1
@@ -6,6 +6,7 @@
6 #include "mcp.h"
7
8 // Tools namespace method dispatcher (transport-agnostic)
9 -MCP_RETURN_CODE mcp_tools_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, uint64_t id);
9 +MCP_RETURN_CODE mcp_tools_route(MCP_CLIENT *mcpc, const char *method, struct json_object *params, MCP_REQUEST_ID id);
10 +
11
12 #endif // NETDATA_MCP_TOOLS_H
\ No newline at end of file
src/web/mcp/mcp-websocket-test.html deleted
-1272
@@ -1,1272 +0,0 @@
1 -<!DOCTYPE html>
2 -<html lang="en">
3 -<head>
4 - <meta charset="UTF-8">
5 - <meta name="viewport" content="width=device-width, initial-scale=1.0">
6 - <title>Netdata MCP WebSocket Test</title>
7 - <style>
8 - body {
9 - font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
10 - margin: 0;
11 - padding: 10px;
12 - line-height: 1.6;
13 - color: #333;
14 - max-width: 100%;
15 - box-sizing: border-box;
16 - overflow: hidden;
17 - }
18 - h1 {
19 - color: #0088cc;
20 - margin-top: 0;
21 - margin-bottom: 10px;
22 - }
23 - .connection-panel {
24 - background-color: #e2f4ff;
25 - padding: 8px;
26 - border-radius: 5px;
27 - margin-bottom: 10px;
28 - display: flex;
29 - align-items: center;
30 - flex-wrap: wrap;
31 - gap: 8px;
32 - }
33 - .four-column-layout {
34 - display: grid;
35 - grid-template-columns: 195px 260px 1fr 2fr;
36 - gap: 10px;
37 - height: calc(100vh - 120px);
38 - min-height: 500px;
39 - }
40 - .column {
41 - background-color: #f5f5f5;
42 - border-radius: 5px;
43 - padding: 10px;
44 - box-shadow: 0 2px 5px rgba(0, 0, 0, 0.1);
45 - display: flex;
46 - flex-direction: column;
47 - overflow: hidden;
48 - position: relative; /* Added for absolute positioning of buttons */
49 - }
50 - .column-header {
51 - margin-top: 0;
52 - margin-bottom: 10px;
53 - padding-bottom: 8px;
54 - border-bottom: 1px solid #ddd;
55 - font-size: 1.2em;
56 - color: #0088cc;
57 - display: flex;
58 - justify-content: space-between;
59 - align-items: center;
60 - }
61 - .column-content {
62 - overflow-y: auto;
63 - flex: 1;
64 - }
65 - .flow-item, .method-item {
66 - padding: 8px;
67 - margin-bottom: 6px;
68 - border-radius: 5px;
69 - cursor: pointer;
70 - transition: background-color 0.2s;
71 - }
72 - .flow-item:hover, .method-item:hover {
73 - background-color: #e0e0e0;
74 - }
75 - .flow-item.active, .method-item.active {
76 - background-color: #d0e8f2;
77 - font-weight: bold;
78 - }
79 - #jsonEditor {
80 - width: 100%;
81 - height: 100%;
82 - font-family: monospace;
83 - font-size: 14px;
84 - padding: 8px;
85 - border: 1px solid #ccc;
86 - border-radius: 5px;
87 - resize: none;
88 - box-sizing: border-box;
89 - }
90 - /* JSON syntax highlighting */
91 - .json-string {
92 - color: #008800;
93 - font-weight: bold;
94 - display: inline-block; /* Allow multiline strings to format properly */
95 - }
96 - .json-number { color: #0055aa; font-weight: bold; }
97 - .json-boolean-true { color: #008800; font-weight: bold; }
98 - .json-boolean-false { color: #dd0000; font-weight: bold; }
99 - .json-null { color: #666666; font-weight: bold; }
100 - .json-key { color: #555555; }
101 - .json-bracket { color: #aaaaaa; }
102 - .json-comma { color: #aaaaaa; }
103 - .json-colon { color: #aaaaaa; }
104 - #requestData {
105 - background-color: #f0f0f0;
106 - padding: 10px;
107 - border-radius: 5px;
108 - margin-top: 10px;
109 - font-family: monospace;
110 - font-size: 12px;
111 - overflow: auto;
112 - white-space: pre-wrap;
113 - max-height: 100px;
114 - }
115 - button {
116 - background-color: #0088cc;
117 - color: white;
118 - border: none;
119 - padding: 8px 12px;
120 - border-radius: 5px;
121 - cursor: pointer;
122 - font-size: 14px;
123 - }
124 - button:hover {
125 - background-color: #006699;
126 - }
127 - button:disabled {
128 - background-color: #cccccc;
129 - cursor: not-allowed;
130 - }
131 - .button-row {
132 - display: flex;
133 - gap: 8px;
134 - margin-top: 8px;
135 - }
136 - .status {
137 - font-weight: bold;
138 - }
139 - .connected {
140 - color: green;
141 - }
142 - .disconnected {
143 - color: red;
144 - }
145 - .tooltip {
146 - position: relative;
147 - display: inline-block;
148 - cursor: help;
149 - }
150 - .tooltip .tooltiptext {
151 - visibility: hidden;
152 - width: 300px;
153 - background-color: #555;
154 - color: #fff;
155 - text-align: left;
156 - border-radius: 6px;
157 - padding: 10px;
158 - position: absolute;
159 - z-index: 1;
160 - bottom: 125%;
161 - left: 50%;
162 - transform: translateX(-50%);
163 - opacity: 0;
164 - transition: opacity 0.3s;
165 - font-size: 12px;
166 - line-height: 1.4;
167 - }
168 - .tooltip:hover .tooltiptext {
169 - visibility: visible;
170 - opacity: 1;
171 - }
172 -
173 - /* Ensure JSON content wraps properly */
174 - pre, #responseViewer, #requestData {
175 - white-space: pre-wrap !important;
176 - word-break: break-word !important;
177 - overflow-x: auto !important;
178 - }
179 - </style>
180 -</head>
181 -<body>
182 - <h1>Netdata MCP WebSocket Test</h1>
183 -
184 - <div class="connection-panel">
185 - <label for="serverUrl">WebSocket Server URL:</label>
186 - <input type="text" id="serverUrl" value="ws://localhost:19999/mcp" style="width: 300px;">
187 - <button id="connectBtn">Connect</button>
188 - <button id="disconnectBtn" disabled>Disconnect</button>
189 - <span id="status" class="status disconnected">Disconnected</span>
190 - </div>
191 -
192 - <div class="four-column-layout">
193 - <!-- First Column - Flows -->
194 - <div class="column">
195 - <h3 class="column-header">Flows</h3>
196 - <div class="column-content" id="flowsContainer">
197 - <div class="flow-item active" data-flow="initialize-flow">Initialize</div>
198 - <div class="flow-item" data-flow="tools-flow">Tools</div>
199 - <div class="flow-item" data-flow="resources-flow">Resources</div>
200 - <div class="flow-item" data-flow="prompts-flow">Prompts</div>
201 - <div class="flow-item" data-flow="notifications-flow">Notifications</div>
202 - <div class="flow-item" data-flow="context-flow">Context</div>
203 - <div class="flow-item" data-flow="system-flow">System</div>
204 - </div>
205 - </div>
206 -
207 - <!-- Second Column - Methods -->
208 - <div class="column">
209 - <h3 class="column-header">Methods</h3>
210 - <div class="column-content" id="methodsContainer">
211 - <!-- Methods will be loaded here -->
212 - </div>
213 - </div>
214 -
215 - <!-- Third Column - Request Editor -->
216 - <div class="column">
217 - <h3 class="column-header">
218 - Request
219 - <span class="tooltip">ⓘ
220 - <span class="tooltiptext">
221 - JSON-RPC 2.0 request format requires "jsonrpc": "2.0", "method" and an optional "id".
222 - </span>
223 - </span>
224 - </h3>
225 - <div style="display: flex; flex-direction: column; height: 100%;">
226 - <!-- 50% height for JSON editor -->
227 - <textarea id="jsonEditor" style="height: 50%; min-height: 200px; margin-bottom: 8px; resize: none;"></textarea>
228 -
229 - <!-- Log container with remaining height minus buttons -->
230 - <div style="display: flex; flex-direction: column; height: calc(50% - 60px); margin-bottom: 50px;">
231 - <div id="requestData" style="background-color: #f0f0f0; padding: 6px; border-radius: 4px; margin-bottom: 4px; font-family: monospace; font-size: 12px; white-space: pre-wrap; word-break: break-word; height: 60px; overflow: auto;"></div>
232 - <pre id="commLog" style="flex: 1; background-color: #f8f8f8; border: 1px solid #ddd; border-radius: 4px; padding: 6px; font-family: monospace; font-size: 12px; overflow: auto; margin: 0;"></pre>
233 - </div>
234 -
235 - <!-- Fixed position buttons at the bottom -->
236 - <div class="button-row" style="position: absolute; bottom: 10px; left: 10px; right: 10px; background: #f5f5f5; padding: 5px 0;">
237 - <button id="sendBtn" disabled>Send</button>
238 - <button id="resetBtn">Reset</button>
239 - <button id="clearLogBtn">Clear Log</button>
240 - </div>
241 - </div>
242 - </div>
243 -
244 - <!-- Fourth Column - Response Viewer -->
245 - <div class="column">
246 - <h3 class="column-header">Response</h3>
247 - <div style="display: flex; flex-direction: column; height: calc(100% - 35px);">
248 - <pre id="responseViewer" style="flex: 1; overflow: auto; margin: 0; padding: 6px; background-color: #f8f8f8; border: 1px solid #ddd; border-radius: 4px; white-space: pre-wrap; word-break: break-word;"><span class="json-null">No response yet.</span></pre>
249 - </div>
250 - </div>
251 - </div>
252 -
253 - <script>
254 - // DOM elements
255 - const serverUrlInput = document.getElementById('serverUrl');
256 - const connectBtn = document.getElementById('connectBtn');
257 - const disconnectBtn = document.getElementById('disconnectBtn');
258 - const statusSpan = document.getElementById('status');
259 - const jsonEditor = document.getElementById('jsonEditor');
260 - const sendBtn = document.getElementById('sendBtn');
261 - const resetBtn = document.getElementById('resetBtn');
262 - const clearLogBtn = document.getElementById('clearLogBtn');
263 - const responseViewer = document.getElementById('responseViewer');
264 - const requestData = document.getElementById('requestData');
265 - const commLog = document.getElementById('commLog');
266 - const flowsContainer = document.getElementById('flowsContainer');
267 - const methodsContainer = document.getElementById('methodsContainer');
268 -
269 - // WebSocket connection
270 - let socket = null;
271 - let requestId = 1;
272 - let isConnected = false;
273 - let requestTimestamp = null;
274 -
275 - // Function to highlight JSON syntax with colors
276 - function syntaxHighlightJson(json) {
277 - // Use more space for indentation to make it easier to read
278 - const jsonStr = JSON.stringify(json, null, 4);
279 -
280 - // A more direct approach for syntax highlighting
281 - let result = '';
282 - let inString = false;
283 - let isKey = false;
284 - let currentStr = '';
285 - let lastChar = '';
286 -
287 - // Process each character individually to maintain proper context
288 - for (let i = 0; i < jsonStr.length; i++) {
289 - const char = jsonStr[i];
290 -
291 - // Handle string delimiters
292 - if (char === '"' && lastChar !== '\\') {
293 - if (!inString) {
294 - // Start of a new string
295 - inString = true;
296 - currentStr = char;
297 -
298 - // Check if this might be a key by looking ahead for a colon
299 - isKey = false;
300 - for (let j = i + 1; j < jsonStr.length; j++) {
301 - if (jsonStr[j] === '"' && jsonStr[j-1] !== '\\') {
302 - // End of string found, now look for colon
303 - for (let k = j + 1; k < jsonStr.length; k++) {
304 - if (jsonStr[k] === ':') {
305 - isKey = true;
306 - break;
307 - } else if (jsonStr[k] !== ' ' && jsonStr[k] !== '\t' && jsonStr[k] !== '\n') {
308 - break;
309 - }
310 - }
311 - break;
312 - }
313 - }
314 - } else {
315 - // End of a string
316 - currentStr += char;
317 - inString = false;
318 -
319 - // Apply the appropriate class for string formatting
320 - if (isKey) {
321 - result += '<span class="json-key">' + escapeHtml(currentStr) + '</span>';
322 - } else {
323 - result += '<span class="json-string">' + escapeHtml(currentStr) + '</span>';
324 - }
325 - currentStr = '';
326 - }
327 - }
328 - else if (inString) {
329 - // Inside a string, collect characters
330 - currentStr += char;
331 - }
332 - else if (char === ':') {
333 - result += '<span class="json-colon">:</span>';
334 - }
335 - else if (char === ',') {
336 - result += '<span class="json-comma">,</span>';
337 - }
338 - else if (char === '{' || char === '}' || char === '[' || char === ']') {
339 - result += '<span class="json-bracket">' + char + '</span>';
340 - }
341 - else if (char === 't' && jsonStr.substring(i, i+4) === 'true') {
342 - result += '<span class="json-boolean-true">true</span>';
343 - i += 3; // Skip the rest of "true"
344 - }
345 - else if (char === 'f' && jsonStr.substring(i, i+5) === 'false') {
346 - result += '<span class="json-boolean-false">false</span>';
347 - i += 4; // Skip the rest of "false"
348 - }
349 - else if (char === 'n' && jsonStr.substring(i, i+4) === 'null') {
350 - result += '<span class="json-null">null</span>';
351 - i += 3; // Skip the rest of "null"
352 - }
353 - else if (/[0-9]/.test(char) || char === '-') {
354 - // Start of a number
355 - let numberStr = char;
356 - for (let j = i + 1; j < jsonStr.length; j++) {
357 - if (/[0-9.eE+-]/.test(jsonStr[j])) {
358 - numberStr += jsonStr[j];
359 - i++;
360 - } else {
361 - break;
362 - }
363 - }
364 - result += '<span class="json-number">' + numberStr + '</span>';
365 - }
366 - else {
367 - // Whitespace and other characters
368 - result += char;
369 - }
370 -
371 - lastChar = char;
372 - }
373 -
374 - return result;
375 - }
376 -
377 - // Helper function to escape HTML entities
378 - function escapeHtml(text) {
379 - return text
380 - .replace(/&/g, '&amp;')
381 - .replace(/</g, '&lt;')
382 - .replace(/>/g, '&gt;')
383 - // Replace literal \n with actual line breaks for display
384 - .replace(/\\n/g, '<br>');
385 - }
386 -
387 - // State storage for method requests and responses
388 - const methodState = new Map();
389 - let currentFlow = 'initialize-flow';
390 - let currentMethod = null;
391 -
392 - // Method definitions by flow
393 - const flows = {
394 - 'initialize-flow': {
395 - name: 'Initialize',
396 - methods: {
397 - 'initialize': {
398 - name: 'initialize',
399 - description: 'Get information about available resources and methods',
400 - defaultRequest: {
401 - jsonrpc: '2.0',
402 - method: 'initialize',
403 - params: {
404 - protocolVersion: '2024-11-05',
405 - capabilities: {
406 - textDocument: {
407 - synchronization: {
408 - incremental: true
409 - }
410 - },
411 - completion: {
412 - contextSupport: true
413 - }
414 - },
415 - clientInfo: {
416 - name: 'netdata-mcp-test',
417 - version: '1.0.0',
418 - description: 'Netdata MCP Protocol Test Client'
419 - }
420 - },
421 - id: 1
422 - }
423 - }
424 - }
425 - },
426 - 'tools-flow': {
427 - name: 'Tools',
428 - methods: {
429 - 'tools/list': {
430 - name: 'tools/list',
431 - description: 'Get list of available tools with their schemas',
432 - defaultRequest: {
433 - jsonrpc: '2.0',
434 - method: 'tools/list',
435 - params: {},
436 - id: 1
437 - }
438 - },
439 - 'tools/execute': {
440 - name: 'tools/execute',
441 - description: 'Execute a tool with parameters',
442 - defaultRequest: {
443 - jsonrpc: '2.0',
444 - method: 'tools/execute',
445 - params: {
446 - name: 'explore_metrics',
447 - input: {
448 - context: 'system.cpu'
449 - }
450 - },
451 - id: 1
452 - }
453 - },
454 - 'tools/cancel': {
455 - name: 'tools/cancel',
456 - description: 'Cancel a running tool execution',
457 - defaultRequest: {
458 - jsonrpc: '2.0',
459 - method: 'tools/cancel',
460 - params: {
461 - executionId: '12345'
462 - },
463 - id: 1
464 - }
465 - },
466 - 'tools/status': {
467 - name: 'tools/status',
468 - description: 'Get status of a tool execution',
469 - defaultRequest: {
470 - jsonrpc: '2.0',
471 - method: 'tools/status',
472 - params: {
473 - executionId: '12345'
474 - },
475 - id: 1
476 - }
477 - },
478 - 'tools/validate': {
479 - name: 'tools/validate',
480 - description: 'Validate parameters for a tool',
481 - defaultRequest: {
482 - jsonrpc: '2.0',
483 - method: 'tools/validate',
484 - params: {
485 - name: 'explore_metrics',
486 - input: {
487 - context: 'system.cpu'
488 - }
489 - },
490 - id: 1
491 - }
492 - },
493 - 'tools/describe': {
494 - name: 'tools/describe',
495 - description: 'Get detailed documentation for a tool',
496 - defaultRequest: {
497 - jsonrpc: '2.0',
498 - method: 'tools/describe',
499 - params: {
500 - name: 'explore_metrics'
501 - },
502 - id: 1
503 - }
504 - },
505 - 'tools/getCapabilities': {
506 - name: 'tools/getCapabilities',
507 - description: 'Get capabilities of the tools subsystem',
508 - defaultRequest: {
509 - jsonrpc: '2.0',
510 - method: 'tools/getCapabilities',
511 - params: {},
512 - id: 1
513 - }
514 - }
515 - }
516 - },
517 - 'resources-flow': {
518 - name: 'Resources',
519 - methods: {
520 - 'resources/list': {
521 - name: 'resources/list',
522 - description: 'Get list of available resources',
523 - defaultRequest: {
524 - jsonrpc: '2.0',
525 - method: 'resources/list',
526 - params: {
527 - // Optional pagination cursor
528 - // cursor: "pagination_token"
529 - },
530 - id: 1
531 - }
532 - },
533 - 'resources/read': {
534 - name: 'resources/read',
535 - description: 'Read a specific resource by URI',
536 - defaultRequest: {
537 - jsonrpc: '2.0',
538 - method: 'resources/read',
539 - params: {
540 - uri: 'nd://contexts'
541 - },
542 - id: 1
543 - }
544 - },
545 - 'resources/templates/list': {
546 - name: 'resources/templates/list',
547 - description: 'Get list of resource templates for URI construction',
548 - defaultRequest: {
549 - jsonrpc: '2.0',
550 - method: 'resources/templates/list',
551 - params: {
552 - // Optional pagination cursor
553 - // cursor: "pagination_token"
554 - },
555 - id: 1
556 - }
557 - },
558 - 'resources/subscribe': {
559 - name: 'resources/subscribe',
560 - description: 'Subscribe to changes in a resource',
561 - defaultRequest: {
562 - jsonrpc: '2.0',
563 - method: 'resources/subscribe',
564 - params: {
565 - uri: 'nd://contexts'
566 - },
567 - id: 1
568 - }
569 - },
570 - 'resources/unsubscribe': {
571 - name: 'resources/unsubscribe',
572 - description: 'Unsubscribe from a resource',
573 - defaultRequest: {
574 - jsonrpc: '2.0',
575 - method: 'resources/unsubscribe',
576 - params: {
577 - uri: 'nd://contexts'
578 - },
579 - id: 1
580 - }
581 - }
582 - }
583 - },
584 - 'prompts-flow': {
585 - name: 'Prompts',
586 - methods: {
587 - 'prompts/list': {
588 - name: 'prompts/list',
589 - description: 'Get list of available prompts',
590 - defaultRequest: {
591 - jsonrpc: '2.0',
592 - method: 'prompts/list',
593 - params: {},
594 - id: 1
595 - }
596 - },
597 - 'prompts/execute': {
598 - name: 'prompts/execute',
599 - description: 'Execute a prompt',
600 - defaultRequest: {
601 - jsonrpc: '2.0',
602 - method: 'prompts/execute',
603 - params: {
604 - name: 'analyze_metrics',
605 - input: {
606 - context: 'system.cpu',
607 - timeframe: '1h'
608 - }
609 - },
610 - id: 1
611 - }
612 - },
613 - 'prompts/get': {
614 - name: 'prompts/get',
615 - description: 'Get a specific prompt',
616 - defaultRequest: {
617 - jsonrpc: '2.0',
618 - method: 'prompts/get',
619 - params: {
620 - name: 'analyze_metrics'
621 - },
622 - id: 1
623 - }
624 - },
625 - 'prompts/save': {
626 - name: 'prompts/save',
627 - description: 'Save a prompt',
628 - defaultRequest: {
629 - jsonrpc: '2.0',
630 - method: 'prompts/save',
631 - params: {
632 - name: 'my_custom_prompt',
633 - description: 'A custom prompt for analysis',
634 - template: 'Analyze the following metrics: {{context}}'
635 - },
636 - id: 1
637 - }
638 - },
639 - 'prompts/delete': {
640 - name: 'prompts/delete',
641 - description: 'Delete a prompt',
642 - defaultRequest: {
643 - jsonrpc: '2.0',
644 - method: 'prompts/delete',
645 - params: {
646 - name: 'my_custom_prompt'
647 - },
648 - id: 1
649 - }
650 - },
651 - 'prompts/getCategories': {
652 - name: 'prompts/getCategories',
653 - description: 'Get categories of prompts',
654 - defaultRequest: {
655 - jsonrpc: '2.0',
656 - method: 'prompts/getCategories',
657 - params: {},
658 - id: 1
659 - }
660 - },
661 - 'prompts/getHistory': {
662 - name: 'prompts/getHistory',
663 - description: 'Get history of prompt executions',
664 - defaultRequest: {
665 - jsonrpc: '2.0',
666 - method: 'prompts/getHistory',
667 - params: {
668 - limit: 10
669 - },
670 - id: 1
671 - }
672 - }
673 - }
674 - },
675 - 'notifications-flow': {
676 - name: 'Notifications',
677 - methods: {
678 - 'notifications/initialized': {
679 - name: 'notifications/initialized',
680 - description: 'Notify server that client is initialized',
681 - defaultRequest: {
682 - jsonrpc: '2.0',
683 - method: 'notifications/initialized'
684 - // No id for notifications
685 - }
686 - },
687 - 'notifications/subscribe': {
688 - name: 'notifications/subscribe',
689 - description: 'Subscribe to notifications',
690 - defaultRequest: {
691 - jsonrpc: '2.0',
692 - method: 'notifications/subscribe',
693 - params: {
694 - type: 'alerts'
695 - },
696 - id: 1
697 - }
698 - },
699 - 'notifications/unsubscribe': {
700 - name: 'notifications/unsubscribe',
701 - description: 'Unsubscribe from notifications',
702 - defaultRequest: {
703 - jsonrpc: '2.0',
704 - method: 'notifications/unsubscribe',
705 - params: {
706 - subscriptionId: '12345'
707 - },
708 - id: 1
709 - }
710 - },
711 - 'notifications/acknowledge': {
712 - name: 'notifications/acknowledge',
713 - description: 'Acknowledge a notification',
714 - defaultRequest: {
715 - jsonrpc: '2.0',
716 - method: 'notifications/acknowledge',
717 - params: {
718 - notificationId: '12345'
719 - },
720 - id: 1
721 - }
722 - },
723 - 'notifications/getHistory': {
724 - name: 'notifications/getHistory',
725 - description: 'Get history of notifications',
726 - defaultRequest: {
727 - jsonrpc: '2.0',
728 - method: 'notifications/getHistory',
729 - params: {
730 - limit: 10
731 - },
732 - id: 1
733 - }
734 - },
735 - 'notifications/send': {
736 - name: 'notifications/send',
737 - description: 'Send a notification',
738 - defaultRequest: {
739 - jsonrpc: '2.0',
740 - method: 'notifications/send',
741 - params: {
742 - type: 'message',
743 - content: 'Test notification from client'
744 - },
745 - id: 1
746 - }
747 - },
748 - 'notifications/getSettings': {
749 - name: 'notifications/getSettings',
750 - description: 'Get notification settings',
751 - defaultRequest: {
752 - jsonrpc: '2.0',
753 - method: 'notifications/getSettings',
754 - params: {},
755 - id: 1
756 - }
757 - }
758 - }
759 - },
760 - 'context-flow': {
761 - name: 'Context',
762 - methods: {
763 - 'context/provide': {
764 - name: 'context/provide',
765 - description: 'Provide context to the server',
766 - defaultRequest: {
767 - jsonrpc: '2.0',
768 - method: 'context/provide',
769 - params: {
770 - context: {
771 - user: 'admin',
772 - preferences: {
773 - theme: 'dark'
774 - }
775 - }
776 - },
777 - id: 1
778 - }
779 - },
780 - 'context/clear': {
781 - name: 'context/clear',
782 - description: 'Clear context',
783 - defaultRequest: {
784 - jsonrpc: '2.0',
785 - method: 'context/clear',
786 - params: {
787 - keys: ['preferences']
788 - },
789 - id: 1
790 - }
791 - },
792 - 'context/status': {
793 - name: 'context/status',
794 - description: 'Get context status',
795 - defaultRequest: {
796 - jsonrpc: '2.0',
797 - method: 'context/status',
798 - params: {},
799 - id: 1
800 - }
801 - },
802 - 'context/save': {
803 - name: 'context/save',
804 - description: 'Save context for future use',
805 - defaultRequest: {
806 - jsonrpc: '2.0',
807 - method: 'context/save',
808 - params: {
809 - name: 'my_saved_context'
810 - },
811 - id: 1
812 - }
813 - },
814 - 'context/load': {
815 - name: 'context/load',
816 - description: 'Load a saved context',
817 - defaultRequest: {
818 - jsonrpc: '2.0',
819 - method: 'context/load',
820 - params: {
821 - name: 'my_saved_context'
822 - },
823 - id: 1
824 - }
825 - }
826 - }
827 - },
828 - 'system-flow': {
829 - name: 'System',
830 - methods: {
831 - 'system/health': {
832 - name: 'system/health',
833 - description: 'Get system health information',
834 - defaultRequest: {
835 - jsonrpc: '2.0',
836 - method: 'system/health',
837 - params: {},
838 - id: 1
839 - }
840 - },
841 - 'system/version': {
842 - name: 'system/version',
843 - description: 'Get detailed version information',
844 - defaultRequest: {
845 - jsonrpc: '2.0',
846 - method: 'system/version',
847 - params: {},
848 - id: 1
849 - }
850 - },
851 - 'system/metrics': {
852 - name: 'system/metrics',
853 - description: 'Get system performance metrics',
854 - defaultRequest: {
855 - jsonrpc: '2.0',
856 - method: 'system/metrics',
857 - params: {},
858 - id: 1
859 - }
860 - },
861 - 'system/restart': {
862 - name: 'system/restart',
863 - description: 'Request system restart',
864 - defaultRequest: {
865 - jsonrpc: '2.0',
866 - method: 'system/restart',
867 - params: {
868 - force: false
869 - },
870 - id: 1
871 - }
872 - },
873 - 'system/status': {
874 - name: 'system/status',
875 - description: 'Get system status',
876 - defaultRequest: {
877 - jsonrpc: '2.0',
878 - method: 'system/status',
879 - params: {},
880 - id: 1
881 - }
882 - }
883 - }
884 - }
885 - // Additional flows can be added here later
886 - };
887 -
888 - // Initialize UI
889 - function initializeUI() {
890 - // Populate flow items
891 - flowsContainer.innerHTML = '';
892 - Object.keys(flows).forEach(flowId => {
893 - const flow = flows[flowId];
894 - const flowElement = document.createElement('div');
895 - flowElement.className = `flow-item ${flowId === currentFlow ? 'active' : ''}`;
896 - flowElement.dataset.flow = flowId;
897 - flowElement.textContent = flow.name;
898 - flowElement.addEventListener('click', () => selectFlow(flowId));
899 - flowsContainer.appendChild(flowElement);
900 - });
901 -
902 - // Load methods for current flow
903 - loadMethodsForFlow(currentFlow);
904 - }
905 -
906 - // Load methods for a specific flow
907 - function loadMethodsForFlow(flowId) {
908 - methodsContainer.innerHTML = '';
909 - const flow = flows[flowId];
910 -
911 - if (!flow) return;
912 -
913 - Object.keys(flow.methods).forEach(methodId => {
914 - const method = flow.methods[methodId];
915 - const methodElement = document.createElement('div');
916 - methodElement.className = 'method-item';
917 - methodElement.dataset.method = methodId;
918 - methodElement.dataset.flow = flowId;
919 -
920 - const methodName = document.createElement('div');
921 - methodName.textContent = method.name;
922 - methodName.style.fontWeight = 'bold';
923 -
924 - const methodDesc = document.createElement('div');
925 - methodDesc.textContent = method.description;
926 - methodDesc.style.fontSize = '0.9em';
927 - methodDesc.style.color = '#666';
928 -
929 - methodElement.appendChild(methodName);
930 - methodElement.appendChild(methodDesc);
931 -
932 - methodElement.addEventListener('click', () => selectMethod(flowId, methodId));
933 - methodsContainer.appendChild(methodElement);
934 - });
935 -
936 - // Select the first method by default, or restore the previously selected method
937 - const savedMethod = localStorage.getItem(`selected-method-${flowId}`);
938 - if (savedMethod && flows[flowId].methods[savedMethod]) {
939 - selectMethod(flowId, savedMethod);
940 - } else {
941 - const firstMethod = Object.keys(flow.methods)[0];
942 - selectMethod(flowId, firstMethod);
943 - }
944 - }
945 -
946 - // Select a flow
947 - function selectFlow(flowId) {
948 - // Update UI
949 - document.querySelectorAll('.flow-item').forEach(item => {
950 - item.classList.toggle('active', item.dataset.flow === flowId);
951 - });
952 -
953 - // Save current request if a method is selected
954 - if (currentMethod) {
955 - saveCurrentRequest();
956 - }
957 -
958 - // Update state
959 - currentFlow = flowId;
960 - localStorage.setItem('selected-flow', flowId);
961 -
962 - // Load methods for this flow
963 - loadMethodsForFlow(flowId);
964 - }
965 -
966 - // Select a method
967 - function selectMethod(flowId, methodId) {
968 - // Save current request if a method is already selected
969 - if (currentMethod) {
970 - saveCurrentRequest();
971 - }
972 -
973 - // Update UI
974 - document.querySelectorAll('.method-item').forEach(item => {
975 - item.classList.toggle('active', item.dataset.method === methodId);
976 - });
977 -
978 - // Update state
979 - currentFlow = flowId;
980 - currentMethod = methodId;
981 - localStorage.setItem('selected-flow', flowId);
982 - localStorage.setItem(`selected-method-${flowId}`, methodId);
983 -
984 - // Load saved or default request for this method
985 - loadRequestForMethod(flowId, methodId);
986 -
987 - // Load saved response for this method if exists
988 - loadResponseForMethod(flowId, methodId);
989 - }
990 -
991 - // Save the current request in the editor
992 - function saveCurrentRequest() {
993 - if (!currentFlow || !currentMethod) return;
994 -
995 - try {
996 - const requestObj = JSON.parse(jsonEditor.value);
997 - const key = `${currentFlow}:${currentMethod}`;
998 -
999 - if (!methodState.has(key)) {
1000 - methodState.set(key, {});
1001 - }
1002 -
1003 - methodState.get(key).request = requestObj;
1004 - localStorage.setItem(`request:${key}`, jsonEditor.value);
1005 - } catch (error) {
1006 - console.error('Failed to save request:', error);
1007 - }
1008 - }
1009 -
1010 - // Load a request for the selected method
1011 - function loadRequestForMethod(flowId, methodId) {
1012 - const key = `${flowId}:${methodId}`;
1013 - const defaultRequest = flows[flowId].methods[methodId].defaultRequest;
1014 -
1015 - // Try to load from session storage
1016 - let savedRequest = localStorage.getItem(`request:${key}`);
1017 -
1018 - if (savedRequest) {
1019 - jsonEditor.value = savedRequest;
1020 - } else if (methodState.has(key) && methodState.get(key).request) {
1021 - jsonEditor.value = JSON.stringify(methodState.get(key).request, null, 4);
1022 - } else {
1023 - // Use default
1024 - jsonEditor.value = JSON.stringify(defaultRequest, null, 4);
1025 - }
1026 - }
1027 -
1028 - // Load a saved response for the selected method
1029 - function loadResponseForMethod(flowId, methodId) {
1030 - const key = `${flowId}:${methodId}`;
1031 -
1032 - if (methodState.has(key) && methodState.get(key).response) {
1033 - responseViewer.innerHTML = syntaxHighlightJson(methodState.get(key).response);
1034 - if (methodState.get(key).requestSent) {
1035 - requestData.textContent = 'Request: ' + JSON.stringify(methodState.get(key).requestSent, null, 2);
1036 - } else {
1037 - requestData.textContent = '';
1038 - }
1039 -
1040 - // Show communication log if available
1041 - if (methodState.get(key).commLog) {
1042 - commLog.innerHTML = methodState.get(key).commLog;
1043 - } else {
1044 - commLog.innerHTML = '';
1045 - }
1046 - } else {
1047 - responseViewer.innerHTML = '<span class="json-null">No response yet.</span>';
1048 - requestData.textContent = '';
1049 - commLog.innerHTML = '';
1050 - }
1051 - }
1052 -
1053 - // Reset request to default
1054 - resetBtn.addEventListener('click', () => {
1055 - if (!currentFlow || !currentMethod) return;
1056 -
1057 - const defaultRequest = flows[currentFlow].methods[currentMethod].defaultRequest;
1058 - jsonEditor.value = JSON.stringify(defaultRequest, null, 4);
1059 -
1060 - // Update the next request ID
1061 - if (defaultRequest.id) {
1062 - defaultRequest.id = requestId++;
1063 - jsonEditor.value = JSON.stringify(defaultRequest, null, 4);
1064 - }
1065 - });
1066 -
1067 - // Clear communication log
1068 - clearLogBtn.addEventListener('click', () => {
1069 - commLog.innerHTML = '';
1070 -
1071 - // Clear the log in the current method state
1072 - if (currentFlow && currentMethod) {
1073 - const key = `${currentFlow}:${currentMethod}`;
1074 - if (methodState.has(key)) {
1075 - methodState.get(key).commLog = '';
1076 - }
1077 - }
1078 - });
1079 -
1080 - // Connect to WebSocket server
1081 - connectBtn.addEventListener('click', () => {
1082 - const url = serverUrlInput.value.trim();
1083 -
1084 - if (!url) {
1085 - alert('Please enter a WebSocket server URL');
1086 - return;
1087 - }
1088 -
1089 - try {
1090 - socket = new WebSocket(url);
1091 -
1092 - socket.onopen = () => {
1093 - isConnected = true;
1094 - statusSpan.textContent = 'Connected';
1095 - statusSpan.classList.remove('disconnected');
1096 - statusSpan.classList.add('connected');
1097 - connectBtn.disabled = true;
1098 - disconnectBtn.disabled = false;
1099 - sendBtn.disabled = false;
1100 - };
1101 -
1102 - socket.onclose = (event) => {
1103 - isConnected = false;
1104 - statusSpan.textContent = 'Disconnected';
1105 - statusSpan.classList.remove('connected');
1106 - statusSpan.classList.add('disconnected');
1107 - connectBtn.disabled = false;
1108 - disconnectBtn.disabled = true;
1109 - sendBtn.disabled = true;
1110 - };
1111 -
1112 - socket.onerror = (error) => {
1113 - console.error('WebSocket error:', error);
1114 - };
1115 -
1116 - socket.onmessage = (event) => {
1117 - try {
1118 - // Calculate response time
1119 - const responseTimestamp = new Date();
1120 - const responseTimeStr = responseTimestamp.toLocaleTimeString() + '.' +
1121 - String(responseTimestamp.getMilliseconds()).padStart(3, '0');
1122 -
1123 - // Calculate response size
1124 - const responseSize = new Blob([event.data]).size;
1125 -
1126 - const response = JSON.parse(event.data);
1127 -
1128 - // Display the response with syntax highlighting
1129 - responseViewer.innerHTML = syntaxHighlightJson(response);
1130 -
1131 - // Update communication log
1132 - let logEntry = '';
1133 - if (requestTimestamp) {
1134 - const elapsed = responseTimestamp - requestTimestamp;
1135 - logEntry = `[${responseTimeStr}] received response (${responseSize} bytes), latency ${elapsed}ms\n`;
1136 - commLog.innerHTML += logEntry;
1137 -
1138 - // Auto-scroll the log to the bottom
1139 - commLog.scrollTop = commLog.scrollHeight;
1140 - }
1141 -
1142 - // Save the response and log for the current method
1143 - if (currentFlow && currentMethod) {
1144 - const key = `${currentFlow}:${currentMethod}`;
1145 -
1146 - if (!methodState.has(key)) {
1147 - methodState.set(key, {});
1148 - }
1149 -
1150 - methodState.get(key).response = response;
1151 - methodState.get(key).responseTimestamp = responseTimestamp;
1152 - methodState.get(key).commLog = commLog.innerHTML;
1153 - }
1154 - } catch (error) {
1155 - // If parse fails, display the raw data
1156 - responseViewer.innerHTML = `<span class="json-string">${escapeHtml(event.data)}</span>`;
1157 - console.error('Error parsing JSON response:', error);
1158 - }
1159 - };
1160 -
1161 - } catch (error) {
1162 - alert(`Failed to connect: ${error.message}`);
1163 - console.error('Connection error:', error);
1164 - }
1165 - });
1166 -
1167 - // Disconnect from WebSocket server
1168 - disconnectBtn.addEventListener('click', () => {
1169 - if (socket) {
1170 - socket.close(1000, 'User initiated disconnect');
1171 - socket = null;
1172 - }
1173 - });
1174 -
1175 - // Send request
1176 - sendBtn.addEventListener('click', () => {
1177 - if (!socket || socket.readyState !== WebSocket.OPEN) {
1178 - alert('Not connected to the WebSocket server');
1179 - return;
1180 - }
1181 -
1182 - try {
1183 - const requestJson = jsonEditor.value.trim();
1184 - const requestObj = JSON.parse(requestJson);
1185 -
1186 - // Add an ID if not present and it's not a notification
1187 - if (requestObj.method && requestObj.method !== 'notification' && !requestObj.id) {
1188 - requestObj.id = requestId++;
1189 - jsonEditor.value = JSON.stringify(requestObj, null, 4);
1190 - }
1191 -
1192 - // Ensure jsonrpc version is set
1193 - if (!requestObj.jsonrpc) {
1194 - requestObj.jsonrpc = '2.0';
1195 - jsonEditor.value = JSON.stringify(requestObj, null, 4);
1196 - }
1197 -
1198 - const request = JSON.stringify(requestObj);
1199 - const requestSize = new Blob([request]).size;
1200 -
1201 - // Record request timestamp
1202 - requestTimestamp = new Date();
1203 - const requestTimeStr = requestTimestamp.toLocaleTimeString() + '.' +
1204 - String(requestTimestamp.getMilliseconds()).padStart(3, '0');
1205 -
1206 - // Add to the communication log
1207 - const logEntry = `[${requestTimeStr}] sending request (${requestSize} bytes)\n`;
1208 - commLog.innerHTML += logEntry;
1209 -
1210 - // Auto-scroll the log to the bottom
1211 - commLog.scrollTop = commLog.scrollHeight;
1212 -
1213 - // Send the request
1214 - socket.send(request);
1215 -
1216 - // Update request display
1217 - requestData.textContent = 'Request: ' + request;
1218 -
1219 - // Save the sent request and timestamp
1220 - if (currentFlow && currentMethod) {
1221 - const key = `${currentFlow}:${currentMethod}`;
1222 -
1223 - if (!methodState.has(key)) {
1224 - methodState.set(key, {});
1225 - }
1226 -
1227 - methodState.get(key).requestSent = requestObj;
1228 - methodState.get(key).requestTimestamp = requestTimestamp;
1229 - methodState.get(key).commLog = commLog.innerHTML;
1230 - }
1231 -
1232 - } catch (error) {
1233 - alert(`Error sending request: ${error.message}`);
1234 - console.error('Send error:', error);
1235 - }
1236 - });
1237 -
1238 - // Load saved state from localStorage
1239 - function loadSavedState() {
1240 - const savedFlow = localStorage.getItem('selected-flow');
1241 - if (savedFlow && flows[savedFlow]) {
1242 - currentFlow = savedFlow;
1243 - }
1244 -
1245 - // For each flow, load saved requests
1246 - Object.keys(flows).forEach(flowId => {
1247 - const flow = flows[flowId];
1248 - Object.keys(flow.methods).forEach(methodId => {
1249 - const key = `${flowId}:${methodId}`;
1250 - const savedRequest = localStorage.getItem(`request:${key}`);
1251 -
1252 - if (savedRequest) {
1253 - try {
1254 - const requestObj = JSON.parse(savedRequest);
1255 - if (!methodState.has(key)) {
1256 - methodState.set(key, {});
1257 - }
1258 - methodState.get(key).request = requestObj;
1259 - } catch (error) {
1260 - console.error(`Failed to parse saved request for ${key}:`, error);
1261 - }
1262 - }
1263 - });
1264 - });
1265 - }
1266 -
1267 - // Initialize the UI
1268 - loadSavedState();
1269 - initializeUI();
1270 - </script>
1271 -</body>
1272 -</html>
\ No newline at end of file
src/web/mcp/mcp.c
+163 -123
@@ -2,13 +2,15 @@
2
3 #include "mcp.h"
4 #include "mcp-initialize.h"
5 +#include "mcp-ping.h"
6 #include "mcp-tools.h"
7 #include "mcp-resources.h"
8 #include "mcp-prompts.h"
8 -#include "mcp-notifications.h"
9 -#include "mcp-context.h"
10 -#include "mcp-system.h"
9 +#include "mcp-logging.h"
10 +#include "mcp-completion.h"
11 +#include "mcp-tools-execute-function-registry.h"
12 #include "adapters/mcp-websocket.h"
13 +#include "mcp-api-key.h"
14
15 // Define the enum to string mapping for protocol versions
16 ENUM_STR_MAP_DEFINE(MCP_PROTOCOL_VERSION) = {
@@ -29,72 +31,77 @@ ENUM_STR_MAP_DEFINE(MCP_RETURN_CODE) = {
31 { .id = MCP_RC_NOT_FOUND, .name = "NOT_FOUND" },
32 { .id = MCP_RC_INTERNAL_ERROR, .name = "INTERNAL_ERROR" },
33 { .id = MCP_RC_NOT_IMPLEMENTED, .name = "NOT_IMPLEMENTED" },
34 + { .id = MCP_RC_BAD_REQUEST, .name = "BAD_REQUEST" },
35
36 // terminator
37 { .name = NULL, .id = 0 }
38 };
39 ENUM_STR_DEFINE_FUNCTIONS(MCP_RETURN_CODE, MCP_RC_ERROR, "ERROR");
40
38 -// Decode a URI component using mcpc's pre-allocated buffer
39 -// Returns a pointer to the decoded string which is valid until the next call
40 -const char *mcp_uri_decode(MCP_CLIENT *mcpc, const char *src) {
41 - if(!mcpc || !src || !*src)
42 - return src;
43 -
44 - // Prepare the buffer
45 - buffer_flush(mcpc->uri);
46 - buffer_need_bytes(mcpc->uri, strlen(src) + 1);
47 -
48 - // Perform URL decoding
49 - char *d = url_decode_r(mcpc->uri->buffer, src, mcpc->uri->size);
50 - if (!d || !*d)
51 - return src;
52 -
53 - // Ensure the buffer's length is updated
54 - mcpc->uri->len = strlen(d);
55 -
56 - return buffer_tostring(mcpc->uri);
57 -}
41 +// Define the enum to string mapping for logging levels
42 +ENUM_STR_MAP_DEFINE(MCP_LOGGING_LEVEL) = {
43 + { .id = MCP_LOGGING_LEVEL_DEBUG, .name = "debug" },
44 + { .id = MCP_LOGGING_LEVEL_INFO, .name = "info" },
45 + { .id = MCP_LOGGING_LEVEL_NOTICE, .name = "notice" },
46 + { .id = MCP_LOGGING_LEVEL_WARNING, .name = "warning" },
47 + { .id = MCP_LOGGING_LEVEL_ERROR, .name = "error" },
48 + { .id = MCP_LOGGING_LEVEL_CRITICAL, .name = "critical" },
49 + { .id = MCP_LOGGING_LEVEL_ALERT, .name = "alert" },
50 + { .id = MCP_LOGGING_LEVEL_EMERGENCY, .name = "emergency" },
51 + { .id = MCP_LOGGING_LEVEL_UNKNOWN, .name = "unknown" },
52 +
53 + // terminator
54 + { .name = NULL, .id = 0 }
55 +};
56 +ENUM_STR_DEFINE_FUNCTIONS(MCP_LOGGING_LEVEL, MCP_LOGGING_LEVEL_UNKNOWN, "unknown");
57
58 // Create a response context for a transport session
59 MCP_CLIENT *mcp_create_client(MCP_TRANSPORT transport, void *transport_ctx) {
61 - MCP_CLIENT *ctx = callocz(1, sizeof(MCP_CLIENT));
62 -
63 - ctx->transport = transport;
64 - ctx->protocol_version = MCP_PROTOCOL_VERSION_UNKNOWN; // Will be set during initialization
60 + MCP_CLIENT *mcpc = callocz(1, sizeof(MCP_CLIENT));
61 +
62 + mcpc->transport = transport;
63 + mcpc->protocol_version = MCP_PROTOCOL_VERSION_UNKNOWN; // Will be set during initialization
64 + mcpc->ready = false; // Client is not ready until initialized notification is received
65
66 // Set capabilities based on transport type
67 switch (transport) {
68 case MCP_TRANSPORT_WEBSOCKET:
69 - ctx->websocket = (struct websocket_server_client *)transport_ctx;
70 - ctx->capabilities = MCP_CAPABILITY_ASYNC_COMMUNICATION |
69 + mcpc->websocket = (struct websocket_server_client *)transport_ctx;
70 + mcpc->capabilities = MCP_CAPABILITY_ASYNC_COMMUNICATION |
71 MCP_CAPABILITY_SUBSCRIPTIONS |
72 MCP_CAPABILITY_NOTIFICATIONS;
73 break;
74
75 case MCP_TRANSPORT_HTTP:
76 - ctx->http = (struct web_client *)transport_ctx;
77 - ctx->capabilities = MCP_CAPABILITY_NONE; // HTTP has no special capabilities
76 + mcpc->http = (struct web_client *)transport_ctx;
77 + mcpc->capabilities = MCP_CAPABILITY_NONE; // HTTP has no special capabilities
78 break;
79
80 default:
81 - ctx->generic = transport_ctx;
82 - ctx->capabilities = MCP_CAPABILITY_NONE;
81 + mcpc->generic = transport_ctx;
82 + mcpc->capabilities = MCP_CAPABILITY_NONE;
83 break;
84 }
85
86 // Default client info (will be updated later from actual client)
87 - ctx->client_name = string_strdupz("unknown");
88 - ctx->client_version = string_strdupz("0.0.0");
87 + mcpc->client_name = string_strdupz("unknown");
88 + mcpc->client_version = string_strdupz("0.0.0");
89 +
90 + // Set default logging level to info
91 + mcpc->logging_level = MCP_LOGGING_LEVEL_INFO;
92
93 // Initialize response buffers
91 - ctx->result = buffer_create(4096, NULL);
92 - ctx->error = buffer_create(1024, NULL);
94 + mcpc->result = buffer_create(4096, NULL);
95 + mcpc->error = buffer_create(1024, NULL);
96
97 // Initialize utility buffers
95 - ctx->uri = buffer_create(1024, NULL);
98 + mcpc->uri = buffer_create(1024, NULL);
99
97 - return ctx;
100 + // Initialize request IDs tracking
101 + mcpc->request_id_counter = 0;
102 + mcpc->request_ids = NULL;
103 +
104 + return mcpc;
105 }
106
107 // Free a response context
@@ -110,6 +117,9 @@ void mcp_free_client(MCP_CLIENT *mcpc) {
117 // Free utility buffers
118 buffer_free(mcpc->uri);
119
120 + // Free request IDs
121 + mcp_request_id_cleanup_all(mcpc);
122 +
123 freez(mcpc);
124 }
125 }
@@ -127,49 +137,58 @@ static int mcp_map_return_code_to_jsonrpc_error(MCP_RETURN_CODE rc) {
137 return -32603; // JSON-RPC Internal error
138 case MCP_RC_NOT_IMPLEMENTED:
139 return -32601; // Use method not found for not implemented
140 + case MCP_RC_BAD_REQUEST:
141 + return -32600; // JSON-RPC Invalid request
142 case MCP_RC_ERROR:
143 default:
144 return -32000; // JSON-RPC Server error
145 }
146 }
147
136 -void mcp_init_success_result(MCP_CLIENT *mcpc, uint64_t id) {
137 - buffer_flush(mcpc->result);;
148 +void mcp_init_success_result(MCP_CLIENT *mcpc, MCP_REQUEST_ID id) {
149 + buffer_flush(mcpc->result);
150 buffer_json_initialize(mcpc->result, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
151 buffer_json_member_add_string(mcpc->result, "jsonrpc", "2.0");
152
141 - if(id)
142 - buffer_json_member_add_uint64(mcpc->result, "id", id);
153 + // Add the ID using our request ID system
154 + mcp_request_id_to_buffer(mcpc, mcpc->result, "id", id);
155 + buffer_json_member_add_object(mcpc->result, "result");
156
157 buffer_flush(mcpc->error);
158 }
159
147 -void mcp_jsonrpc_error(BUFFER *result, const char *error, uint64_t id, int jsonrpc_code) {
148 - buffer_flush(result);
149 - buffer_json_initialize(result, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
150 - buffer_json_member_add_string(result, "jsonrpc", "2.0");
160 +MCP_RETURN_CODE mcp_error_result(MCP_CLIENT *mcpc, MCP_REQUEST_ID id, MCP_RETURN_CODE rc) {
161 + if (!mcpc) return rc;
162
152 - if (id)
153 - buffer_json_member_add_uint64(result, "id", id);
154 -
155 - buffer_json_member_add_int64(result, "error", jsonrpc_code);
156 -
157 - if(error && *error)
158 - buffer_json_member_add_string(result, "message", error);
159 -
160 - buffer_json_finalize(result);
161 -}
163 + buffer_flush(mcpc->result);
164 + buffer_json_initialize(mcpc->result, "\"", "\"", 0, true, BUFFER_JSON_OPTIONS_MINIFY);
165 + buffer_json_member_add_string(mcpc->result, "jsonrpc", "2.0");
166 +
167 + // Add the ID using our request ID system
168 + mcp_request_id_to_buffer(mcpc, mcpc->result, "id", id);
169
163 -MCP_RETURN_CODE mcp_error_result(MCP_CLIENT *mcpc, uint64_t id, MCP_RETURN_CODE rc) {
164 - mcp_jsonrpc_error(mcpc->result,
165 - buffer_strlen(mcpc->error) ? buffer_tostring(mcpc->error) : MCP_RETURN_CODE_2str(rc),
166 - id, mcp_map_return_code_to_jsonrpc_error(rc));
170 + buffer_json_member_add_object(mcpc->result, "error");
171 + buffer_json_member_add_int64(mcpc->result, "code", mcp_map_return_code_to_jsonrpc_error(rc));
172 +
173 + const char *error_message = buffer_strlen(mcpc->error)
174 + ? buffer_tostring(mcpc->error)
175 + : MCP_RETURN_CODE_2str(rc);
176 +
177 + if(error_message && *error_message)
178 + buffer_json_member_add_string(mcpc->result, "message", error_message);
179 +
180 + buffer_json_object_close(mcpc->result); // Close error
181 +
182 + buffer_json_finalize(mcpc->result);
183 return rc;
184 }
185
186 +// No longer needed - we're using mcp_request_id_del directly in mcp_single_request
187 +
188 // Send the content of a buffer using the appropriate transport
189 int mcp_send_response_buffer(MCP_CLIENT *mcpc) {
172 - if (!mcpc || !mcpc->result || !buffer_strlen(mcpc->result)) return -1;
190 + if (!mcpc || !mcpc->result || !buffer_strlen(mcpc->result))
191 + return -1;
192
193 switch (mcpc->transport) {
194 case MCP_TRANSPORT_WEBSOCKET:
@@ -186,8 +205,8 @@ int mcp_send_response_buffer(MCP_CLIENT *mcpc) {
205 }
206
207 // Parse and extract client info from initialize request params
189 -static void mcp_extract_client_info(MCP_CLIENT *ctx, struct json_object *params) {
190 - if (!ctx || !params) return;
208 +static void mcp_extract_client_info(MCP_CLIENT *mcpc, struct json_object *params) {
209 + if (!mcpc || !params) return;
210
211 struct json_object *client_info_obj = NULL;
212 struct json_object *client_name_obj = NULL;
@@ -195,12 +214,12 @@ static void mcp_extract_client_info(MCP_CLIENT *ctx, struct json_object *params)
214
215 if (json_object_object_get_ex(params, "clientInfo", &client_info_obj)) {
216 if (json_object_object_get_ex(client_info_obj, "name", &client_name_obj)) {
198 - string_freez(ctx->client_name);
199 - ctx->client_name = string_strdupz(json_object_get_string(client_name_obj));
217 + string_freez(mcpc->client_name);
218 + mcpc->client_name = string_strdupz(json_object_get_string(client_name_obj));
219 }
220 if (json_object_object_get_ex(client_info_obj, "version", &client_version_obj)) {
202 - string_freez(ctx->client_version);
203 - ctx->client_version = string_strdupz(json_object_get_string(client_version_obj));
221 + string_freez(mcpc->client_version);
222 + mcpc->client_version = string_strdupz(json_object_get_string(client_version_obj));
223 }
224 }
225 }
@@ -212,13 +231,12 @@ static MCP_RETURN_CODE mcp_single_request(MCP_CLIENT *mcpc, struct json_object *
231 }
232
233 // Flush buffers before processing the request
215 - buffer_flush(mcpc->result);
216 - buffer_flush(mcpc->error);
234 + buffer_reset(mcpc->result);
235 + buffer_reset(mcpc->error);
236
237 // Extract JSON-RPC fields
238 struct json_object *method_obj = NULL;
239 struct json_object *params_obj = NULL;
221 - struct json_object *id_obj = NULL;
240 struct json_object *jsonrpc_obj = NULL;
241
242 // Validate jsonrpc version
@@ -239,6 +257,7 @@ static MCP_RETURN_CODE mcp_single_request(MCP_CLIENT *mcpc, struct json_object *
257 const char *method = json_object_get_string(method_obj);
258
259 // Extract params (optional)
260 + bool params_created = false;
261 if (json_object_object_get_ex(request, "params", &params_obj)) {
262 if (json_object_get_type(params_obj) != json_type_object) {
263 buffer_strcat(mcpc->error, "params must be an object");
@@ -248,62 +267,69 @@ static MCP_RETURN_CODE mcp_single_request(MCP_CLIENT *mcpc, struct json_object *
267 } else {
268 // Create an empty params object if none provided
269 params_obj = json_object_new_object();
270 + params_created = true;
271 }
272
253 - // Extract ID (optional, for notifications)
254 - uint64_t id = 0;
255 - bool has_id = json_object_object_get_ex(request, "id", &id_obj);
256 -
273 + // Extract and register the request ID
274 + MCP_REQUEST_ID id = mcp_request_id_add(mcpc, request);
275 + bool has_id = (id != 0);
276 +
277 + // If we have a request ID, log it
278 if (has_id) {
258 - if (json_object_get_type(id_obj) == json_type_int) {
259 - id = json_object_get_int64(id_obj);
260 - }
261 - else if (json_object_get_type(id_obj) == json_type_string) {
262 - const char *id_str = json_object_get_string(id_obj);
263 - char *endptr;
264 - id = strtoull(id_str, &endptr, 10);
265 - if (*endptr != '\0') {
266 - // If the string is not a number, use a hash of the string as the ID
267 - id = 0;
268 - while (*id_str) {
269 - id = id * 31 + (*id_str++);
270 - }
271 - }
272 - }
279 + netdata_log_debug(D_WEB_CLIENT, "MCP: Handling method call: %s (request_id: %zu)", method, id);
280 + } else {
281 + netdata_log_debug(D_WEB_CLIENT, "MCP: Handling notification: %s (no id)", method);
282 }
283
275 - netdata_log_debug(D_WEB_CLIENT, "MCP: Handling method call: %s (id: %"PRIu64")", method, id);
276 -
284 // Handle method calls based on namespace
285 MCP_RETURN_CODE rc;
286
287 + // Check for notifications/initialized method which marks client as ready
288 if(!method || !*method) {
289 buffer_strcat(mcpc->error, "Empty method name");
290 rc = MCP_RC_INVALID_PARAMS;
291 }
292 + else if (strcmp(method, "notifications/initialized") == 0) {
293 + mcpc->ready = true;
294 + netdata_log_debug(D_WEB_CLIENT, "MCP client %s v%s is now ready",
295 + string2str(mcpc->client_name), string2str(mcpc->client_version));
296 + rc = MCP_RC_OK;
297 + }
298 else if (strncmp(method, "tools/", 6) == 0) {
299 // Tools namespace
300 rc = mcp_tools_route(mcpc, method + 6, params_obj, id);
301 + // Mark client as ready if not already
302 + if (!mcpc->ready) {
303 + mcpc->ready = true;
304 + }
305 }
306 else if (strncmp(method, "resources/", 10) == 0) {
307 // Resources namespace
308 rc = mcp_resources_route(mcpc, method + 10, params_obj, id);
291 - }
292 - else if (strncmp(method, "notifications/", 14) == 0) {
293 - // Notifications namespace
294 - rc = mcp_notifications_route(mcpc, method + 14, params_obj, id);
309 + // Mark client as ready if not already
310 + if (!mcpc->ready) {
311 + mcpc->ready = true;
312 + }
313 }
314 else if (strncmp(method, "prompts/", 8) == 0) {
315 // Prompts namespace
316 rc = mcp_prompts_route(mcpc, method + 8, params_obj, id);
317 + // Mark client as ready if not already
318 + if (!mcpc->ready) {
319 + mcpc->ready = true;
320 + }
321 }
300 - else if (strncmp(method, "context/", 8) == 0) {
301 - // Context namespace
302 - rc = mcp_context_route(mcpc, method + 8, params_obj, id);
322 + else if (strncmp(method, "logging/", 8) == 0) {
323 + // Logging namespace - don't alter ready state
324 + rc = mcp_logging_route(mcpc, method + 8, params_obj, id);
325 }
304 - else if (strncmp(method, "system/", 7) == 0) {
305 - // System namespace
306 - rc = mcp_system_route(mcpc, method + 7, params_obj, id);
326 + else if (strncmp(method, "completion/", 11) == 0) {
327 + // Completion namespace
328 + rc = mcp_completion_route(mcpc, method + 11, params_obj, id);
329 + // Mark client as ready if not already
330 + if (!mcpc->ready) {
331 + mcpc->ready = true;
332 + }
333 }
334 else if (strcmp(method, "initialize") == 0) {
335 // Extract client info from initialize request
@@ -314,13 +340,23 @@ static MCP_RETURN_CODE mcp_single_request(MCP_CLIENT *mcpc, struct json_object *
340 // Handle initialize method
341 rc = mcp_method_initialize(mcpc, params_obj, id);
342 }
343 + else if (strcmp(method, "ping") == 0) {
344 + // Handle ping method - simple connection health check
345 + // Don't alter ready state for ping requests
346 + rc = mcp_method_ping(mcpc, params_obj, id);
347 + }
348 else {
349 buffer_sprintf(mcpc->error, "Method '%s' not found", method);
350 rc = MCP_RC_NOT_FOUND;
351 + // Method not found shouldn't alter ready state
352 }
353
354 // If this is a notification (no ID), don't generate a response
355 if (!has_id) {
356 + // Clean up the params object if we created it
357 + if (params_created) {
358 + json_object_put(params_obj);
359 + }
360 return rc;
361 }
362
@@ -334,12 +370,21 @@ static MCP_RETURN_CODE mcp_single_request(MCP_CLIENT *mcpc, struct json_object *
370 mcp_error_result(mcpc, id, MCP_RC_INTERNAL_ERROR);
371 }
372
373 + // Clean up the request ID
374 + mcp_request_id_del(mcpc, id);
375 +
376 + // Clean up the params object if we created it
377 + if (params_created) {
378 + json_object_put(params_obj);
379 + }
380 +
381 return rc;
382 }
383
384 // Main MCP entry point - handle a JSON-RPC request (can be single or batch)
385 MCP_RETURN_CODE mcp_handle_request(MCP_CLIENT *mcpc, struct json_object *request) {
342 - if (!mcpc || !request) return MCP_RC_INTERNAL_ERROR;
386 + if (!mcpc || !request)
387 + return MCP_RC_INTERNAL_ERROR;
388
389 // Clear previous response buffers
390 buffer_flush(mcpc->result);
@@ -347,7 +392,7 @@ MCP_RETURN_CODE mcp_handle_request(MCP_CLIENT *mcpc, struct json_object *request
392
393 // Check if this is a batch request (JSON array)
394 if (json_object_get_type(request) == json_type_array) {
350 - int array_len = json_object_array_length(request);
395 + int array_len = (int)json_object_array_length(request);
396
397 // Empty batch should return nothing according to JSON-RPC 2.0 spec
398 if (array_len == 0) {
@@ -362,7 +407,7 @@ MCP_RETURN_CODE mcp_handle_request(MCP_CLIENT *mcpc, struct json_object *request
407 buffer_strcat(batch_buffer, "[");
408
409 // Track if we've added any responses (for comma handling)
365 - bool has_responses = false;
410 + size_t responses_added = 0;
411
412 // Process each request in the batch
413 for (int i = 0; i < array_len; i++) {
@@ -372,30 +417,26 @@ MCP_RETURN_CODE mcp_handle_request(MCP_CLIENT *mcpc, struct json_object *request
417 buffer_flush(mcpc->result);
418 buffer_flush(mcpc->error);
419
375 - // Extract ID to determine if it's a request or notification
376 - struct json_object *id_obj = NULL;
377 - bool has_id = json_object_object_get_ex(req_item, "id", &id_obj);
378 -
420 // Call the single request handler
421 mcp_single_request(mcpc, req_item);
422
423 // For notifications (no id), don't add to response
383 - if (!has_id || buffer_strlen(mcpc->result) == 0) {
424 + if (buffer_strlen(mcpc->result) == 0) {
425 continue;
426 }
427
428 // Add comma if this isn't the first response
388 - if (has_responses) {
429 + if (responses_added) {
430 buffer_strcat(batch_buffer, ", ");
431 }
432
433 // Add the response to the batch
434 buffer_strcat(batch_buffer, buffer_tostring(mcpc->result));
394 - has_responses = true;
435 + responses_added++;
436 }
437
438 // If no responses were added (all notifications), don't send anything per JSON-RPC spec
398 - if (!has_responses) {
439 + if (!responses_added) {
440 buffer_free(batch_buffer);
441 return MCP_RC_OK;
442 }
@@ -416,21 +457,20 @@ MCP_RETURN_CODE mcp_handle_request(MCP_CLIENT *mcpc, struct json_object *request
457 else {
458 // Handle single request
459 MCP_RETURN_CODE rc = mcp_single_request(mcpc, request);
419 -
420 - // Extract ID to determine if it's a request or notification
421 - struct json_object *id_obj = NULL;
422 - bool has_id = json_object_object_get_ex(request, "id", &id_obj);
423 -
424 - // Only send responses for requests with IDs, not for notifications
425 - if (has_id && buffer_strlen(mcpc->result) > 0) {
426 - mcp_send_response_buffer(mcpc);
427 - }
428 -
460 + mcp_send_response_buffer(mcpc);
461 return rc;
462 }
463 }
464
465 // Initialize the MCP subsystem
466 void mcp_initialize_subsystem(void) {
467 + mcp_functions_registry_init();
468 +
469 +#ifdef NETDATA_MCP_DEV_PREVIEW_API_KEY
470 + mcp_api_key_initialize();
471 +#endif
472 +
473 + // debug_flags |= D_MCP;
474 +
475 netdata_log_info("MCP subsystem initialized");
476 }
src/web/mcp/mcp.h
+108 -17
@@ -5,6 +5,82 @@
5
6 #include "libnetdata/libnetdata.h"
7 #include <json-c/json.h>
8 +#include "mcp-request-id.h"
9 +
10 +// MCP tool names - use these constants when referring to tools
11 +#define MCP_TOOL_LIST_METRICS "list_metrics"
12 +#define MCP_TOOL_GET_METRICS_DETAILS "get_metrics_details"
13 +#define MCP_TOOL_LIST_NODES "list_nodes"
14 +#define MCP_TOOL_GET_NODES_DETAILS "get_nodes_details"
15 +#define MCP_TOOL_LIST_FUNCTIONS "list_functions"
16 +#define MCP_TOOL_EXECUTE_FUNCTION "execute_function"
17 +#define MCP_TOOL_QUERY_METRICS "query_metrics"
18 +#define MCP_TOOL_FIND_CORRELATED_METRICS "find_correlated_metrics"
19 +#define MCP_TOOL_FIND_ANOMALOUS_METRICS "find_anomalous_metrics"
20 +#define MCP_TOOL_FIND_UNSTABLE_METRICS "find_unstable_metrics"
21 +#define MCP_TOOL_LIST_RAISED_ALERTS "list_raised_alerts"
22 +#define MCP_TOOL_LIST_ALL_ALERTS "list_running_alerts"
23 +#define MCP_TOOL_LIST_ALERT_TRANSITIONS "list_alert_transitions"
24 +
25 +#define MCP_INFO_TOO_MANY_CONTEXTS_GROUPED_IN_CATEGORIES \
26 + "Next Steps: provide a better pattern to match less contexts, " \
27 + "or run '" MCP_TOOL_GET_METRICS_DETAILS "' to get more information for the contexts of interest."
28 +
29 +#define MCP_INFO_CONTEXT_ARRAY_RESPONSE \
30 + "Next Steps: run the '"MCP_TOOL_GET_METRICS_DETAILS"' tool to get more information for the contexts of interest."
31 +
32 +#define MCP_INFO_CONTEXT_NEXT_STEPS \
33 + "Next Steps: Query time-series data with the '"MCP_TOOL_QUERY_METRICS"' tool, using different aggregations to inspect different views:\n" \
34 + " - 'group_by: dimension' will aggregate all time-series by the listed dimensions\n" \
35 + " - 'group_by: instance' will aggregate all time-series by the listed instances\n" \
36 + " - 'group_by: label, group_by_label: {label_key}' will aggregate by the listed label values\n" \
37 + "\n" \
38 + "Dimensions, instances and labels can also be used for filtering in '"MCP_TOOL_QUERY_METRICS"':\n" \
39 + " - 'dimensions: dimension1|dimension2|*dimension*' will select only the time-series with the given dimension\n" \
40 + " - 'instances: instance1|instance2|*instance*' will select only the time-series with the given instance\n" \
41 + " - 'labels' can be specified in two formats:\n" \
42 + " • String format: 'labels: key1:value1|key1:value2|key2:value3' (values with same key are ORed, different keys are ANDed)\n" \
43 + " • Structured format: 'labels: {\"key1\": [\"value1\", \"value2\"], \"key2\": \"value3\"}' (array values are ORed, different keys are ANDed)"
44 +
45 +// MCP default values for all tools
46 +#define MCP_DEFAULT_AFTER_TIME (-3600) // 1 hour ago
47 +#define MCP_DEFAULT_BEFORE_TIME 0 // now
48 +#define MCP_METADATA_CARDINALITY_LIMIT 50 // For metadata queries
49 +#define MCP_DATA_CARDINALITY_LIMIT 10 // For data queries
50 +#define MCP_WEIGHTS_CARDINALITY_LIMIT 50 // For weights queries (minimum is 30)
51 +#define MCP_METADATA_CARDINALITY_LIMIT_MAX 500 // For metadata queries
52 +#define MCP_DATA_CARDINALITY_LIMIT_MAX 500 // For data queries
53 +#define MCP_WEIGHTS_CARDINALITY_LIMIT_MAX 500 // For weights queries
54 +#define MCP_ALERTS_CARDINALITY_LIMIT 100 // For alert queries
55 +#define MCP_ALERTS_CARDINALITY_LIMIT_MAX 500 // For alert queries
56 +
57 +// MCP query info messages
58 +#define MCP_QUERY_INFO_SUMMARY_SECTION \
59 + "The summary section breaks down the different sources that contribute " \
60 + "data to the query. Use this to detect spikes, dives, anomalies (the % of anomalous samples vs the total samples) " \
61 + "and evaluate the different groupings that may be beneficial for the task at hand."
62 +
63 +#define MCP_QUERY_INFO_DATABASE_SECTION \
64 + "The database section provides metadata about the underlying data storage, " \
65 + "including retention periods and update frequencies, and data availability " \
66 + "across different storage tiers."
67 +
68 +#define MCP_QUERY_INFO_VIEW_SECTION \
69 + "The view section provides summarized data for the visible time window. " \
70 + "For each dimension returned, it contains the minimum, maximum, and average values, " \
71 + "the anomaly rate (% of anomalous samples vs total samples) and contribution percentages, " \
72 + "across all points."
73 +
74 +#define MCP_QUERY_INFO_RESULT_SECTION \
75 + "The 'result' section contains the actual time-series data points.\n" \
76 + "Each point of each dimension is represented as an array of 3 values:\n" \
77 + " a) the value itself, aggregated as requested\n" \
78 + " b) the point anomaly rate percentage (% of anomalous samples vs total samples)\n" \
79 + " c) the point annotations, a combined bitmap of 1+2+4, where:\n" \
80 + " 1 = empty data, value should be ignored\n" \
81 + " 2 = counter has been reset or overflown, value may not be accurate\n" \
82 + " 4 = partial data, at least one of the sources aggregated had gaps at that time\n" \
83 + "Summarized data across the entire time-frame is provided at the 'view' section."
84
85 // MCP protocol versions
86 typedef enum {
@@ -18,16 +94,6 @@ typedef enum {
94 } MCP_PROTOCOL_VERSION;
95 ENUM_STR_DEFINE_FUNCTIONS_EXTERN(MCP_PROTOCOL_VERSION);
96
21 -// JSON-RPC error codes (standard)
22 -#define MCP_ERROR_PARSE_ERROR -32700
23 -#define MCP_ERROR_INVALID_REQUEST -32600
24 -#define MCP_ERROR_METHOD_NOT_FOUND -32601
25 -#define MCP_ERROR_INVALID_PARAMS -32602
26 -#define MCP_ERROR_INTERNAL_ERROR -32603
27 -// Server error codes (implementation-defined)
28 -#define MCP_ERROR_SERVER_ERROR_MIN -32099
29 -#define MCP_ERROR_SERVER_ERROR_MAX -32000
30 -
97 // Content types (for messages and tool responses)
98 typedef enum {
99 MCP_CONTENT_TYPE_TEXT = 0,
@@ -35,6 +101,20 @@ typedef enum {
101 MCP_CONTENT_TYPE_AUDIO = 2, // New in 2025-03-26
102 } MCP_CONTENT_TYPE;
103
104 +// Logging levels (as defined in MCP schema)
105 +typedef enum {
106 + MCP_LOGGING_LEVEL_UNKNOWN = 0,
107 + MCP_LOGGING_LEVEL_DEBUG,
108 + MCP_LOGGING_LEVEL_INFO,
109 + MCP_LOGGING_LEVEL_NOTICE,
110 + MCP_LOGGING_LEVEL_WARNING,
111 + MCP_LOGGING_LEVEL_ERROR,
112 + MCP_LOGGING_LEVEL_CRITICAL,
113 + MCP_LOGGING_LEVEL_ALERT,
114 + MCP_LOGGING_LEVEL_EMERGENCY
115 +} MCP_LOGGING_LEVEL;
116 +ENUM_STR_DEFINE_FUNCTIONS_EXTERN(MCP_LOGGING_LEVEL);
117 +
118 // Forward declarations for transport-specific types
119 struct websocket_server_client;
120 struct web_client;
@@ -63,7 +143,8 @@ typedef enum {
143 MCP_RC_INVALID_PARAMS = 2, // Invalid parameters in request
144 MCP_RC_NOT_FOUND = 3, // Resource or method not found
145 MCP_RC_INTERNAL_ERROR = 4, // Internal server error
66 - MCP_RC_NOT_IMPLEMENTED = 5 // Method not implemented
146 + MCP_RC_NOT_IMPLEMENTED = 5, // Method not implemented
147 + MCP_RC_BAD_REQUEST = 6 // Bad or malformed request
148 // Can add more specific errors as needed
149 } MCP_RETURN_CODE;
150 ENUM_STR_DEFINE_FUNCTIONS_EXTERN(MCP_RETURN_CODE);
@@ -77,6 +158,9 @@ typedef struct mcp_client {
158 // Protocol version (detected during initialization)
159 MCP_PROTOCOL_VERSION protocol_version;
160
161 + // Client state
162 + bool ready; // Set to true when client is ready for normal operations
163 +
164 // Transport-specific context
165 union {
166 struct websocket_server_client *websocket; // WebSocket client
@@ -84,16 +168,26 @@ typedef struct mcp_client {
168 void *generic; // Generic context
169 };
170
171 + // Authentication and authorization
172 + USER_AUTH *user_auth; // Pointer to user auth from the underlying transport
173 +
174 // Client information
175 STRING *client_name; // Client name (for logging, interned)
176 STRING *client_version; // Client version (for logging, interned)
177
178 + // Logging configuration
179 + MCP_LOGGING_LEVEL logging_level; // Current logging level set by client
180 +
181 // Response buffers
182 BUFFER *result; // Pre-allocated buffer for success responses
183 BUFFER *error; // Pre-allocated buffer for error messages
184
185 // Utility buffers
96 - BUFFER *uri; // Pre-allocated buffer for URI decoding
186 + BUFFER *uri; // Pre-allocated buffer for URI decoding
187 +
188 + // Request IDs tracking
189 + size_t request_id_counter; // Counter for generating sequential request IDs
190 + Pvoid_t request_ids; // JudyL array for mapping internal IDs to client IDs
191 } MCP_CLIENT;
192
193 // Helper function to convert string version to numeric version
@@ -111,9 +205,8 @@ void mcp_free_client(MCP_CLIENT *mcpc);
205 // Helper functions for creating and sending JSON-RPC responses
206
207 // Functions to initialize and build MCP responses
114 -void mcp_init_success_result(MCP_CLIENT *mcpc, uint64_t id);
115 -MCP_RETURN_CODE mcp_error_result(MCP_CLIENT *mcpc, uint64_t id, MCP_RETURN_CODE rc);
116 -void mcp_jsonrpc_error(BUFFER *result, const char *error, uint64_t id, int jsonrpc_code);
208 +void mcp_init_success_result(MCP_CLIENT *mcpc, MCP_REQUEST_ID id);
209 +MCP_RETURN_CODE mcp_error_result(MCP_CLIENT *mcpc, MCP_REQUEST_ID id, MCP_RETURN_CODE rc);
210
211 // Send prepared buffer content as response
212 int mcp_send_response_buffer(MCP_CLIENT *mcpc);
@@ -129,6 +222,4 @@ void mcp_initialize_subsystem(void);
222 // Main MCP entry point - handle a JSON-RPC request (single or batch)
223 MCP_RETURN_CODE mcp_handle_request(MCP_CLIENT *mcpc, struct json_object *request);
224
132 -const char *mcp_uri_decode(MCP_CLIENT *mcpc, const char *src);
133 -
225 #endif // NETDATA_MCP_H
src/web/server/README.md
+20
@@ -277,3 +277,23 @@ When you start using Netdata with TLS, you may find errors in the Netdata log, w
277 Most of the time, these errors are due to incompatibilities between your browser's options related to TLS/SSL protocols and Netdata's internal configuration. The most common error is `error:00000006:lib(0):func(0):EVP lib`.
278
279 </details>
280 +
281 +### WebSocket Support
282 +
283 +<details><summary>details</summary>
284 +
285 +Netdata supports WebSocket connections for real-time data streaming and interactive features. For detailed information on WebSocket protocols, configuration options, and examples, see the [WebSocket documentation](/src/web/websocket/README.md).
286 +
287 +#### WebSocket Frame Size Configuration
288 +
289 +Netdata automatically fragments large WebSocket messages to ensure browser compatibility. The default maximum outgoing frame size is 4MB, which works well for most browsers.
290 +
291 +For clients with specific requirements, you can customize the maximum frame size on a per-connection basis by adding the `max_frame_size` parameter to the WebSocket URL:
292 +
293 +```
294 +ws://localhost:19999/echo?max_frame_size=32768
295 +```
296 +
297 +This is particularly useful for resource-constrained devices or network environments with specific limitations.
298 +
299 +</details>
src/web/server/h2o/rrdpush.c
+3 -3
@@ -296,9 +296,9 @@ void stream_process(h2o_stream_conn_t *conn, int initial)
296 socklen_t len = h2o_socket_getpeername(conn->sock, &client);
297 char peername[NI_MAXHOST];
298 size_t peername_len = h2o_socket_getnumerichost(&client, len, peername);
299 - size_t cpy_len = sizeof(w.client_ip) < peername_len ? sizeof(w.client_ip) : peername_len;
300 - memcpy(w.client_ip, peername, cpy_len);
301 - w.client_ip[cpy_len - 1] = 0;
299 + size_t cpy_len = sizeof(w.user_auth.client_ip) < peername_len ? sizeof(w.user_auth.client_ip) : peername_len;
300 + memcpy(w.user_auth.client_ip, peername, cpy_len);
301 + w.user_auth.client_ip[cpy_len - 1] = 0;
302 w.user_agent = conn->user_agent;
303
304 rc = stream_receiver_accept_connection(&w, conn->url, conn);
src/web/server/static/static-threaded.c
+2 -2
@@ -32,11 +32,11 @@ static struct web_client *web_client_create_on_fd(POLLINFO *pi) {
32 w = web_client_get_from_cache();
33 w->fd = pi->fd;
34
35 - strncpyz(w->client_ip, pi->client_ip, sizeof(w->client_ip) - 1);
35 + strncpyz(w->user_auth.client_ip, pi->client_ip, sizeof(w->user_auth.client_ip) - 1);
36 strncpyz(w->client_port, pi->client_port, sizeof(w->client_port) - 1);
37 strncpyz(w->client_host, pi->client_host, sizeof(w->client_host) - 1);
38
39 - if(unlikely(!*w->client_ip)) strcpy(w->client_ip, "-");
39 + if(unlikely(!*w->user_auth.client_ip)) strcpy(w->user_auth.client_ip, "-");
40 if(unlikely(!*w->client_port)) strcpy(w->client_port, "-");
41 w->port_acl = pi->port_acl;
42
src/web/server/web_client.c
+34 -36
@@ -32,16 +32,16 @@ void web_client_set_conn_webrtc(struct web_client *w) {
32 }
33
34 void web_client_reset_permissions(struct web_client *w) {
35 - web_client_flags_clear_auth(w);
36 - w->access = HTTP_ACCESS_NONE;
37 - w->user_role = HTTP_USER_ROLE_NONE;
35 + w->user_auth.method = USER_AUTH_METHOD_NONE;
36 + w->user_auth.access = HTTP_ACCESS_NONE;
37 + w->user_auth.user_role = HTTP_USER_ROLE_NONE;
38 }
39
40 -void web_client_set_permissions(struct web_client *w, HTTP_ACCESS access, HTTP_USER_ROLE role, WEB_CLIENT_FLAGS auth) {
40 +void web_client_set_permissions(struct web_client *w, HTTP_ACCESS access, HTTP_USER_ROLE role, USER_AUTH_METHOD type) {
41 web_client_reset_permissions(w);
42 - web_client_flag_set(w, auth & WEB_CLIENT_FLAG_ALL_AUTHS);
43 - w->access = access;
44 - w->user_role = role;
42 + w->user_auth.method = type;
43 + w->user_auth.access = access;
44 + w->user_auth.user_role = role;
45 }
46
47 inline int web_client_permission_denied_acl(struct web_client *w) {
@@ -56,14 +56,14 @@ inline int web_client_permission_denied(struct web_client *w) {
56 w->response.data->content_type = CT_TEXT_PLAIN;
57 buffer_flush(w->response.data);
58
59 - if(w->access & HTTP_ACCESS_SIGNED_ID)
59 + if(w->user_auth.access & HTTP_ACCESS_SIGNED_ID)
60 buffer_strcat(w->response.data,
61 "You don't have enough permissions to access this resource");
62 else
63 buffer_strcat(w->response.data,
64 "You need to be authorized to access this resource");
65
66 - w->response.code = HTTP_ACCESS_PERMISSION_DENIED_HTTP_CODE(w->access);
66 + w->response.code = HTTP_ACCESS_PERMISSION_DENIED_HTTP_CODE(w->user_auth.access);
67 return w->response.code;
68 }
69
@@ -153,9 +153,6 @@ static void web_client_reset_allocations(struct web_client *w, bool free_all) {
153 freez(w->forwarded_host);
154 w->forwarded_host = NULL;
155
156 - freez(w->forwarded_for);
157 - w->forwarded_for = NULL;
158 -
156 freez(w->origin);
157 w->origin = NULL;
158
@@ -189,6 +186,7 @@ static void web_client_reset_allocations(struct web_client *w, bool free_all) {
186
187 memset(w->transaction, 0, sizeof(w->transaction));
188 memset(&w->auth, 0, sizeof(w->auth));
189 + memset(&w->user_auth, 0, sizeof(w->user_auth));
190
191 web_client_reset_permissions(w);
192 web_client_flag_clear(w, WEB_CLIENT_ENCODING_GZIP|WEB_CLIENT_ENCODING_DEFLATE);
@@ -219,13 +217,13 @@ void web_client_log_completed_request(struct web_client *w, bool update_web_stat
217 ND_LOG_FIELD_U64(NDF_RESPONSE_PREPARATION_TIME_USEC, prep_ut),
218 ND_LOG_FIELD_U64(NDF_RESPONSE_SENT_TIME_USEC, sent_ut),
219 ND_LOG_FIELD_U64(NDF_RESPONSE_TOTAL_TIME_USEC, total_ut),
222 - ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
220 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->user_auth.client_ip),
221 ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
224 - ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->forwarded_for),
225 - ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->auth.cloud_account_id),
226 - ND_LOG_FIELD_TXT(NDF_USER_NAME, w->auth.client_name),
227 - ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_role)),
228 - ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->access),
222 + ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->user_auth.forwarded_for),
223 + ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->user_auth.cloud_account_id.uuid),
224 + ND_LOG_FIELD_TXT(NDF_USER_NAME, w->user_auth.client_name),
225 + ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_auth.user_role)),
226 + ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->user_auth.access),
227 ND_LOG_FIELD_END(),
228 };
229 ND_LOG_STACK_PUSH(lgs);
@@ -538,19 +536,19 @@ static inline int check_host_and_call(RRDHOST *host, struct web_client *w, char
536
537 int web_client_api_request(RRDHOST *host, struct web_client *w, char *url_path_fragment) {
538 ND_LOG_STACK lgs[] = {
541 - ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
539 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->user_auth.client_ip),
540 ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
541 ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_HOST, w->forwarded_host),
544 - ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->forwarded_for),
542 + ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->user_auth.forwarded_for),
543 ND_LOG_FIELD_TXT(NDF_NIDL_NODE, w->client_host),
544 ND_LOG_FIELD_TXT(NDF_REQUEST_METHOD, HTTP_REQUEST_MODE_2str(w->mode)),
545 ND_LOG_FIELD_BFR(NDF_REQUEST, w->url_as_received),
546 ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
547 ND_LOG_FIELD_UUID(NDF_TRANSACTION_ID, &w->transaction),
550 - ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->auth.cloud_account_id),
551 - ND_LOG_FIELD_TXT(NDF_USER_NAME, w->auth.client_name),
552 - ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_role)),
553 - ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->access),
548 + ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->user_auth.cloud_account_id.uuid),
549 + ND_LOG_FIELD_TXT(NDF_USER_NAME, w->user_auth.client_name),
550 + ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_auth.user_role)),
551 + ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->user_auth.access),
552 ND_LOG_FIELD_END(),
553 };
554 ND_LOG_STACK_PUSH(lgs);
@@ -560,7 +558,7 @@ int web_client_api_request(RRDHOST *host, struct web_client *w, char *url_path_f
558 query_progress_start_or_update(&w->transaction, 0, w->mode, w->acl,
559 buffer_tostring(w->url_as_received),
560 w->payload,
563 - w->forwarded_for ? w->forwarded_for : w->client_ip);
561 + w->user_auth.forwarded_for[0] ? w->user_auth.forwarded_for : w->user_auth.client_ip);
562 }
563
564 // get the api version
@@ -1060,10 +1058,10 @@ int web_client_api_request_with_node_selection(RRDHOST *host, struct web_client
1058 ND_LOG_FIELD_BFR(NDF_REQUEST, w->url_as_received),
1059 ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
1060 ND_LOG_FIELD_UUID(NDF_TRANSACTION_ID, &w->transaction),
1063 - ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->auth.cloud_account_id),
1064 - ND_LOG_FIELD_TXT(NDF_USER_NAME, w->auth.client_name),
1065 - ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_role)),
1066 - ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->access),
1061 + ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->user_auth.cloud_account_id.uuid),
1062 + ND_LOG_FIELD_TXT(NDF_USER_NAME, w->user_auth.client_name),
1063 + ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_auth.user_role)),
1064 + ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->user_auth.access),
1065 ND_LOG_FIELD_END(),
1066 };
1067 ND_LOG_STACK_PUSH(lgs);
@@ -1281,19 +1279,19 @@ void web_client_process_request_from_web_server(struct web_client *w) {
1279
1280 ND_LOG_STACK lgs[] = {
1281 ND_LOG_FIELD_CB(NDF_SRC_TRANSPORT, web_server_log_transport, w),
1284 - ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
1282 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->user_auth.client_ip),
1283 ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
1284 ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_HOST, w->forwarded_host),
1287 - ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->forwarded_for),
1285 + ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->user_auth.forwarded_for),
1286 ND_LOG_FIELD_TXT(NDF_NIDL_NODE, w->client_host),
1287 ND_LOG_FIELD_TXT(NDF_REQUEST_METHOD, HTTP_REQUEST_MODE_2str(w->mode)),
1288 ND_LOG_FIELD_BFR(NDF_REQUEST, w->url_as_received),
1289 ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
1290 ND_LOG_FIELD_UUID(NDF_TRANSACTION_ID, &w->transaction),
1293 - ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->auth.cloud_account_id),
1294 - ND_LOG_FIELD_TXT(NDF_USER_NAME, w->auth.client_name),
1295 - ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_role)),
1296 - ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->access),
1291 + ND_LOG_FIELD_UUID(NDF_ACCOUNT_ID, &w->user_auth.cloud_account_id.uuid),
1292 + ND_LOG_FIELD_TXT(NDF_USER_NAME, w->user_auth.client_name),
1293 + ND_LOG_FIELD_TXT(NDF_USER_ROLE, http_id2user_role(w->user_auth.user_role)),
1294 + ND_LOG_FIELD_CB(NDF_USER_ACCESS, log_cb_http_access_to_hex, &w->user_auth.access),
1295 ND_LOG_FIELD_END(),
1296 };
1297 ND_LOG_STACK_PUSH(lgs);
@@ -1312,7 +1310,7 @@ void web_client_process_request_from_web_server(struct web_client *w) {
1310 query_progress_start_or_update(&w->transaction, 0, w->mode, w->acl,
1311 buffer_tostring(w->url_as_received),
1312 w->payload,
1315 - w->forwarded_for ? w->forwarded_for : w->client_ip);
1313 + w->user_auth.forwarded_for[0] ? w->user_auth.forwarded_for : w->user_auth.client_ip);
1314 }
1315
1316 // Check if this is a WebSocket upgrade request
src/web/server/web_client.h
+7 -21
@@ -49,7 +49,7 @@ typedef enum __attribute__((packed)) {
49 WEB_CLIENT_FLAG_CONN_CLOUD = (1 << 13), // the client is using Netdata Cloud
50 WEB_CLIENT_FLAG_CONN_WEBRTC = (1 << 14), // the client is using WebRTC
51
52 - // streaming
52 + // streaming and websocket
53 WEB_CLIENT_FLAG_DONT_CLOSE_SOCKET = (1 << 15), // don't close the socket when cleaning up
54
55 // dashboard version
@@ -60,17 +60,12 @@ typedef enum __attribute__((packed)) {
60 WEB_CLIENT_FLAG_PATH_HAS_TRAILING_SLASH = (1 << 20), // the path has a trailing hash
61 WEB_CLIENT_FLAG_PATH_HAS_FILE_EXTENSION = (1 << 21), // the path ends with a filename extension
62
63 - // authorization
64 - WEB_CLIENT_FLAG_AUTH_CLOUD = (1 << 22),
65 - WEB_CLIENT_FLAG_AUTH_BEARER = (1 << 23),
66 - WEB_CLIENT_FLAG_AUTH_GOD = (1 << 24),
67 -
63 // transient settings
69 - WEB_CLIENT_FLAG_PROGRESS_TRACKING = (1 << 25), // flag to avoid redoing progress work
64 + WEB_CLIENT_FLAG_PROGRESS_TRACKING = (1 << 22), // flag to avoid redoing progress work
65
66 // websocket flags
72 - WEB_CLIENT_FLAG_WEBSOCKET_CLIENT = (1 << 26), // this is a websocket client
73 - WEB_CLIENT_FLAG_WEBSOCKET_HANDSHAKE = (1 << 27), // websocket handshake detected
67 + WEB_CLIENT_FLAG_WEBSOCKET_CLIENT = (1 << 23), // this is a websocket client
68 + WEB_CLIENT_FLAG_WEBSOCKET_HANDSHAKE = (1 << 24), // websocket handshake detected
69 } WEB_CLIENT_FLAGS;
70
71 #define WEB_CLIENT_FLAG_PATH_WITH_VERSION (WEB_CLIENT_FLAG_PATH_IS_V0|WEB_CLIENT_FLAG_PATH_IS_V1|WEB_CLIENT_FLAG_PATH_IS_V2|WEB_CLIENT_FLAG_PATH_IS_V3)
@@ -117,10 +112,6 @@ typedef enum __attribute__((packed)) {
112 #define web_client_check_conn_webrtc(w) web_client_flag_check(w, WEB_CLIENT_FLAG_CONN_WEBRTC)
113 #define web_client_flags_clear_conn(w) web_client_flag_clear(w, WEB_CLIENT_FLAG_CONN_TCP | WEB_CLIENT_FLAG_CONN_UNIX | WEB_CLIENT_FLAG_CONN_CLOUD | WEB_CLIENT_FLAG_CONN_WEBRTC)
114
120 -#define WEB_CLIENT_FLAG_ALL_AUTHS (WEB_CLIENT_FLAG_AUTH_CLOUD | WEB_CLIENT_FLAG_AUTH_BEARER | WEB_CLIENT_FLAG_AUTH_GOD)
121 -#define web_client_flags_check_auth(w) web_client_flag_check(w, WEB_CLIENT_FLAG_ALL_AUTHS)
122 -#define web_client_flags_clear_auth(w) web_client_flag_clear(w, WEB_CLIENT_FLAG_ALL_AUTHS)
123 -
115 #define web_client_is_websocket(w) web_client_flag_check(w, WEB_CLIENT_FLAG_WEBSOCKET_CLIENT)
116 #define web_client_set_websocket(w) web_client_flag_set(w, WEB_CLIENT_FLAG_WEBSOCKET_CLIENT)
117 #define web_client_clear_websocket(w) web_client_flag_clear(w, WEB_CLIENT_FLAG_WEBSOCKET_CLIENT)
@@ -130,7 +121,7 @@ typedef enum __attribute__((packed)) {
121 #define web_client_clear_websocket_handshake(w) web_client_flag_clear(w, WEB_CLIENT_FLAG_WEBSOCKET_HANDSHAKE)
122
123 void web_client_reset_permissions(struct web_client *w);
133 -void web_client_set_permissions(struct web_client *w, HTTP_ACCESS access, HTTP_USER_ROLE role, WEB_CLIENT_FLAGS auth);
124 +void web_client_set_permissions(struct web_client *w, HTTP_ACCESS access, HTTP_USER_ROLE role, USER_AUTH_METHOD type);
125
126 void web_client_set_conn_tcp(struct web_client *w);
127 void web_client_set_conn_unix(struct web_client *w);
@@ -147,8 +138,6 @@ void web_client_set_conn_webrtc(struct web_client *w);
138 #define NETDATA_WEB_REQUEST_MAX_SIZE (128 * 1024)
139 #define NETDATA_WEB_DECODED_URL_INITIAL_SIZE 512
140
150 -#define CLOUD_CLIENT_NAME_LENGTH 64
151 -
141 struct response {
142 BUFFER *header; // our response header
143 BUFFER *header_output; // internal use
@@ -178,13 +167,13 @@ struct web_client {
167 HTTP_ACL acl; // the access list of the client
168 HTTP_ACL port_acl; // the operations permitted on the port the client connected to
169 HTTP_ACCESS access; // the access permissions of the client
181 - HTTP_USER_ROLE user_role; // the user role of the client
170 size_t header_parse_tries;
171 size_t header_parse_last_size;
172
173 int fd;
174
187 - char client_ip[INET6_ADDRSTRLEN]; // Defined buffer sizes include null-terminators
175 + USER_AUTH user_auth; // the user authentication data
176 +
177 char client_port[NI_MAXSERV];
178 char client_host[NI_MAXHOST];
179
@@ -196,7 +185,6 @@ struct web_client {
185 char *auth_bearer_token; // the Bearer auth token (if sent)
186 char *server_host; // the Host: header
187 char *forwarded_host; // the X-Forwarded-Host: header
199 - char *forwarded_for; // the X-Forwarded-For: header
188 char *origin; // the Origin: header
189 char *user_agent; // the User-Agent: header
190
@@ -215,8 +203,6 @@ struct web_client {
203
204 struct {
205 nd_uuid_t bearer_token;
218 - nd_uuid_t cloud_account_id;
219 - char client_name[CLOUD_CLIENT_NAME_LENGTH];
206 } auth;
207
208 struct { // A callback to check if the query should be interrupted / stopped
src/web/server/web_server.c
+9 -9
@@ -99,32 +99,32 @@ void web_client_update_acl_matches(struct web_client *w) {
99
100 if(!(w->port_acl & HTTP_ACL_TRANSPORTS_WITHOUT_CLIENT_IP_VALIDATION)) {
101 if (!web_allow_dashboard_from ||
102 - connection_allowed(w->fd, w->client_ip, w->client_host, sizeof(w->client_host),
102 + connection_allowed(w->fd, w->user_auth.client_ip, w->client_host, sizeof(w->client_host),
103 web_allow_dashboard_from, "dashboard", web_allow_dashboard_dns))
104 w->acl |= HTTP_ACL_DASHBOARD;
105
106 if (!web_allow_registry_from ||
107 - connection_allowed(w->fd, w->client_ip, w->client_host, sizeof(w->client_host),
107 + connection_allowed(w->fd, w->user_auth.client_ip, w->client_host, sizeof(w->client_host),
108 web_allow_registry_from, "registry", web_allow_registry_dns))
109 w->acl |= HTTP_ACL_REGISTRY;
110
111 if (!web_allow_badges_from ||
112 - connection_allowed(w->fd, w->client_ip, w->client_host, sizeof(w->client_host),
112 + connection_allowed(w->fd, w->user_auth.client_ip, w->client_host, sizeof(w->client_host),
113 web_allow_badges_from, "badges", web_allow_badges_dns))
114 w->acl |= HTTP_ACL_BADGES;
115
116 if (!web_allow_mgmt_from ||
117 - connection_allowed(w->fd, w->client_ip, w->client_host, sizeof(w->client_host),
117 + connection_allowed(w->fd, w->user_auth.client_ip, w->client_host, sizeof(w->client_host),
118 web_allow_mgmt_from, "management", web_allow_mgmt_dns))
119 w->acl |= HTTP_ACL_MANAGEMENT;
120
121 if (!web_allow_streaming_from ||
122 - connection_allowed(w->fd, w->client_ip, w->client_host, sizeof(w->client_host),
122 + connection_allowed(w->fd, w->user_auth.client_ip, w->client_host, sizeof(w->client_host),
123 web_allow_streaming_from, "streaming", web_allow_streaming_dns))
124 w->acl |= HTTP_ACL_STREAMING;
125
126 if (!web_allow_netdataconf_from ||
127 - connection_allowed(w->fd, w->client_ip, w->client_host, sizeof(w->client_host),
127 + connection_allowed(w->fd, w->user_auth.client_ip, w->client_host, sizeof(w->client_host),
128 web_allow_netdataconf_from, "netdata.conf", web_allow_netdataconf_dns))
129 w->acl |= HTTP_ACL_NETDATACONF;
130 }
@@ -139,13 +139,13 @@ void web_server_log_connection(struct web_client *w, const char *msg) {
139 ND_LOG_STACK lgs[] = {
140 ND_LOG_FIELD_U64(NDF_CONNECTION_ID, w->id),
141 ND_LOG_FIELD_TXT(NDF_SRC_TRANSPORT, SSL_connection(&w->ssl) ? "https" : "http"),
142 - ND_LOG_FIELD_TXT(NDF_SRC_IP, w->client_ip),
142 + ND_LOG_FIELD_TXT(NDF_SRC_IP, w->user_auth.client_ip),
143 ND_LOG_FIELD_TXT(NDF_SRC_PORT, w->client_port),
144 ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_HOST, w->forwarded_host),
145 - ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->forwarded_for),
145 + ND_LOG_FIELD_TXT(NDF_SRC_FORWARDED_FOR, w->user_auth.forwarded_for),
146 ND_LOG_FIELD_END(),
147 };
148 ND_LOG_STACK_PUSH(lgs);
149
150 - nd_log(NDLS_ACCESS, NDLP_DEBUG, "[%s]:%s %s", w->client_ip, w->client_port, msg);
150 + nd_log(NDLS_ACCESS, NDLP_DEBUG, "[%s]:%s %s", w->user_auth.client_ip, w->client_port, msg);
151 }
src/web/websocket/README.md new
+84
@@ -0,0 +1,84 @@
1 +# WebSocket
2 +
3 +Netdata supports WebSocket connections for real-time data streaming and interactive features.
4 +
5 +## WebSocket Protocols
6 +
7 +Netdata supports the following WebSocket protocols:
8 +
9 +- `echo`: A simple echo protocol for testing WebSocket functionality (available only when compiled with internal checks)
10 +- `mcp`: Model-Context Protocol for AI/ML interactions
11 +
12 +These protocols can be specified either in the WebSocket protocol header during the handshake or through the URL path (e.g., `ws://localhost:19999/echo`).
13 +
14 +## Configuration Options
15 +
16 +### Maximum Frame Size
17 +
18 +WebSocket communication is done through frames, with most browsers having limitations on maximum frame size (typically around 16MB).
19 +
20 +#### Global Configuration
21 +
22 +Netdata sets a default outgoing frame size limit of 4MB to ensure browser compatibility while maintaining good performance.
23 +
24 +For larger messages, Netdata automatically fragments them into multiple frames for transmission.
25 +
26 +#### Per-Connection Configuration
27 +
28 +Users can customize the maximum outgoing frame size on a per-connection basis by adding the `max_frame_size` parameter to the WebSocket URL:
29 +
30 +```
31 +ws://localhost:19999/echo?max_frame_size=32768
32 +```
33 +
34 +This allows specific clients to set their preferred frame size, which is especially useful for:
35 +
36 +- Resource-constrained devices that may need smaller frames
37 +- Environments with specific network limitations
38 +- Custom applications with specific buffer size requirements
39 +
40 +The accepted range for `max_frame_size` is between 1KB (1024 bytes) and 20MB. Values outside this range will be automatically adjusted to the nearest bound.
41 +
42 +### WebSocket Compression
43 +
44 +Netdata supports permessage-deflate compression per RFC 7692. Compression is negotiated during the WebSocket handshake and can significantly reduce bandwidth usage for text-based protocols.
45 +
46 +The compression settings can be configured through the standard WebSocket extension negotiation mechanism.
47 +
48 +## Limitations
49 +
50 +- Control frames (ping, pong, close) cannot be compressed and must be less than 125 bytes
51 +- Control frames cannot be fragmented
52 +- The maximum size for decompressed incoming messages is 200MB
53 +- Incoming frames are limited to 20MB
54 +
55 +## Examples
56 +
57 +### Basic Connection
58 +```javascript
59 +// Connect to echo protocol
60 +const ws = new WebSocket('ws://localhost:19999/echo');
61 +
62 +// Event handlers
63 +ws.onopen = () => console.log('Connected');
64 +ws.onmessage = (event) => console.log('Received:', event.data);
65 +ws.onerror = (error) => console.error('Error:', error);
66 +ws.onclose = () => console.log('Disconnected');
67 +
68 +// Send a message
69 +ws.send('Hello, Netdata!');
70 +```
71 +
72 +### Connection with Custom Frame Size
73 +```javascript
74 +// Connect with smaller frame size (32KB)
75 +const ws = new WebSocket('ws://localhost:19999/echo?max_frame_size=32768');
76 +
77 +// Send a large message (will be automatically fragmented)
78 +const largeMessage = new Array(1000000).join('a');
79 +ws.send(largeMessage);
80 +```
81 +
82 +## Debugging
83 +
84 +Netdata logs WebSocket connection details including compression settings and frame size limits at the DEBUG log level. To view these logs, run Netdata with increased verbosity or check the error log file.
src/web/websocket/autobahn-test-suite/config/fuzzingclient.json
+1 -1
@@ -3,7 +3,7 @@
3 "servers": [
4 {
5 "agent": "Netdata WebSocket Server",
6 - "url": "ws://localhost:19999/echo",
6 + "url": "ws://10.20.4.205:19999/echo?max_frame_size=32768",
7 "options": {
8 "version": 18
9 }
src/web/websocket/websocket-compression.c
+97 -2
@@ -2,6 +2,101 @@
2
3 #include "websocket-internal.h"
4
5 +/**
6 + * @brief Compresses a message into the client's c_payload buffer
7 + *
8 + * This function compresses the given data using the client's deflate stream
9 + * and stores the result in the client's c_payload buffer.
10 + *
11 + * @param wsc WebSocket client
12 + * @param data Data to compress
13 + * @param length Length of the data
14 + * @return true if compression was successful, false otherwise
15 + */
16 +bool websocket_client_compress_message(WS_CLIENT *wsc, const char *data, size_t length) {
17 + if (!wsc || !data || !length || !wsc->compression.enabled || !wsc->compression.deflate_stream)
18 + return false;
19 +
20 + if (length < WS_COMPRESS_MIN_SIZE)
21 + return false; // Too small to benefit from compression
22 +
23 + // Clear and prepare the compression buffer
24 + wsb_reset(&wsc->c_payload);
25 +
26 + z_stream *zstrm = wsc->compression.deflate_stream;
27 +
28 + // Calculate maximum possible compressed size
29 + uLong max_compressed_size = deflateBound(zstrm, length) + 4; // +4 for Z_SYNC_FLUSH trailer
30 +
31 + // Ensure the buffer has enough capacity
32 + wsb_need_bytes(&wsc->c_payload, max_compressed_size);
33 +
34 + // Set up the deflate stream
35 + zstrm->next_in = (Bytef *)data;
36 + zstrm->avail_in = length;
37 + zstrm->next_out = (Bytef *)wsb_data(&wsc->c_payload);
38 + zstrm->avail_out = wsb_size(&wsc->c_payload);
39 + zstrm->total_in = 0;
40 + zstrm->total_out = 0;
41 +
42 + // Compress with sync flush
43 + int ret = deflate(zstrm, Z_SYNC_FLUSH);
44 +
45 + bool success = false;
46 + if (ret == Z_STREAM_END || (ret == Z_OK && zstrm->avail_in == 0 && zstrm->avail_out > 0))
47 + success = true;
48 + else if (ret == Z_OK && zstrm->avail_in == 0 && zstrm->avail_out == 0) {
49 + unsigned pending = 0;
50 + int bits = 0;
51 + if(deflatePending(zstrm, &pending, &bits) == Z_OK &&
52 + (pending == 0 && bits == 0))
53 + success = true;
54 + }
55 +
56 + uLong total_out = zstrm->total_out;
57 +
58 + // Reset the stream for future use
59 + if (deflateReset(zstrm) != Z_OK) {
60 + websocket_error(wsc, "Deflate reset failed");
61 +
62 + // Clear pointers for safety
63 + zstrm->next_in = NULL;
64 + zstrm->avail_in = 0;
65 + zstrm->next_out = NULL;
66 + zstrm->avail_out = 0;
67 + zstrm->total_in = 0;
68 + zstrm->total_out = 0;
69 +
70 + return false;
71 + }
72 +
73 + // Clear all pointers for safety
74 + zstrm->next_in = NULL;
75 + zstrm->avail_in = 0;
76 + zstrm->next_out = NULL;
77 + zstrm->avail_out = 0;
78 + zstrm->total_in = 0;
79 + zstrm->total_out = 0;
80 +
81 + if (!success || total_out <= 4) {
82 + // Compression failed or didn't save space
83 + websocket_debug(wsc, "Compression not beneficial (in=%zu, out=%lu) - not using compression",
84 + length, total_out);
85 + return false;
86 + }
87 +
88 + // As per RFC 7692, remove trailing 4 bytes (00 00 FF FF) from Z_SYNC_FLUSH
89 + size_t compressed_size = total_out - 4;
90 +
91 + // Update the buffer's length
92 + wsb_set_length(&wsc->c_payload, compressed_size);
93 +
94 + websocket_debug(wsc, "Compressed message from %zu to %zu bytes (%.1f%%)",
95 + length, compressed_size, (double)compressed_size * 100.0 / (double)length);
96 +
97 + return true;
98 +}
99 +
100 // Initialize compression resources using the parsed options
101 bool websocket_compression_init(WS_CLIENT *wsc) {
102 internal_fatal(wsc->wth->tid != gettid_cached(), "Function %s() should only be used by the websocket thread", __FUNCTION__ );
@@ -202,8 +297,8 @@ bool websocket_client_decompress_message(WS_CLIENT *wsc) {
297
298 // Check if we need more output space
299 if (!success && (ret == Z_BUF_ERROR || ret == Z_OK)) {
205 - wanted_size = MIN(wanted_size * 2, WEBSOCKET_MAX_UNCOMPRESSED_SIZE);
206 - if (wanted_size == WEBSOCKET_MAX_UNCOMPRESSED_SIZE && wanted_size == wsb_size(&wsc->u_payload))
300 + wanted_size = MIN(wanted_size * 2, WS_MAX_DECOMPRESSED_SIZE);
301 + if (wanted_size == WS_MAX_DECOMPRESSED_SIZE && wanted_size == wsb_size(&wsc->u_payload))
302 break; // we cannot resize more
303 }
304 } while (!success && retries-- > 0);
src/web/websocket/websocket-compression.h
+3
@@ -55,4 +55,7 @@ bool websocket_decompression_init(struct websocket_server_client *wsc);
55 void websocket_decompression_cleanup(struct websocket_server_client *wsc);
56 bool websocket_decompression_reset(struct websocket_server_client *wsc);
57
58 +// Compression function for outbound messages
59 +bool websocket_client_compress_message(struct websocket_server_client *wsc, const char *data, size_t length);
60 +
61 #endif // NETDATA_WEBSOCKET_COMPRESSION_H
\ No newline at end of file
src/web/websocket/websocket-echo.c
+2 -1
@@ -22,7 +22,8 @@ void echo_on_message_callback(struct websocket_server_client *wsc, const char *m
22 length);
23
24 // Simply echo back the same message with the same opcode
25 - websocket_protocol_send_frame(wsc, message, length, opcode, true);
25 + // Use send_payload to automatically handle large messages with fragmentation
26 + websocket_protocol_send_payload(wsc, message, length, opcode, true);
27 }
28
29 // Called before sending a close frame to the client
src/web/websocket/websocket-handshake.c
+74 -12
@@ -5,6 +5,7 @@
5 #include "websocket-jsonrpc.h"
6 #include "websocket-echo.h"
7 #include "../mcp/adapters/mcp-websocket.h"
8 +#include "../mcp/mcp-api-key.h"
9
10 // Global array of WebSocket threads
11 WEBSOCKET_THREAD websocket_threads[WEBSOCKET_MAX_THREADS];
@@ -287,6 +288,8 @@ static bool websocket_send_first_response(WS_CLIENT *wsc, const char *accept_key
288
289 // Handle the WebSocket handshake procedure
290 short int websocket_handle_handshake(struct web_client *w) {
291 + web_client_ensure_proper_authorization(w);
292 +
293 if (!websocket_detect_handshake_request(w))
294 return HTTP_RESP_BAD_REQUEST;
295
@@ -299,9 +302,66 @@ short int websocket_handle_handshake(struct web_client *w) {
302 WS_CLIENT *wsc = websocket_client_create();
303
304 // Copy client information
302 - strncpyz(wsc->client_ip, w->client_ip, sizeof(wsc->client_ip));
305 + strncpyz(wsc->client_ip, w->user_auth.client_ip, sizeof(wsc->client_ip));
306 strncpyz(wsc->client_port, w->client_port, sizeof(wsc->client_port));
307
308 + // Copy user authentication and authorization information
309 + wsc->user_auth = w->user_auth;
310 +
311 + // Check for max_frame_size parameter in the URL query string
312 + if (w->url_query_string_decoded && buffer_strlen(w->url_query_string_decoded) > 0) {
313 + const char *query = buffer_tostring(w->url_query_string_decoded);
314 + char *max_frame_size_str = strstr(query, "max_frame_size=");
315 +
316 + if (max_frame_size_str) {
317 + max_frame_size_str += strlen("max_frame_size=");
318 +
319 + char *end_ptr;
320 + size_t max_frame_size = strtoull(max_frame_size_str, &end_ptr, 10);
321 +
322 + // Validate the max frame size with reasonable bounds
323 + if (max_frame_size > 0) {
324 + // Set minimum and maximum limits
325 + if (max_frame_size < 1024) // Minimum 1KB
326 + max_frame_size = 1024;
327 + else if (max_frame_size > (20ULL * 1024 * 1024)) // Maximum 20MB
328 + max_frame_size = 20ULL * 1024 * 1024;
329 +
330 + // Set the client's max outbound frame size
331 + wsc->max_outbound_frame_size = max_frame_size;
332 + websocket_debug(wsc, "Setting custom max outbound frame size: %zu bytes", max_frame_size);
333 + }
334 + }
335 +
336 +#ifdef NETDATA_MCP_DEV_PREVIEW_API_KEY
337 + // Check for api_key parameter for MCP developer preview
338 + char *api_key_str = strstr(query, "api_key=");
339 + if (api_key_str) {
340 + api_key_str += strlen("api_key=");
341 +
342 + // Extract the API key value (until & or end of string)
343 + char api_key_buffer[MCP_DEV_PREVIEW_API_KEY_LENGTH + 1];
344 + size_t i = 0;
345 + while (api_key_str[i] && api_key_str[i] != '&' && i < MCP_DEV_PREVIEW_API_KEY_LENGTH) {
346 + api_key_buffer[i] = api_key_str[i];
347 + i++;
348 + }
349 + api_key_buffer[i] = '\0';
350 +
351 + // Verify the API key
352 + if (mcp_api_key_verify(api_key_buffer)) {
353 + // Override authentication with god mode
354 + wsc->user_auth.access = HTTP_ACCESS_ALL;
355 + wsc->user_auth.method = USER_AUTH_METHOD_GOD;
356 + wsc->user_auth.user_role = HTTP_USER_ROLE_ADMIN;
357 + websocket_debug(wsc, "MCP developer preview API key verified - enabling full access");
358 + } else {
359 + websocket_debug(wsc, "Invalid MCP developer preview API key provided");
360 + }
361 + }
362 +#endif
363 + }
364 +
365 bool url_protocol = false;
366 wsc->protocol = w->websocket.protocol;
367
@@ -357,6 +417,15 @@ short int websocket_handle_handshake(struct web_client *w) {
417
418 // Set up protocol-specific callbacks based on the selected protocol
419 switch (wsc->protocol) {
420 + case WS_PROTOCOL_MCP:
421 + // Set up callbacks for MCP protocol
422 + wsc->on_connect = mcp_websocket_on_connect;
423 + wsc->on_message = mcp_websocket_on_message;
424 + wsc->on_close = mcp_websocket_on_close;
425 + wsc->on_disconnect = mcp_websocket_on_disconnect;
426 + websocket_debug(wsc, "Setting up MCP protocol callbacks");
427 + break;
428 +
429 #ifdef NETDATA_INTERNAL_CHECKS
430 case WS_PROTOCOL_JSONRPC:
431 // Set up callbacks for jsonrpc protocol
@@ -375,15 +444,6 @@ short int websocket_handle_handshake(struct web_client *w) {
444 wsc->on_disconnect = echo_on_disconnect;
445 websocket_debug(wsc, "Setting up echo protocol callbacks");
446 break;
378 -
379 - case WS_PROTOCOL_MCP:
380 - // Set up callbacks for MCP protocol
381 - wsc->on_connect = mcp_websocket_on_connect;
382 - wsc->on_message = mcp_websocket_on_message;
383 - wsc->on_close = mcp_websocket_on_close;
384 - wsc->on_disconnect = mcp_websocket_on_disconnect;
385 - websocket_debug(wsc, "Setting up MCP protocol callbacks");
386 - break;
447 #endif
448
449 default:
@@ -420,7 +480,8 @@ short int websocket_handle_handshake(struct web_client *w) {
480 nd_log(NDLS_DAEMON, NDLP_DEBUG,
481 "WebSocket connection established with %s:%s using protocol: %s (client ID: %u, thread: %zu), "
482 "compression: %s (client context takeover: %s, server context takeover: %s, "
423 - "client window bits: %d, server window bits: %d)",
483 + "client window bits: %d, server window bits: %d), "
484 + "max outbound frame size: %zu bytes",
485 wsc->client_ip, wsc->client_port,
486 WEBSOCKET_PROTOCOL_2str(wsc->protocol),
487 wsc->id, wth->id,
@@ -428,7 +489,8 @@ short int websocket_handle_handshake(struct web_client *w) {
489 wsc->compression.client_context_takeover ? "enabled" : "disabled",
490 wsc->compression.server_context_takeover ? "enabled" : "disabled",
491 wsc->compression.client_max_window_bits,
431 - wsc->compression.server_max_window_bits);
492 + wsc->compression.server_max_window_bits,
493 + wsc->max_outbound_frame_size);
494
495 // Important: This code doesn't actually get sent to the client since we've already
496 // taken over the socket. It's just used by the caller to identify what happened.
src/web/websocket/websocket-internal.h
+14 -10
@@ -39,15 +39,14 @@ struct websocket_thread;
39 #define WS_GUID "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
40
41 // WebSocket frame constants
42 -#define WS_FIN 0x80 // Final frame bit
43 -#define WS_RSV1 0x40 // Reserved bit 1 (used for compression)
44 -#define WS_MASK 0x80 // Mask bit
45 -// Frame size limit - affects fragmentation but not total message size
46 -#define WS_MAX_FRAME_LENGTH (20 * 1024 * 1024) // 20MB max frame size
42 +#define WS_FIN 0x80 // Final frame bit
43 +#define WS_RSV1 0x40 // Reserved bit 1 (used for compression)
44 +#define WS_MASK 0x80 // Mask bit
45
48 -// Total message size limits - these prevent resource exhaustion
49 -#define WEBSOCKET_MAX_COMPRESSED_SIZE (20ULL * 1024 * 1024) // 20MB max compressed message
50 -#define WEBSOCKET_MAX_UNCOMPRESSED_SIZE (200ULL * 1024 * 1024) // 200MB max uncompressed message
46 +// Frame size limits for protection against DoS and browser compatibility
47 +#define WS_MAX_INCOMING_FRAME_SIZE (20ULL * 1024 * 1024) // 20MB max incoming frame size (browsers have ~16MiB)
48 +#define WS_MAX_OUTGOING_FRAME_SIZE (4ULL * 1024 * 1024) // 4MB max outgoing frame size for browser compatibility
49 +#define WS_MAX_DECOMPRESSED_SIZE (200ULL * 1024 * 1024) // 200MB max inbound uncompressed message
50
51 // WebSocket frame header structure - used for processing frame headers
52 typedef struct websocket_frame_header {
@@ -95,6 +94,7 @@ struct websocket_server_client {
94 ND_SOCK sock; // Socket with SSL abstraction
95 uint32_t id; // Unique client ID
96 size_t max_message_size;
97 + size_t max_outbound_frame_size; // Maximum size of outgoing frames for this client
98 time_t connected_t; // Connection timestamp
99 time_t last_activity_t; // Last activity timestamp
100
@@ -108,6 +108,9 @@ struct websocket_server_client {
108 char client_port[NI_MAXSERV];
109 WEBSOCKET_PROTOCOL protocol; // The negotiated subprotocol
110
111 + // Authentication info
112 + USER_AUTH user_auth; // Authentication information copied from web_client
113 +
114 // Thread management
115 struct websocket_thread *wth; // The thread handling this client
116 struct websocket_server_client *prev; // Linked list for thread's client management
@@ -116,6 +119,7 @@ struct websocket_server_client {
119 // Message processing state
120 WS_BUF payload; // Pre-allocated buffer for message data
121 WS_BUF u_payload; // Pre-allocated buffer for uncompressed message data
122 + WS_BUF c_payload; // Pre-allocated buffer for outbound compressed data
123 WEBSOCKET_OPCODE opcode; // Current message opcode
124 bool is_compressed; // Whether the current message is compressed
125 bool message_complete; // Whether the current message is complete
@@ -229,8 +233,8 @@ void websocket_protocol_exception(WS_CLIENT *wsc, WEBSOCKET_CLOSE_CODE reason_co
233 // Protocol receiver functions - websocket-protocol-rcv.c
234 ssize_t websocket_protocol_got_data(WS_CLIENT *wsc, char *data, size_t length);
235
232 -// Protocol sender functions - websocket-protocol-snd.c
233 -int websocket_protocol_send_frame(
236 +// Payload sender - breaks large messages into multiple frames
237 +int websocket_protocol_send_payload(
238 WS_CLIENT *wsc, const char *payload,
239 size_t payload_len, WEBSOCKET_OPCODE opcode, bool use_compression);
240 int websocket_protocol_send_text(WS_CLIENT *wsc, const char *text);
src/web/websocket/websocket-receive.c
+1 -1
@@ -386,7 +386,7 @@ static bool websocket_protocol_validate_header(
386 }
387
388 // Validate payload length against limits
389 - if (payload_length > (uint64_t)WS_MAX_FRAME_LENGTH) {
389 + if (payload_length > (uint64_t)WS_MAX_INCOMING_FRAME_SIZE) {
390 websocket_error(wsc, "Invalid frame: Payload too large (%llu bytes)",
391 (unsigned long long)payload_length);
392 websocket_protocol_exception(wsc, WS_CLOSE_MESSAGE_TOO_BIG, "Frame payload too large");
src/web/websocket/websocket-send.c
+170 -137
@@ -57,16 +57,28 @@ static inline size_t select_header_size(size_t payload_len) {
57 return 10;
58 }
59
60 -// Create and send a WebSocket frame
61 -int websocket_protocol_send_frame(
60 +/**
61 + * @brief Create and send a WebSocket frame
62 + *
63 + * Creates a WebSocket frame with the given payload and sends it.
64 + * This function directly creates a frame with the specified FIN and RSV1 bits.
65 + *
66 + * @param wsc WebSocket client
67 + * @param payload Payload data to send
68 + * @param payload_len Length of the payload
69 + * @param opcode WebSocket opcode (text, binary, continuation, etc.)
70 + * @param compressed Whether the payload is already compressed (RSV1 bit)
71 + * @param final Whether this is the final frame in a message (FIN bit)
72 + * @return Number of bytes sent or -1 on error
73 + */
74 +static int websocket_protocol_send_frame(
75 WS_CLIENT *wsc, const char *payload, size_t payload_len,
63 - WEBSOCKET_OPCODE opcode, bool use_compression) {
76 + WEBSOCKET_OPCODE opcode, bool compressed, bool final) {
77
78 if(!wsc)
79 return -1;
80
81 const char *disconnect_msg = "";
69 - z_stream *zstrm = wsc->compression.deflate_stream;
82
83 if (wsc->sock.fd < 0) {
84 disconnect_msg = "Client not connected";
@@ -79,35 +91,30 @@ int websocket_protocol_send_frame(
91 goto abnormal_disconnect;
92 }
93
82 - // Check if we should actually use compression
83 - bool compress = !websocket_frame_is_control_opcode(opcode) &&
84 - use_compression &&
85 - payload && payload_len &&
86 - wsc->compression.enabled &&
87 - zstrm &&
88 - payload_len >= WS_COMPRESS_MIN_SIZE;
89 -
90 - // Calculate maximum possible compressed size using deflateBound
91 - size_t max_compressed_size = payload_len;
92 - if (compress) {
93 - // Use deflateBound to accurately calculate maximum possible size
94 - max_compressed_size = deflateBound(zstrm, payload_len);
95 -
96 - // Add 4 bytes for Z_SYNC_FLUSH trailer (will be removed later)
97 - max_compressed_size += 4;
98 -
99 - // Ensure the destination can fit the uncompressed data too
100 - max_compressed_size = MAX(payload_len, max_compressed_size);
94 + // Control frames must not be fragmented
95 + if (websocket_frame_is_control_opcode(opcode) && !final) {
96 + disconnect_msg = "Control frames cannot be fragmented";
97 + goto abnormal_disconnect;
98 + }
99 +
100 + // Control frames cannot be compressed
101 + if (websocket_frame_is_control_opcode(opcode) && compressed) {
102 + disconnect_msg = "Control frames cannot be compressed";
103 + goto abnormal_disconnect;
104 }
105 +
106 + // For compressed payloads, we already did the compression outside this function
107 + // so we don't need to compress again, just use the provided payload directly
108 + size_t final_payload_len = payload_len;
109
103 - // Determine header size based on maximum potential size
104 - size_t header_size = select_header_size(max_compressed_size);
110 + // Determine header size based on payload length
111 + size_t header_size = select_header_size(final_payload_len);
112
106 - // Calculate maximum potential frame size
107 - size_t max_frame_size = header_size + max_compressed_size;
113 + // Calculate frame size
114 + size_t frame_size = header_size + final_payload_len;
115
116 // Reserve space in the circular buffer for the entire frame
110 - unsigned char *header_dst = (unsigned char *)cbuffer_reserve_unsafe(&wsc->out_buffer, max_frame_size);
117 + unsigned char *header_dst = (unsigned char *)cbuffer_reserve_unsafe(&wsc->out_buffer, frame_size);
118 if (!header_dst) {
119 disconnect_msg = "Buffer full - too much outgoing data";
120 goto abnormal_disconnect;
@@ -115,89 +122,17 @@ int websocket_protocol_send_frame(
122
123 // The payload will be written directly after the header in our reserved buffer
124 char *payload_dst = (char *)(header_dst + header_size);
118 - size_t final_payload_len = 0;
119 -
120 - if (compress) {
121 - // Setup temporary parameters for the compressor to use
122 - // We'll use the space after our header for the compressed data
123 - zstrm->next_in = (Bytef *)payload;
124 - zstrm->avail_in = payload_len;
125 - zstrm->next_out = (Bytef *)payload_dst;
126 - zstrm->avail_out = max_compressed_size;
127 - zstrm->total_in = 0;
128 - zstrm->total_out = 0;
129 -
130 - // Compress with sync flush to ensure data is flushed
131 - int ret = deflate(zstrm, Z_SYNC_FLUSH);
132 -
133 - bool success = false;
134 - if (ret == Z_STREAM_END || (ret == Z_OK && zstrm->avail_in == 0 && zstrm->avail_out > 0))
135 - success = true;
136 - else if (ret == Z_OK && zstrm->avail_in == 0 && zstrm->avail_out == 0) {
137 - unsigned pending = Z_NULL;
138 - int bits = Z_NULL;
139 - if(deflatePending(zstrm, &pending, &bits) == Z_OK &&
140 - (pending == Z_NULL && bits == Z_NULL))
141 - success = true;
142 - }
143 -
144 - uInt avail_in = zstrm->avail_in;
145 - uInt avail_out = zstrm->avail_out;
146 - uLong total_in = zstrm->total_in;
147 - uLong total_out = zstrm->total_out;
148 -
149 - // we are done - reset the stream to avoid heap-use-after-free issues later
150 - if (deflateReset(zstrm) != Z_OK) {
151 - disconnect_msg = "Deflate reset failed";
152 - goto abnormal_disconnect;
153 - }
125
155 - // Calculate compressed size if successful
156 - if (!success || total_out <= 4) {
157 - // Compression failed
158 - websocket_error(wsc, "Compression failed: %s "
159 - "(ret = %d, avail_in = %u, avail_out = %u, total_in = %lu, total_out = %lu) - "
160 - "sending uncompressed payload",
161 - zError(ret), ret, avail_in, avail_out, total_in, total_out);
162 - compress = false;
163 - }
164 - else {
165 - // As per RFC 7692, remove trailing 4 bytes (00 00 FF FF) from Z_SYNC_FLUSH
166 - final_payload_len = max_compressed_size - avail_out - 4;
167 -
168 - websocket_debug(wsc, "Compressed payload from %zu to %zu bytes (%.1f%%)",
169 - payload_len, final_payload_len,
170 - (double)final_payload_len * 100.0 / (double)payload_len);
171 -
172 - // we may have selected a bigger header size than needed
173 - // so we need to move the payload to the right place
174 - size_t optimal_header_size = select_header_size(final_payload_len);
175 - if(optimal_header_size < header_size) {
176 - char *dst = (char *)header_dst + optimal_header_size;
177 - char *src = payload_dst;
178 - memmove(dst, src, final_payload_len);
179 - payload_dst = dst;
180 - header_size = optimal_header_size;
181 - }
182 - }
183 -
184 - // ensure all pointer values are NULL, so that there is no trace back to this compression
185 - zstrm->next_in = NULL;
186 - zstrm->avail_in = 0;
187 - zstrm->next_out = NULL;
188 - zstrm->avail_out = 0;
189 - zstrm->total_in = 0;
190 - zstrm->total_out = 0;
191 - }
192 -
193 - if(!compress && payload && payload_len > 0) {
126 + // Copy payload data to our buffer
127 + if (payload && payload_len > 0) {
128 memcpy(payload_dst, payload, payload_len);
195 - final_payload_len = payload_len;
129 }
130
131 // Write the header
199 - // First byte: FIN(1) + RSV1(compress) + RSV2(0) + RSV3(0) + OPCODE(4)
200 - header_dst[0] = 0x80 | (compress ? 0x40 : 0) | (opcode & 0x0F);
132 + // First byte: FIN bit, RSV1 bit (compression), and opcode
133 + // Only set FIN bit if this is the final frame
134 + // Only set RSV1 if this frame uses compression
135 + header_dst[0] = (final ? 0x80 : 0) | (compressed ? 0x40 : 0) | (opcode & 0x0F);
136
137 // Write payload length with the appropriate format
138 switch(header_size) {
@@ -242,9 +177,11 @@ int websocket_protocol_send_frame(
177 }
178
179 websocket_debug(wsc,
245 - "TX FRAME: OPCODE=0x%x, FIN=%s, RSV1=%d, RSV2=%d, RSV3=%d, MASK=%s, LEN=%d, "
180 + "TX FRAME: OPCODE=0x%x (%s), FIN=%s, RSV1=%d, RSV2=%d, RSV3=%d, MASK=%s, LEN=%d, "
181 "PAYLOAD_LEN=%zu, HEADER_SIZE=%zu, FRAME_SIZE=%zu, MASK=%02x%02x%02x%02x",
247 - header.opcode, header.fin ? "True" : "False", header.rsv1, header.rsv2, header.rsv3,
182 + header.opcode,
183 + WEBSOCKET_OPCODE_2str(opcode),
184 + header.fin ? "True" : "False", header.rsv1, header.rsv2, header.rsv3,
185 header.mask ? "True" : "False", header.len,
186 header.payload_length, header.header_size, header.frame_size,
187 header.mask_key[0], header.mask_key[1], header.mask_key[2], header.mask_key[3]);
@@ -267,20 +204,115 @@ abnormal_disconnect:
204 // return -1;
205 }
206
270 -// Send a text message
207 +/**
208 + * @brief Send a potentially large payload with automatic fragmentation
209 + *
210 + * This function handles large message fragmentation to ensure browser compatibility.
211 + * It compresses the message if requested and then splits it into fragments
212 + * that are smaller than WS_MAX_OUTGOING_FRAME_SIZE.
213 + *
214 + * @param wsc WebSocket client
215 + * @param payload Payload data to send
216 + * @param payload_len Length of the payload
217 + * @param opcode WebSocket opcode (text, binary)
218 + * @param use_compression Whether to attempt compression
219 + * @return Total number of bytes sent or -1 on error
220 + */
221 +int websocket_protocol_send_payload(
222 + WS_CLIENT *wsc, const char *payload, size_t payload_len,
223 + WEBSOCKET_OPCODE opcode, bool use_compression) {
224 +
225 + if (!wsc || wsc->sock.fd < 0)
226 + return -1;
227 +
228 + // Control frames should never use this function as they can't be fragmented
229 + if (websocket_frame_is_control_opcode(opcode) || !payload || !payload_len) {
230 + return websocket_protocol_send_frame(wsc, payload, payload_len, opcode, false, true);
231 + }
232 +
233 + // Attempt compression if requested and conditions are met
234 + bool compressed = false;
235 + const char *data_to_send = payload;
236 + size_t data_len = payload_len;
237 +
238 + if (use_compression) {
239 + compressed = websocket_client_compress_message(wsc, payload, payload_len);
240 + if (compressed) {
241 + data_to_send = wsb_data(&wsc->c_payload);
242 + data_len = wsb_length(&wsc->c_payload);
243 +
244 + websocket_debug(wsc, "Using compressed payload for transmission (%zu -> %zu bytes)",
245 + payload_len, data_len);
246 + }
247 + }
248 +
249 + // Check if we need to fragment the message
250 + if (data_len <= wsc->max_outbound_frame_size) {
251 + // Small enough to send in a single frame
252 + return websocket_protocol_send_frame(wsc, data_to_send, data_len, opcode, compressed, true);
253 + }
254 +
255 + // We need to fragment the message
256 + websocket_debug(wsc, "Fragmenting large message (%zu bytes) into frames of max %zu bytes",
257 + data_len, wsc->max_outbound_frame_size);
258 +
259 + size_t total_sent = 0;
260 + size_t bytes_remaining = data_len;
261 + size_t offset = 0;
262 + bool first_frame = true;
263 +
264 + // Send fragments
265 + while (bytes_remaining > 0) {
266 + // Determine size for this fragment
267 + size_t fragment_size = MIN(bytes_remaining, wsc->max_outbound_frame_size);
268 + bool is_final = (fragment_size == bytes_remaining);
269 +
270 + // First frame uses original opcode, subsequent frames use continuation
271 + WEBSOCKET_OPCODE frame_opcode = first_frame ? opcode : WS_OPCODE_CONTINUATION;
272 +
273 + // Compression flag is only set on the first frame
274 + bool frame_compressed = compressed && first_frame;
275 +
276 + // Send this fragment
277 + int result = websocket_protocol_send_frame(
278 + wsc,
279 + data_to_send + offset,
280 + fragment_size,
281 + frame_opcode,
282 + frame_compressed,
283 + is_final
284 + );
285 +
286 + if (result < 0) {
287 + // Failed to send frame
288 + websocket_error(wsc, "Failed to send message fragment at offset %zu", offset);
289 + return -1;
290 + }
291 +
292 + // Update counters for next iteration
293 + total_sent += result;
294 + offset += fragment_size;
295 + bytes_remaining -= fragment_size;
296 + first_frame = false;
297 + }
298 +
299 + websocket_debug(wsc, "Successfully sent fragmented message in multiple frames, total bytes: %zu", total_sent);
300 + return total_sent;
301 +}
302 +
303 +/**
304 + * @brief Send a text message with automatic fragmentation
305 + *
306 + * This function sends a WebSocket text message with automatic compression and fragmentation.
307 + *
308 + * @param wsc WebSocket client
309 + * @param text Text to send
310 + * @return Number of bytes sent or -1 on error
311 + */
312 int websocket_protocol_send_text(WS_CLIENT *wsc, const char *text) {
313 if (!wsc)
314 return -1;
315
275 - // Special handling for null or empty text message
276 - if (!text || text[0] == '\0') {
277 - websocket_debug(wsc, "Sending empty text message");
278 -
279 - // Use an empty buffer for zero-length text messages
280 - static const char empty_data[1] = {0};
281 - return websocket_protocol_send_frame(wsc, empty_data, 0, WS_OPCODE_TEXT, false);
282 - }
283 -
316 size_t text_len = strlen(text);
317
318 websocket_debug(wsc, "Sending text message, length=%zu", text_len);
@@ -288,30 +320,31 @@ int websocket_protocol_send_text(WS_CLIENT *wsc, const char *text) {
320 // Dump text message for debugging
321 websocket_dump_debug(wsc, text, text_len, "TX TEXT MSG");
322
291 - // Enable compression for text messages by default
292 - return websocket_protocol_send_frame(wsc, text, text_len, WS_OPCODE_TEXT, true);
323 + // Enable compression for text messages by default, with automatic fragmentation
324 + return websocket_protocol_send_payload(wsc, text, text_len, WS_OPCODE_TEXT, true);
325 }
326
295 -// Send a binary message
327 +/**
328 + * @brief Send a binary message with automatic fragmentation
329 + *
330 + * This function sends a WebSocket binary message with automatic compression and fragmentation.
331 + *
332 + * @param wsc WebSocket client
333 + * @param data Binary data to send
334 + * @param length Length of the binary data
335 + * @return Number of bytes sent or -1 on error
336 + */
337 int websocket_protocol_send_binary(WS_CLIENT *wsc, const void *data, size_t length) {
338 if (!wsc)
339 return -1;
340
300 - // Special handling for empty binary message
301 - if (!data || length == 0) {
302 - websocket_debug(wsc, "Sending empty binary message");
303 -
304 - // Use an empty buffer for zero-length binary messages
305 - static const char empty_data[1] = {0};
306 - return websocket_protocol_send_frame(wsc, empty_data, 0, WS_OPCODE_BINARY, false);
307 - }
308 -
341 websocket_debug(wsc, "Sending binary message, length=%zu", length);
342
343 // Dump binary message for debugging
344 websocket_dump_debug(wsc, data, length, "TX BIN MSG");
345
314 - return websocket_protocol_send_frame(wsc, data, length, WS_OPCODE_BINARY, true);
346 + // Enable compression for binary messages by default, with automatic fragmentation
347 + return websocket_protocol_send_payload(wsc, data, length, WS_OPCODE_BINARY, true);
348 }
349
350 // Send a close frame
@@ -360,8 +393,8 @@ int websocket_protocol_send_close(WS_CLIENT *wsc, WEBSOCKET_CLOSE_CODE code, con
393 if(wsc->on_close)
394 wsc->on_close(wsc, WS_CLOSE_GOING_AWAY, reason);
395
363 - // Send close frame (never compressed)
364 - int result = websocket_protocol_send_frame(wsc, payload, payload_len, WS_OPCODE_CLOSE, false);
396 + // Send close frame (never compressed, always final)
397 + int result = websocket_protocol_send_frame(wsc, payload, payload_len, WS_OPCODE_CLOSE, false, true);
398
399 return result;
400 }
@@ -384,8 +417,8 @@ int websocket_protocol_send_ping(WS_CLIENT *wsc, const char *data, size_t length
417 length = 0;
418 }
419
387 - // Send ping frame (never compressed)
388 - return websocket_protocol_send_frame(wsc, data, length, WS_OPCODE_PING, false);
420 + // Send ping frame (never compressed, always final)
421 + return websocket_protocol_send_frame(wsc, data, length, WS_OPCODE_PING, false, true);
422 }
423
424 // Send a pong frame
@@ -406,6 +439,6 @@ int websocket_protocol_send_pong(WS_CLIENT *wsc, const char *data, size_t length
439 length = 0;
440 }
441
409 - // Send pong frame (never compressed)
410 - return websocket_protocol_send_frame(wsc, data, length, WS_OPCODE_PONG, false);
442 + // Send pong frame (never compressed, always final)
443 + return websocket_protocol_send_frame(wsc, data, length, WS_OPCODE_PONG, false, true);
444 }
src/web/websocket/websocket-thread.c
+17 -6
@@ -36,7 +36,7 @@ static bool websocket_thread_add_client(WEBSOCKET_THREAD *wth, WS_CLIENT *wsc) {
36 return true;
37 }
38
39 -static void websocket_thread_remove_client(WEBSOCKET_THREAD *wth, WS_CLIENT *wsc) {
39 +static void websocket_thread_remove_client(WEBSOCKET_THREAD *wth, WS_CLIENT *wsc, bool have_lock) {
40 internal_fatal(wth->tid != gettid_cached(), "Function %s() should only be used by the websocket thread", __FUNCTION__ );
41
42 // Notify the protocol handler that the client is being disconnected
@@ -64,13 +64,15 @@ static void websocket_thread_remove_client(WEBSOCKET_THREAD *wth, WS_CLIENT *wsc
64 DOUBLE_LINKED_LIST_REMOVE_ITEM_UNSAFE(wth->clients, wsc, prev, next);
65
66 // Lock the thread clients
67 - spinlock_lock(&wth->clients_spinlock);
67 + if(!have_lock)
68 + spinlock_lock(&wth->clients_spinlock);
69
70 if(wth->clients_current > 0)
71 wth->clients_current--;
72
73 // Release the thread clients lock
73 - spinlock_unlock(&wth->clients_spinlock);
74 + if(!have_lock)
75 + spinlock_unlock(&wth->clients_spinlock);
76
77 websocket_debug(wsc, "Removed and resources freed", wth->id, wsc->id);
78 websocket_client_free(wsc);
@@ -265,7 +267,7 @@ static void websocket_thread_process_commands(WEBSOCKET_THREAD *wth) {
267 continue;
268 }
269
268 - websocket_thread_remove_client(wth, wsc);
270 + websocket_thread_remove_client(wth, wsc, false);
271 break;
272 }
273
@@ -382,6 +384,15 @@ void *websocket_thread(void *ptr) {
384 continue;
385 }
386
387 + // Push client connection info to log stack for all subsequent logs
388 + ND_LOG_STACK lgs[] = {
389 + ND_LOG_FIELD_U64(NDF_CONNECTION_ID, wsc->id),
390 + ND_LOG_FIELD_TXT(NDF_SRC_IP, wsc->client_ip),
391 + ND_LOG_FIELD_TXT(NDF_SRC_PORT, wsc->client_port),
392 + ND_LOG_FIELD_END(),
393 + };
394 + ND_LOG_STACK_PUSH(lgs);
395 +
396 // Check for errors
397 if(ev.events & ND_POLL_HUP) {
398 websocket_thread_client_socket_error(wth, wsc, "Client hangup");
@@ -410,7 +421,7 @@ void *websocket_thread(void *ptr) {
421 // Check if this client is waiting to be closed after flushing outgoing data
422 if(wsc->flush_and_remove_client && cbuffer_used_size_unsafe(&wsc->out_buffer) == 0) {
423 // All data flushed - remove client
413 - websocket_thread_remove_client(wth, wsc);
424 + websocket_thread_remove_client(wth, wsc, false);
425 }
426 }
427 }
@@ -478,7 +489,7 @@ void *websocket_thread(void *ptr) {
489
490 websocket_protocol_send_close(wsc, WS_CLOSE_GOING_AWAY, "Server shutting down");
491 websocket_write_data(wsc);
481 - websocket_thread_remove_client(wth, wsc);
492 + websocket_thread_remove_client(wth, wsc, true);
493
494 wsc = next;
495 }
src/web/websocket/websocket.c
+14 -16
@@ -86,7 +86,7 @@ static struct websocket_server ws_server = (struct websocket_server){
86
87 // Initialize WebSocket subsystem
88 void websocket_initialize(void) {
89 - debug_flags |= D_WEBSOCKET;
89 + // debug_flags |= D_WEBSOCKET;
90
91 // Initialize thread system
92 websocket_threads_init();
@@ -110,6 +110,7 @@ WS_CLIENT *websocket_client_create(void) {
110
111 wsc->connected_t = now_realtime_sec();
112 wsc->last_activity_t = wsc->connected_t;
113 + wsc->max_outbound_frame_size = WS_MAX_OUTGOING_FRAME_SIZE; // Default value
114
115 // initialize callbacks to NULL
116 wsc->on_connect = NULL;
@@ -130,6 +131,9 @@ WS_CLIENT *websocket_client_create(void) {
131 // Initialize uncompressed buffer with a larger size since decompressed data can expand
132 // For compressed content, the expanded data can be much larger than the input
133 wsb_init(&wsc->u_payload, WEBSOCKET_UNPACKED_INITIAL_SIZE);
134 +
135 + // Initialize compressed output buffer for outbound messages
136 + wsb_init(&wsc->c_payload, WEBSOCKET_PAYLOAD_INITIAL_SIZE);
137
138 // Set the initial message state
139 wsc->opcode = WS_OPCODE_TEXT; // Default opcode
@@ -162,9 +166,10 @@ void websocket_client_free(WS_CLIENT *wsc) {
166 cbuffer_cleanup(&wsc->in_buffer);
167 cbuffer_cleanup(&wsc->out_buffer);
168
165 - // Cleanup pre-allocated message and uncompressed buffers
169 + // Cleanup pre-allocated message, uncompressed, and compressed buffers
170 wsb_cleanup(&wsc->payload);
171 wsb_cleanup(&wsc->u_payload);
172 + wsb_cleanup(&wsc->c_payload);
173
174 // Clean up compression resources if needed
175 websocket_compression_cleanup(wsc);
@@ -247,18 +252,11 @@ int websocket_broadcast_message(const char *message, WEBSOCKET_OPCODE opcode) {
252 int websocket_send_message(WS_CLIENT *wsc, const char *message, size_t length, WEBSOCKET_OPCODE opcode) {
253 if (!wsc || !message || wsc->state != WS_STATE_OPEN)
254 return -1;
250 -
251 - // Use the appropriate protocol function based on opcode
252 - if (opcode == WS_OPCODE_TEXT) {
253 - return websocket_protocol_send_text(wsc, message);
254 - } else if (opcode == WS_OPCODE_BINARY) {
255 - return websocket_protocol_send_binary(wsc, message, length);
256 - } else {
257 - // For other opcodes, use the generic frame sender
258 - bool use_compression = wsc->compression.enabled &&
259 - !websocket_frame_is_control_opcode(opcode) &&
260 - length >= WS_COMPRESS_MIN_SIZE;
261 -
262 - return websocket_protocol_send_frame(wsc, message, length, opcode, use_compression);
263 - }
255 +
256 + // For other opcodes, use the generic frame sender
257 + bool use_compression = wsc->compression.enabled &&
258 + !websocket_frame_is_control_opcode(opcode) &&
259 + length >= WS_COMPRESS_MIN_SIZE;
260 +
261 + return websocket_protocol_send_payload(wsc, message, length, opcode, use_compression);
262 }