DB error attribution (#21645)
Co-authored-by: Ilya Mashchenko <ilya@netdata.cloud>
Costa Tsaousis committed
Jan 27, 2026 at 08:44 UTC
9e856e3787d40b9e2c7adc779bd70c5b7dab3690
35 files changed
+7617
-58
docs/.map/map.csv
+1
@@ -156,6 +156,7 @@ https://github.com/netdata/netdata/edit/master/docs/logs/active_journal_centrali
156
,,,,,
157
https://github.com/netdata/netdata/edit/master/docs/top-monitoring-netdata-functions.md,Top Consumers,Published,Top Consumers,,Present the Netdata Functions what these are and why they should be used.
158
https://github.com/netdata/netdata/edit/master/docs/functions/processes.md,Processes,Published,Top Consumers,,
159
+https://github.com/netdata/netdata/edit/master/docs/functions/databases.md,Database Queries,Published,Top Consumers,,Top and running database queries with deadlock and error attribution for 13 databases.
160
,,,,,
161
https://github.com/netdata/netdata/edit/master/src/health/README.md,Alerts & Notifications,Published,Alerts & Notifications,,
162
https://github.com/netdata/netdata/edit/master/docs/alerts-and-notifications/creating-alerts-with-netdata-alerts-configuration-manager.md,Creating Alerts with the Alerts Configuration Manager,Published,Alerts & Notifications,,
docs/functions/databases.md
new
+134
@@ -0,0 +1,134 @@
1
+# Database Query Functions
2
+
3
+## Overview
4
+
5
+Database query functions provide deep visibility into SQL and NoSQL database performance through Netdata's Top tab. They help you identify problematic queries, detect deadlocks, and understand error patterns—all from the Netdata dashboard.
6
+
7
+| Capability | Description |
8
+|------------|-------------|
9
+| **Top Queries** | Identify the most expensive queries by execution time, I/O, rows processed, or other metrics |
10
+| **Running Queries** | See currently executing queries in real-time |
11
+| **Deadlock Detection** | View the latest detected deadlock with full transaction details |
12
+| **Error Attribution** | Correlate SQL errors with the queries that caused them |
13
+
14
+## Supported Databases
15
+
16
+| Database | Top Queries | Running Queries | Deadlock Info | Error Info | Integration Docs |
17
+|----------|:-----------:|:---------------:|:-------------:|:----------:|------------------|
18
+| ClickHouse | ✅ | - | - | - | [ClickHouse](/src/go/plugin/go.d/collector/clickhouse/integrations/clickhouse.md) |
19
+| CockroachDB | ✅ | ✅ | - | - | [CockroachDB](/src/go/plugin/go.d/collector/cockroachdb/integrations/cockroachdb.md) |
20
+| Couchbase | ✅ | - | - | - | [Couchbase](/src/go/plugin/go.d/collector/couchbase/integrations/couchbase.md) |
21
+| Elasticsearch | ✅ | - | - | - | [Elasticsearch](/src/go/plugin/go.d/collector/elasticsearch/integrations/elasticsearch.md) |
22
+| MongoDB | ✅ | - | - | - | [MongoDB](/src/go/plugin/go.d/collector/mongodb/integrations/mongodb.md) |
23
+| Microsoft SQL Server | ✅ | - | ✅ | ✅* | [MSSQL](/src/go/plugin/go.d/collector/mssql/integrations/microsoft_sql_server.md) |
24
+| MySQL | ✅ | - | ✅ | ✅* | [MySQL](/src/go/plugin/go.d/collector/mysql/integrations/mysql.md) |
25
+| MariaDB | ✅ | - | ✅ | ✅* | [MariaDB](/src/go/plugin/go.d/collector/mysql/integrations/mariadb.md) |
26
+| Percona Server | ✅ | - | ✅ | ✅* | [Percona](/src/go/plugin/go.d/collector/mysql/integrations/percona_mysql.md) |
27
+| Oracle Database | ✅ | ✅ | - | - | [Oracle](/src/go/plugin/go.d/collector/oracledb/integrations/oracle_db.md) |
28
+| PostgreSQL | ✅ | - | - | - | [PostgreSQL](/src/go/plugin/go.d/collector/postgres/integrations/postgresql.md) |
29
+| ProxySQL | ✅ | - | - | - | [ProxySQL](/src/go/plugin/go.d/collector/proxysql/integrations/proxysql.md) |
30
+| Redis | ✅ | - | - | - | [Redis](/src/go/plugin/go.d/collector/redis/integrations/redis.md) |
31
+| RethinkDB | - | ✅ | - | - | [RethinkDB](/src/go/plugin/go.d/collector/rethinkdb/integrations/rethinkdb.md) |
32
+| YugabyteDB | ✅ | ✅ | - | - | [YugabyteDB](/src/go/plugin/go.d/collector/yugabytedb/integrations/yugabytedb.md) |
33
+
34
+*\* Error Info is integrated directly into Top Queries results—each query row shows its associated errors.*
35
+
36
+## Function Types
37
+
38
+### Top Queries
39
+
40
+Retrieves **accumulated query statistics** over a time window. These are aggregated metrics (total calls, total time, average time, etc.) from the database's query statistics infrastructure—not real-time snapshots.
41
+
42
+**Filter options** vary by database but typically include:
43
+- Execution time (total, average)
44
+- Call count
45
+- Rows processed (read, written, returned)
46
+- I/O metrics (logical reads, physical reads)
47
+- Resource usage (CPU, memory, locks)
48
+
49
+The number of queries returned is configurable (default: 500). This is a two-stage process:
50
+1. **Server-side**: Database returns the top N queries ranked by your chosen metric
51
+2. **Client-side**: UI can further sort, filter, and explore the returned data
52
+
53
+### Running Queries
54
+
55
+Shows **currently executing queries** at the moment of request. Essential for diagnosing stuck queries, long-running transactions, or unexpected load.
56
+
57
+**Supported**: CockroachDB, Oracle, RethinkDB, YugabyteDB
58
+
59
+### Deadlock Info
60
+
61
+Displays the **most recently detected deadlock**—not a historical list. When a new deadlock occurs, it replaces the previous one.
62
+
63
+**Supported**: MySQL/MariaDB/Percona, Microsoft SQL Server
64
+
65
+Information provided:
66
+- Deadlock timestamp and ID
67
+- Participating transactions
68
+- Victim transaction (rolled back)
69
+- Query text and lock details
70
+- Wait resource
71
+
72
+### Error Info
73
+
74
+Shows **recent SQL errors** from the database's error history. Error attribution is embedded directly in Top Queries results—each query row includes error details when available.
75
+
76
+**Supported**: MySQL/MariaDB/Percona, Microsoft SQL Server
77
+
78
+Attribution status values:
79
+- `enabled` — Error details available for this query
80
+- `no_data` — No recent errors for this query
81
+- `not_enabled` — Error tracking not configured
82
+- `not_supported` — Database version lacks required features
83
+
84
+## Security Considerations
85
+
86
+### Query Text Exposure
87
+
88
+Some databases normalize queries (replacing literals with placeholders), while others show actual values that may contain sensitive data:
89
+
90
+| Database | Query Text |
91
+|----------|:----------:|
92
+| ClickHouse | Normalized |
93
+| CockroachDB | ⚠️ Raw |
94
+| Couchbase | ⚠️ Raw |
95
+| Elasticsearch | ⚠️ Raw |
96
+| MongoDB | ⚠️ Raw |
97
+| Microsoft SQL Server | ⚠️ Raw |
98
+| MySQL/MariaDB/Percona | Normalized (Top Queries), ⚠️ Raw (Deadlock) |
99
+| Oracle | ⚠️ Raw |
100
+| PostgreSQL | Normalized |
101
+| ProxySQL | Normalized |
102
+| Redis | ⚠️ Raw |
103
+| RethinkDB | ⚠️ Raw |
104
+| YugabyteDB | ⚠️ Raw |
105
+
106
+**Legend**:
107
+- **Normalized**: Literals replaced with placeholders (`SELECT * FROM users WHERE id = ?`)
108
+- **⚠️ Raw**: May contain actual values (`SELECT * FROM users WHERE id = 12345`)
109
+
110
+:::caution
111
+
112
+Error messages may contain sensitive values regardless of query normalization. Ensure appropriate access controls are in place.
113
+
114
+:::
115
+
116
+### Recommendations
117
+
118
+1. **Disable unneeded functions** — Each integration supports configuration options to disable specific functions
119
+2. **Use Netdata Cloud access controls** — Assign users to appropriate Rooms and Roles
120
+3. **Use dedicated database users** — Grant only the permissions required for monitoring
121
+
122
+## Getting Started
123
+
124
+Each database requires specific setup. Click the integration link in the table above for:
125
+
126
+- Prerequisites and permissions required
127
+- Configuration options
128
+- Available metrics and columns
129
+- Database-specific notes
130
+
131
+## Related Documentation
132
+
133
+- [Top Consumers Overview](/docs/top-monitoring-netdata-functions.md)
134
+- [Processes Function](/docs/functions/processes.md)
src/go/plugin/go.d/collector/mssql/collector.go
+45
@@ -7,6 +7,7 @@ import (
7
"database/sql"
8
_ "embed"
9
"errors"
10
+ "strings"
11
"sync"
12
"time"
13
@@ -71,6 +72,26 @@ type Config struct {
72
// Default: true - MSSQL Query Store may contain unmasked PII in query text
73
QueryStoreFunctionEnabled *bool `yaml:"query_store_function_enabled,omitempty" json:"query_store_function_enabled"`
74
75
+ // DeadlockInfoFunctionEnabled controls whether the deadlock-info function is available
76
+ // Uses pointer to distinguish "unset" from explicit "false":
77
+ // - nil (unset): Apply default of true (enabled)
78
+ // - false: Explicitly disabled
79
+ // - true: Explicitly enabled
80
+ // Default: true
81
+ DeadlockInfoFunctionEnabled *bool `yaml:"deadlock_info_function_enabled,omitempty" json:"deadlock_info_function_enabled"`
82
+
83
+ // ErrorInfoFunctionEnabled controls whether the error-info function is available
84
+ // Uses pointer to distinguish "unset" from explicit "false":
85
+ // - nil (unset): Apply default of true (enabled)
86
+ // - false: Explicitly disabled
87
+ // - true: Explicitly enabled
88
+ // Default: true
89
+ ErrorInfoFunctionEnabled *bool `yaml:"error_info_function_enabled,omitempty" json:"error_info_function_enabled"`
90
+
91
+ // ErrorInfoSessionName sets the Extended Events session name for error-info
92
+ // Default: "netdata_errors"
93
+ ErrorInfoSessionName string `yaml:"error_info_session_name,omitempty" json:"error_info_session_name,omitempty"`
94
+
95
// TopQueriesLimit is the maximum number of queries to return
96
TopQueriesLimit int `yaml:"top_queries_limit,omitempty" json:"top_queries_limit,omitempty"`
97
}
@@ -91,6 +112,30 @@ func (c *Config) GetQueryStoreFunctionEnabled() bool {
112
return *c.QueryStoreFunctionEnabled
113
}
114
115
+// GetDeadlockInfoFunctionEnabled returns whether the deadlock-info function is enabled (default: true)
116
+func (c *Config) GetDeadlockInfoFunctionEnabled() bool {
117
+ if c.DeadlockInfoFunctionEnabled == nil {
118
+ return true
119
+ }
120
+ return *c.DeadlockInfoFunctionEnabled
121
+}
122
+
123
+// GetErrorInfoFunctionEnabled returns whether the error-info function is enabled (default: true)
124
+func (c *Config) GetErrorInfoFunctionEnabled() bool {
125
+ if c.ErrorInfoFunctionEnabled == nil {
126
+ return true
127
+ }
128
+ return *c.ErrorInfoFunctionEnabled
129
+}
130
+
131
+// GetErrorInfoSessionName returns the Extended Events session name for error-info.
132
+func (c *Config) GetErrorInfoSessionName() string {
133
+ if strings.TrimSpace(c.ErrorInfoSessionName) == "" {
134
+ return "netdata_errors"
135
+ }
136
+ return c.ErrorInfoSessionName
137
+}
138
+
139
type Collector struct {
140
module.Base
141
Config `yaml:",inline" json:""`
src/go/plugin/go.d/collector/mssql/config_schema.json
+29
-1
@@ -50,6 +50,24 @@
50
"minimum": 1,
51
"maximum": 5000,
52
"default": 500
53
+ },
54
+ "deadlock_info_function_enabled": {
55
+ "title": "Enable Deadlock Info Function",
56
+ "description": "Enable the deadlock-info function. WARNING: query text may contain unmasked sensitive literals (PII). This function reads deadlock graphs from the system_health session and requires VIEW SERVER STATE. Grant with: GRANT VIEW SERVER STATE TO [netdata_user];",
57
+ "type": "boolean",
58
+ "default": true
59
+ },
60
+ "error_info_function_enabled": {
61
+ "title": "Enable Error Info Function",
62
+ "description": "Enable the error-info function. WARNING: error messages and query text may contain unmasked sensitive literals (PII). This function reads from a user-managed Extended Events session and requires VIEW SERVER STATE.",
63
+ "type": "boolean",
64
+ "default": true
65
+ },
66
+ "error_info_session_name": {
67
+ "title": "Error Info Session Name",
68
+ "description": "Name of the Extended Events session that captures error_reported events for error-info. The session must be created by an administrator and include a ring_buffer target.",
69
+ "type": "string",
70
+ "default": "netdata_errors"
71
}
72
},
73
"required": [
@@ -79,6 +97,15 @@
97
"query_store_time_window_days": {
98
"ui:help": "Limits Query Store data to recent days. Lower values improve performance on busy servers."
99
},
100
+ "deadlock_info_function_enabled": {
101
+ "ui:help": "When enabled, the deadlock-info function becomes available in the Netdata dashboard. WARNING: query text may contain unmasked sensitive literals and requires VIEW SERVER STATE permission."
102
+ },
103
+ "error_info_function_enabled": {
104
+ "ui:help": "When enabled, the error-info function becomes available in the Netdata dashboard. WARNING: error messages and query text may include sensitive literals."
105
+ },
106
+ "error_info_session_name": {
107
+ "ui:help": "The Extended Events session must be created by an administrator and include a ring_buffer target capturing sqlserver.error_reported with sql_text action."
108
+ },
109
"ui:flavour": "tabs",
110
"ui:options": {
111
"tabs": [
@@ -88,7 +115,8 @@
115
"update_every",
116
"dsn",
117
"timeout",
91
- "vnode"
118
+ "vnode",
119
+ "deadlock_info_function_enabled"
120
]
121
},
122
{
src/go/plugin/go.d/collector/mssql/deadlock_info.go
new
+717
@@ -0,0 +1,717 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+package mssql
4
+
5
+import (
6
+ "context"
7
+ "database/sql"
8
+ "encoding/xml"
9
+ "errors"
10
+ "fmt"
11
+ "sort"
12
+ "strconv"
13
+ "strings"
14
+ "time"
15
+
16
+ mssqlDriver "github.com/microsoft/go-mssqldb"
17
+
18
+ "github.com/netdata/netdata/go/plugins/pkg/funcapi"
19
+ "github.com/netdata/netdata/go/plugins/plugin/go.d/pkg/strmutil"
20
+)
21
+
22
+const (
23
+ deadlockInfoHelp = "Latest deadlock from the system_health Extended Events ring buffer. WARNING: query text may include unmasked sensitive literals; restrict dashboard access."
24
+ deadlockParseErrorStatus = 561
25
+)
26
+
27
+const deadlockInfoMethodID = "deadlock-info"
28
+
29
+func deadlockInfoMethodConfig() funcapi.MethodConfig {
30
+ return funcapi.MethodConfig{
31
+ ID: deadlockInfoMethodID,
32
+ Name: "Deadlock Info",
33
+ UpdateEvery: 10,
34
+ Help: deadlockInfoHelp,
35
+ RequireCloud: true,
36
+ RequiredParams: []funcapi.ParamConfig{},
37
+ }
38
+}
39
+
40
+// funcDeadlockInfo handles the deadlock-info function.
41
+type funcDeadlockInfo struct {
42
+ router *funcRouter
43
+}
44
+
45
+func newFuncDeadlockInfo(r *funcRouter) *funcDeadlockInfo {
46
+ return &funcDeadlockInfo{router: r}
47
+}
48
+
49
+// Compile-time interface check.
50
+var _ funcapi.MethodHandler = (*funcDeadlockInfo)(nil)
51
+
52
+func (f *funcDeadlockInfo) MethodParams(ctx context.Context, method string) ([]funcapi.ParamConfig, error) {
53
+ if !f.router.collector.Config.GetDeadlockInfoFunctionEnabled() {
54
+ return nil, fmt.Errorf("deadlock-info function disabled in configuration")
55
+ }
56
+ return []funcapi.ParamConfig{}, nil
57
+}
58
+
59
+func (f *funcDeadlockInfo) Handle(ctx context.Context, method string, params funcapi.ResolvedParams) *funcapi.FunctionResponse {
60
+ if f.router.collector.db == nil {
61
+ db, err := f.router.collector.openConnection()
62
+ if err != nil {
63
+ return funcapi.UnavailableResponse("collector is still initializing, please retry in a few seconds")
64
+ }
65
+ f.router.collector.db = db
66
+ }
67
+ return f.router.collector.collectDeadlockInfo(ctx)
68
+}
69
+
70
+func (f *funcDeadlockInfo) Cleanup(ctx context.Context) {}
71
+
72
+const (
73
+ deadlockIdxRowID = iota
74
+ deadlockIdxDeadlockID
75
+ deadlockIdxTimestamp
76
+ deadlockIdxProcessID
77
+ deadlockIdxSpid
78
+ deadlockIdxEcid
79
+ deadlockIdxIsVictim
80
+ deadlockIdxQueryText
81
+ deadlockIdxLockMode
82
+ deadlockIdxLockStatus
83
+ deadlockIdxWaitResource
84
+ deadlockIdxDatabase
85
+ deadlockColumnCount
86
+)
87
+
88
+type mssqlDeadlockTxn struct {
89
+ processID string
90
+ spid string
91
+ ecid string
92
+ dbid string
93
+ queryText string
94
+ lockMode string
95
+ lockStatus string
96
+ waitResource string
97
+}
98
+
99
+type mssqlDeadlockParseResult struct {
100
+ deadlockTime time.Time
101
+ transactions []*mssqlDeadlockTxn
102
+ victimProcessID string
103
+ parseErr error
104
+ found bool
105
+}
106
+
107
+type mssqlDeadlockGraph struct {
108
+ XMLName xml.Name `xml:"deadlock"`
109
+ VictimList mssqlDeadlockVictimList `xml:"victim-list"`
110
+ ProcessList mssqlDeadlockProcessList `xml:"process-list"`
111
+ ResourceList mssqlDeadlockResourceList `xml:"resource-list"`
112
+}
113
+
114
+type mssqlDeadlockResourceList struct {
115
+ Resources []mssqlDeadlockResource `xml:",any"`
116
+}
117
+
118
+type mssqlDeadlockVictimList struct {
119
+ Victims []mssqlDeadlockVictim `xml:"victimProcess"`
120
+}
121
+
122
+type mssqlDeadlockVictim struct {
123
+ ID string `xml:"id,attr"`
124
+}
125
+
126
+type mssqlDeadlockProcessList struct {
127
+ Processes []mssqlDeadlockProcess `xml:"process"`
128
+}
129
+
130
+type mssqlDeadlockProcess struct {
131
+ ID string `xml:"id,attr"`
132
+ SPID string `xml:"spid,attr"`
133
+ ECID string `xml:"ecid,attr"`
134
+ DBID string `xml:"dbid,attr"`
135
+ LockMode string `xml:"lockMode,attr"`
136
+ WaitResource string `xml:"waitresource,attr"`
137
+ InputBuf string `xml:"inputbuf"`
138
+}
139
+
140
+type mssqlDeadlockResource struct {
141
+ XMLName xml.Name
142
+ DBID string `xml:"dbid,attr"`
143
+ OwnerList mssqlDeadlockOwnerList `xml:"owner-list"`
144
+ WaiterList mssqlDeadlockWaiterList `xml:"waiter-list"`
145
+}
146
+
147
+type mssqlDeadlockOwnerList struct {
148
+ Owners []mssqlDeadlockResourceEntry `xml:"owner"`
149
+}
150
+
151
+type mssqlDeadlockWaiterList struct {
152
+ Waiters []mssqlDeadlockResourceEntry `xml:"waiter"`
153
+}
154
+
155
+type mssqlDeadlockResourceEntry struct {
156
+ ID string `xml:"id,attr"`
157
+ Mode string `xml:"mode,attr"`
158
+}
159
+
160
+func (c *Collector) deadlockInfoParams(context.Context) ([]funcapi.ParamConfig, error) {
161
+ if !c.Config.GetDeadlockInfoFunctionEnabled() {
162
+ return nil, fmt.Errorf("deadlock-info function disabled in configuration")
163
+ }
164
+ return []funcapi.ParamConfig{}, nil
165
+}
166
+
167
+func (c *Collector) collectDeadlockInfo(ctx context.Context) *funcapi.FunctionResponse {
168
+ if !c.Config.GetDeadlockInfoFunctionEnabled() {
169
+ return &funcapi.FunctionResponse{
170
+ Status: 503,
171
+ Message: "deadlock-info function has been disabled in configuration. " +
172
+ "To enable, set deadlock_info_function_enabled: true in the MSSQL collector config.",
173
+ }
174
+ }
175
+
176
+ deadlockTime, deadlockXML, err := c.queryLatestDeadlock(ctx)
177
+ if err != nil {
178
+ if errors.Is(err, context.DeadlineExceeded) {
179
+ return c.deadlockInfoResponse(504, "deadlock query timed out", nil)
180
+ }
181
+ if isDeadlockPermissionError(err) {
182
+ return c.deadlockInfoResponse(403, deadlockPermissionMessage(), nil)
183
+ }
184
+ c.Warningf("deadlock-info: query failed: %v", err)
185
+ return c.deadlockInfoResponse(500, fmt.Sprintf("deadlock query failed: %v", err), nil)
186
+ }
187
+
188
+ if deadlockXML == "" {
189
+ return c.deadlockInfoResponse(200, "no deadlock found in system_health ring buffer", nil)
190
+ }
191
+
192
+ dbNames, dbErr := c.queryDatabaseNames(ctx)
193
+ if dbErr != nil {
194
+ c.Debugf("deadlock-info: database name mapping failed: %v", dbErr)
195
+ dbNames = map[int]string{}
196
+ }
197
+
198
+ parseRes := parseDeadlockGraph(deadlockXML, deadlockTime)
199
+ if parseRes.parseErr != nil {
200
+ c.Warningf("deadlock-info: parse failed: %v", parseRes.parseErr)
201
+ return c.deadlockInfoResponse(deadlockParseErrorStatus, "deadlock graph could not be parsed", nil)
202
+ }
203
+ if !parseRes.found {
204
+ return c.deadlockInfoResponse(200, "no deadlock found in system_health ring buffer", nil)
205
+ }
206
+
207
+ deadlockID := generateDeadlockID(parseRes.deadlockTime)
208
+ rows := buildDeadlockRows(parseRes, deadlockID, dbNames)
209
+ if len(rows) == 0 {
210
+ return c.deadlockInfoResponse(200, "deadlock detected but no processes could be parsed", nil)
211
+ }
212
+
213
+ return c.deadlockInfoResponse(200, "latest detected deadlock", rows)
214
+}
215
+
216
+func (c *Collector) deadlockInfoResponse(status int, message string, data [][]any) *funcapi.FunctionResponse {
217
+ if data == nil {
218
+ data = make([][]any, 0)
219
+ }
220
+ return &funcapi.FunctionResponse{
221
+ Status: status,
222
+ Help: deadlockInfoHelp,
223
+ Message: message,
224
+ Columns: c.buildDeadlockColumns(),
225
+ Data: data,
226
+ DefaultSortColumn: "timestamp",
227
+ }
228
+}
229
+
230
+func (c *Collector) queryLatestDeadlock(ctx context.Context) (time.Time, string, error) {
231
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
232
+ defer cancel()
233
+
234
+ var deadlockTime sql.NullTime
235
+ var deadlockXML sql.NullString
236
+ err := c.db.QueryRowContext(qctx, querySystemHealthLatestDeadlock).Scan(&deadlockTime, &deadlockXML)
237
+ if err != nil {
238
+ if errors.Is(err, sql.ErrNoRows) {
239
+ return time.Time{}, "", nil
240
+ }
241
+ return time.Time{}, "", err
242
+ }
243
+
244
+ if !deadlockXML.Valid || strings.TrimSpace(deadlockXML.String) == "" {
245
+ return time.Time{}, "", nil
246
+ }
247
+
248
+ if deadlockTime.Valid {
249
+ return deadlockTime.Time, deadlockXML.String, nil
250
+ }
251
+ return time.Now().UTC(), deadlockXML.String, nil
252
+}
253
+
254
+func (c *Collector) queryDatabaseNames(ctx context.Context) (map[int]string, error) {
255
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
256
+ defer cancel()
257
+
258
+ rows, err := c.db.QueryContext(qctx, queryDatabaseNamesByID)
259
+ if err != nil {
260
+ return nil, err
261
+ }
262
+ defer rows.Close()
263
+
264
+ names := make(map[int]string)
265
+ for rows.Next() {
266
+ var id int
267
+ var name string
268
+ if err := rows.Scan(&id, &name); err != nil {
269
+ return nil, err
270
+ }
271
+ names[id] = name
272
+ }
273
+ if err := rows.Err(); err != nil {
274
+ return nil, err
275
+ }
276
+ return names, nil
277
+}
278
+
279
+func (c *Collector) buildDeadlockColumns() map[string]any {
280
+ const (
281
+ ftString = funcapi.FieldTypeString
282
+ ftInteger = funcapi.FieldTypeInteger
283
+ ftTimestamp = funcapi.FieldTypeTimestamp
284
+
285
+ trNone = funcapi.FieldTransformNone
286
+ trNumber = funcapi.FieldTransformNumber
287
+ trDatetime = funcapi.FieldTransformDatetime
288
+
289
+ visValue = funcapi.FieldVisualValue
290
+ visPill = funcapi.FieldVisualPill
291
+
292
+ sortAsc = funcapi.FieldSortAscending
293
+ sortDesc = funcapi.FieldSortDescending
294
+
295
+ summaryCount = funcapi.FieldSummaryCount
296
+ summaryMax = funcapi.FieldSummaryMax
297
+
298
+ filterMulti = funcapi.FieldFilterMultiselect
299
+ filterRange = funcapi.FieldFilterRange
300
+ )
301
+
302
+ return map[string]any{
303
+ "row_id": funcapi.Column{
304
+ Index: deadlockIdxRowID,
305
+ Name: "Row ID",
306
+ Type: ftString,
307
+ Visualization: visValue,
308
+ Sort: sortAsc,
309
+ Sortable: true,
310
+ Summary: summaryCount,
311
+ Filter: filterMulti,
312
+ UniqueKey: true,
313
+ Visible: false,
314
+ ValueOptions: funcapi.ValueOptions{
315
+ Transform: trNone,
316
+ },
317
+ }.BuildColumn(),
318
+ "deadlock_id": funcapi.Column{
319
+ Index: deadlockIdxDeadlockID,
320
+ Name: "Deadlock ID",
321
+ Type: ftString,
322
+ Visualization: visValue,
323
+ Sort: sortDesc,
324
+ Sortable: true,
325
+ Sticky: true,
326
+ Summary: summaryCount,
327
+ Filter: filterMulti,
328
+ Visible: true,
329
+ ValueOptions: funcapi.ValueOptions{
330
+ Transform: trNone,
331
+ },
332
+ }.BuildColumn(),
333
+ "timestamp": funcapi.Column{
334
+ Index: deadlockIdxTimestamp,
335
+ Name: "Timestamp",
336
+ Type: ftTimestamp,
337
+ Units: "",
338
+ Visualization: visValue,
339
+ Sort: sortDesc,
340
+ Sortable: true,
341
+ Sticky: true,
342
+ Summary: summaryMax,
343
+ Filter: filterRange,
344
+ Visible: true,
345
+ ValueOptions: funcapi.ValueOptions{
346
+ Transform: trDatetime,
347
+ },
348
+ }.BuildColumn(),
349
+ "process_id": funcapi.Column{
350
+ Index: deadlockIdxProcessID,
351
+ Name: "Process ID",
352
+ Type: ftString,
353
+ Visualization: visValue,
354
+ Sort: sortAsc,
355
+ Sortable: true,
356
+ Sticky: true,
357
+ Summary: summaryCount,
358
+ Filter: filterMulti,
359
+ Visible: true,
360
+ ValueOptions: funcapi.ValueOptions{
361
+ Transform: trNone,
362
+ },
363
+ }.BuildColumn(),
364
+ "spid": funcapi.Column{
365
+ Index: deadlockIdxSpid,
366
+ Name: "SPID",
367
+ Type: ftInteger,
368
+ Visualization: visValue,
369
+ Sort: sortAsc,
370
+ Sortable: true,
371
+ Sticky: false,
372
+ Summary: summaryCount,
373
+ Filter: filterRange,
374
+ Visible: true,
375
+ ValueOptions: funcapi.ValueOptions{
376
+ Transform: trNumber,
377
+ },
378
+ }.BuildColumn(),
379
+ "ecid": funcapi.Column{
380
+ Index: deadlockIdxEcid,
381
+ Name: "ECID",
382
+ Type: ftInteger,
383
+ Visualization: visValue,
384
+ Sort: sortAsc,
385
+ Sortable: true,
386
+ Sticky: false,
387
+ Summary: summaryCount,
388
+ Filter: filterRange,
389
+ Visible: true,
390
+ ValueOptions: funcapi.ValueOptions{
391
+ Transform: trNumber,
392
+ },
393
+ }.BuildColumn(),
394
+ "is_victim": funcapi.Column{
395
+ Index: deadlockIdxIsVictim,
396
+ Name: "Victim",
397
+ Type: ftString,
398
+ Visualization: visPill,
399
+ Sort: sortAsc,
400
+ Sortable: true,
401
+ Summary: summaryCount,
402
+ Filter: filterMulti,
403
+ Visible: true,
404
+ ValueOptions: funcapi.ValueOptions{
405
+ Transform: trNone,
406
+ },
407
+ }.BuildColumn(),
408
+ "query_text": funcapi.Column{
409
+ Index: deadlockIdxQueryText,
410
+ Name: "Query",
411
+ Type: ftString,
412
+ Visualization: visValue,
413
+ Sort: sortAsc,
414
+ Sortable: false,
415
+ Sticky: false,
416
+ Summary: summaryCount,
417
+ Filter: filterMulti,
418
+ FullWidth: true,
419
+ Wrap: true,
420
+ Visible: true,
421
+ ValueOptions: funcapi.ValueOptions{
422
+ Transform: trNone,
423
+ },
424
+ }.BuildColumn(),
425
+ "lock_mode": funcapi.Column{
426
+ Index: deadlockIdxLockMode,
427
+ Name: "Lock Mode",
428
+ Type: ftString,
429
+ Visualization: visValue,
430
+ Sort: sortAsc,
431
+ Sortable: true,
432
+ Sticky: false,
433
+ Summary: summaryCount,
434
+ Filter: filterMulti,
435
+ Visible: true,
436
+ ValueOptions: funcapi.ValueOptions{
437
+ Transform: trNone,
438
+ },
439
+ }.BuildColumn(),
440
+ "lock_status": funcapi.Column{
441
+ Index: deadlockIdxLockStatus,
442
+ Name: "Lock Status",
443
+ Type: ftString,
444
+ Visualization: visPill,
445
+ Sort: sortAsc,
446
+ Sortable: true,
447
+ Sticky: false,
448
+ Summary: summaryCount,
449
+ Filter: filterMulti,
450
+ Visible: true,
451
+ ValueOptions: funcapi.ValueOptions{
452
+ Transform: trNone,
453
+ },
454
+ }.BuildColumn(),
455
+ "wait_resource": funcapi.Column{
456
+ Index: deadlockIdxWaitResource,
457
+ Name: "Wait Resource",
458
+ Type: ftString,
459
+ Visualization: visValue,
460
+ Sort: sortAsc,
461
+ Sortable: false,
462
+ Sticky: false,
463
+ Summary: summaryCount,
464
+ Filter: filterMulti,
465
+ FullWidth: true,
466
+ Wrap: true,
467
+ Visible: true,
468
+ ValueOptions: funcapi.ValueOptions{
469
+ Transform: trNone,
470
+ },
471
+ }.BuildColumn(),
472
+ "database": funcapi.Column{
473
+ Index: deadlockIdxDatabase,
474
+ Name: "Database",
475
+ Type: ftString,
476
+ Visualization: visValue,
477
+ Sort: sortAsc,
478
+ Sortable: true,
479
+ Sticky: false,
480
+ Summary: summaryCount,
481
+ Filter: filterMulti,
482
+ Visible: true,
483
+ ValueOptions: funcapi.ValueOptions{
484
+ Transform: trNone,
485
+ },
486
+ }.BuildColumn(),
487
+ }
488
+}
489
+
490
+func parseDeadlockGraph(deadlockXML string, deadlockTime time.Time) mssqlDeadlockParseResult {
491
+ now := time.Now().UTC()
492
+ result := mssqlDeadlockParseResult{
493
+ deadlockTime: now,
494
+ found: strings.TrimSpace(deadlockXML) != "",
495
+ }
496
+ if result.found && !deadlockTime.IsZero() {
497
+ result.deadlockTime = deadlockTime.UTC()
498
+ }
499
+
500
+ if !result.found {
501
+ return result
502
+ }
503
+
504
+ var graph mssqlDeadlockGraph
505
+ if err := xml.Unmarshal([]byte(deadlockXML), &graph); err != nil {
506
+ result.parseErr = fmt.Errorf("failed to parse deadlock XML: %w", err)
507
+ return result
508
+ }
509
+
510
+ if len(graph.VictimList.Victims) > 0 {
511
+ result.victimProcessID = strings.TrimSpace(graph.VictimList.Victims[0].ID)
512
+ }
513
+
514
+ txnByID := make(map[string]*mssqlDeadlockTxn)
515
+ ensureTxn := func(id string) *mssqlDeadlockTxn {
516
+ if id == "" {
517
+ return &mssqlDeadlockTxn{}
518
+ }
519
+ if txn, ok := txnByID[id]; ok {
520
+ return txn
521
+ }
522
+ txn := &mssqlDeadlockTxn{processID: id}
523
+ txnByID[id] = txn
524
+ return txn
525
+ }
526
+
527
+ for _, proc := range graph.ProcessList.Processes {
528
+ processID := strings.TrimSpace(proc.ID)
529
+ if processID == "" {
530
+ continue
531
+ }
532
+ txn := ensureTxn(processID)
533
+ txn.spid = strings.TrimSpace(proc.SPID)
534
+ txn.ecid = strings.TrimSpace(proc.ECID)
535
+ txn.dbid = strings.TrimSpace(proc.DBID)
536
+ txn.queryText = strings.TrimSpace(proc.InputBuf)
537
+ txn.lockMode = strings.TrimSpace(proc.LockMode)
538
+ txn.waitResource = strings.TrimSpace(proc.WaitResource)
539
+ if txn.waitResource != "" {
540
+ txn.lockStatus = "WAITING"
541
+ } else if txn.lockStatus == "" {
542
+ txn.lockStatus = "GRANTED"
543
+ }
544
+ }
545
+
546
+ for _, resource := range graph.ResourceList.Resources {
547
+ resourceDBID := strings.TrimSpace(resource.DBID)
548
+ for _, owner := range resource.OwnerList.Owners {
549
+ id := strings.TrimSpace(owner.ID)
550
+ if id == "" {
551
+ continue
552
+ }
553
+ txn := ensureTxn(id)
554
+ if txn.dbid == "" && resourceDBID != "" {
555
+ txn.dbid = resourceDBID
556
+ }
557
+ if txn.lockStatus != "WAITING" {
558
+ if txn.lockStatus == "" {
559
+ txn.lockStatus = "GRANTED"
560
+ }
561
+ mode := strings.TrimSpace(owner.Mode)
562
+ if mode != "" {
563
+ txn.lockMode = mode
564
+ }
565
+ }
566
+ }
567
+ for _, waiter := range resource.WaiterList.Waiters {
568
+ id := strings.TrimSpace(waiter.ID)
569
+ if id == "" {
570
+ continue
571
+ }
572
+ txn := ensureTxn(id)
573
+ if txn.dbid == "" && resourceDBID != "" {
574
+ txn.dbid = resourceDBID
575
+ }
576
+ txn.lockStatus = "WAITING"
577
+ mode := strings.TrimSpace(waiter.Mode)
578
+ if mode != "" {
579
+ txn.lockMode = mode
580
+ }
581
+ }
582
+ }
583
+
584
+ if len(txnByID) == 0 {
585
+ result.parseErr = fmt.Errorf("deadlock graph detected but no processes could be parsed")
586
+ return result
587
+ }
588
+
589
+ result.transactions = make([]*mssqlDeadlockTxn, 0, len(txnByID))
590
+ for _, txn := range txnByID {
591
+ if txn.processID == "" {
592
+ continue
593
+ }
594
+ if txn.lockStatus == "" {
595
+ if strings.TrimSpace(txn.waitResource) != "" {
596
+ txn.lockStatus = "WAITING"
597
+ } else {
598
+ txn.lockStatus = "GRANTED"
599
+ }
600
+ }
601
+ result.transactions = append(result.transactions, txn)
602
+ }
603
+
604
+ sort.Slice(result.transactions, func(i, j int) bool {
605
+ return result.transactions[i].processID < result.transactions[j].processID
606
+ })
607
+
608
+ if len(result.transactions) == 0 {
609
+ result.parseErr = fmt.Errorf("deadlock graph detected but no valid processes could be parsed")
610
+ }
611
+
612
+ return result
613
+}
614
+
615
+func buildDeadlockRows(parseRes mssqlDeadlockParseResult, deadlockID string, dbNames map[int]string) [][]any {
616
+ timestamp := parseRes.deadlockTime.UTC().Format(time.RFC3339Nano)
617
+ rows := make([][]any, 0, len(parseRes.transactions))
618
+
619
+ for _, txn := range parseRes.transactions {
620
+ processID := strings.TrimSpace(txn.processID)
621
+ if processID == "" {
622
+ continue
623
+ }
624
+
625
+ spid := parseOptionalInt(txn.spid)
626
+ ecid := parseOptionalInt(txn.ecid)
627
+ dbidInt, hasDBID := parseIntString(txn.dbid)
628
+
629
+ var database any
630
+ if hasDBID {
631
+ if name, ok := dbNames[dbidInt]; ok {
632
+ database = name
633
+ }
634
+ }
635
+
636
+ isVictim := "false"
637
+ if parseRes.victimProcessID != "" && processID == parseRes.victimProcessID {
638
+ isVictim = "true"
639
+ }
640
+
641
+ queryText := strmutil.TruncateText(strings.TrimSpace(txn.queryText), topQueriesMaxTextLength)
642
+ lockMode := strings.TrimSpace(txn.lockMode)
643
+ lockStatus := strings.TrimSpace(txn.lockStatus)
644
+ waitResource := strmutil.TruncateText(strings.TrimSpace(txn.waitResource), topQueriesMaxTextLength)
645
+
646
+ row := make([]any, deadlockColumnCount)
647
+ row[deadlockIdxRowID] = fmt.Sprintf("%s:%s", deadlockID, processID)
648
+ row[deadlockIdxDeadlockID] = deadlockID
649
+ row[deadlockIdxTimestamp] = timestamp
650
+ row[deadlockIdxProcessID] = processID
651
+ row[deadlockIdxSpid] = spid
652
+ row[deadlockIdxEcid] = ecid
653
+ row[deadlockIdxIsVictim] = isVictim
654
+ row[deadlockIdxQueryText] = queryText
655
+ row[deadlockIdxLockMode] = lockMode
656
+ row[deadlockIdxLockStatus] = lockStatus
657
+ row[deadlockIdxWaitResource] = waitResource
658
+ row[deadlockIdxDatabase] = database
659
+
660
+ rows = append(rows, row)
661
+ }
662
+
663
+ return rows
664
+}
665
+
666
+func parseIntString(s string) (int, bool) {
667
+ s = strings.TrimSpace(s)
668
+ if s == "" {
669
+ return 0, false
670
+ }
671
+ n, err := strconv.Atoi(s)
672
+ if err != nil {
673
+ return 0, false
674
+ }
675
+ return n, true
676
+}
677
+
678
+func parseOptionalInt(s string) any {
679
+ if n, ok := parseIntString(s); ok {
680
+ return n
681
+ }
682
+ return nil
683
+}
684
+
685
+func generateDeadlockID(t time.Time) string {
686
+ if t.IsZero() {
687
+ t = time.Now().UTC()
688
+ }
689
+ t = t.UTC()
690
+ micros := t.Nanosecond() / 1000
691
+ return t.Format("20060102150405") + fmt.Sprintf("%06d", micros)
692
+}
693
+
694
+func isDeadlockPermissionError(err error) bool {
695
+ var sqlErr mssqlDriver.Error
696
+ if errors.As(err, &sqlErr) {
697
+ if sqlErr.Number == 297 || sqlErr.Number == 229 {
698
+ return true
699
+ }
700
+ if permissionMessage := strings.ToLower(sqlErr.Message); permissionMessage != "" {
701
+ if strings.Contains(permissionMessage, "view server state") ||
702
+ strings.Contains(permissionMessage, "permission") ||
703
+ strings.Contains(permissionMessage, "denied") {
704
+ return true
705
+ }
706
+ }
707
+ }
708
+
709
+ msg := strings.ToLower(err.Error())
710
+ return strings.Contains(msg, "view server state") ||
711
+ strings.Contains(msg, "permission") ||
712
+ strings.Contains(msg, "denied")
713
+}
714
+
715
+func deadlockPermissionMessage() string {
716
+ return "deadlock info requires VIEW SERVER STATE permission. Grant with: GRANT VIEW SERVER STATE TO [netdata_user];"
717
+}
src/go/plugin/go.d/collector/mssql/deadlock_info_test.go
new
+440
@@ -0,0 +1,440 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+package mssql
4
+
5
+import (
6
+ "context"
7
+ "errors"
8
+ "strings"
9
+ "testing"
10
+ "time"
11
+
12
+ "github.com/DATA-DOG/go-sqlmock"
13
+ mssqlDriver "github.com/microsoft/go-mssqldb"
14
+ "github.com/stretchr/testify/assert"
15
+ "github.com/stretchr/testify/require"
16
+)
17
+
18
+func TestConfig_GetDeadlockInfoFunctionEnabled(t *testing.T) {
19
+ tests := []struct {
20
+ name string
21
+ cfg Config
22
+ want bool
23
+ }{
24
+ {
25
+ name: "default enabled when unset",
26
+ cfg: Config{},
27
+ want: true,
28
+ },
29
+ {
30
+ name: "explicitly enabled",
31
+ cfg: Config{
32
+ DeadlockInfoFunctionEnabled: boolPtr(true),
33
+ },
34
+ want: true,
35
+ },
36
+ {
37
+ name: "explicitly disabled",
38
+ cfg: Config{
39
+ DeadlockInfoFunctionEnabled: boolPtr(false),
40
+ },
41
+ want: false,
42
+ },
43
+ }
44
+
45
+ for _, tt := range tests {
46
+ t.Run(tt.name, func(t *testing.T) {
47
+ assert.Equal(t, tt.want, tt.cfg.GetDeadlockInfoFunctionEnabled())
48
+ })
49
+ }
50
+}
51
+
52
+func TestConfig_GetErrorInfoFunctionEnabled(t *testing.T) {
53
+ tests := []struct {
54
+ name string
55
+ cfg Config
56
+ want bool
57
+ }{
58
+ {
59
+ name: "default enabled when unset",
60
+ cfg: Config{},
61
+ want: true,
62
+ },
63
+ {
64
+ name: "explicitly enabled",
65
+ cfg: Config{
66
+ ErrorInfoFunctionEnabled: boolPtr(true),
67
+ },
68
+ want: true,
69
+ },
70
+ {
71
+ name: "explicitly disabled",
72
+ cfg: Config{
73
+ ErrorInfoFunctionEnabled: boolPtr(false),
74
+ },
75
+ want: false,
76
+ },
77
+ }
78
+
79
+ for _, tt := range tests {
80
+ t.Run(tt.name, func(t *testing.T) {
81
+ assert.Equal(t, tt.want, tt.cfg.GetErrorInfoFunctionEnabled())
82
+ })
83
+ }
84
+}
85
+
86
+func TestConfig_GetErrorInfoSessionName(t *testing.T) {
87
+ tests := []struct {
88
+ name string
89
+ cfg Config
90
+ want string
91
+ }{
92
+ {
93
+ name: "default session name",
94
+ cfg: Config{},
95
+ want: "netdata_errors",
96
+ },
97
+ {
98
+ name: "explicit session name",
99
+ cfg: Config{
100
+ ErrorInfoSessionName: "custom_errors",
101
+ },
102
+ want: "custom_errors",
103
+ },
104
+ }
105
+
106
+ for _, tt := range tests {
107
+ t.Run(tt.name, func(t *testing.T) {
108
+ assert.Equal(t, tt.want, tt.cfg.GetErrorInfoSessionName())
109
+ })
110
+ }
111
+}
112
+
113
+func TestParseDeadlockGraph_WithDeadlock(t *testing.T) {
114
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 123456000, time.UTC)
115
+ res := parseDeadlockGraph(sampleDeadlockGraph, now)
116
+
117
+ require.True(t, res.found)
118
+ require.NoError(t, res.parseErr)
119
+ require.Equal(t, now.UTC(), res.deadlockTime)
120
+ require.Equal(t, "process1", res.victimProcessID)
121
+ require.Len(t, res.transactions, 2)
122
+
123
+ txn1 := findTxn(res.transactions, "process1")
124
+ txn2 := findTxn(res.transactions, "process2")
125
+
126
+ require.NotNil(t, txn1)
127
+ require.NotNil(t, txn2)
128
+
129
+ assert.Equal(t, "WAITING", txn1.lockStatus)
130
+ assert.Equal(t, "X", txn1.lockMode)
131
+ assert.Contains(t, txn1.queryText, "deadlock_a")
132
+
133
+ assert.Equal(t, "WAITING", txn2.lockStatus)
134
+ assert.Equal(t, "X", txn2.lockMode)
135
+ assert.Contains(t, txn2.queryText, "deadlock_b")
136
+}
137
+
138
+func TestParseDeadlockGraph_ThreeWayDeadlock(t *testing.T) {
139
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 222000000, time.UTC)
140
+ res := parseDeadlockGraph(sampleDeadlockGraphThreeWay, now)
141
+
142
+ require.True(t, res.found)
143
+ require.NoError(t, res.parseErr)
144
+ require.Equal(t, "process2", res.victimProcessID)
145
+ require.Len(t, res.transactions, 3)
146
+
147
+ deadlockID := generateDeadlockID(now)
148
+ rows := buildDeadlockRows(res, deadlockID, map[int]string{5: "netdata"})
149
+ require.Len(t, rows, 3)
150
+
151
+ victimCount := 0
152
+ for _, row := range rows {
153
+ assert.Equal(t, deadlockID, row[deadlockIdxDeadlockID])
154
+ assert.True(t, strings.HasPrefix(row[deadlockIdxRowID].(string), deadlockID+":"))
155
+ if row[deadlockIdxIsVictim] == "true" {
156
+ victimCount++
157
+ }
158
+ }
159
+ assert.Equal(t, 1, victimCount)
160
+}
161
+
162
+func TestParseDeadlockGraph_WaitingWinsOverOwner(t *testing.T) {
163
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
164
+ res := parseDeadlockGraph(sampleDeadlockOwnerAfterWaiter, now)
165
+
166
+ require.True(t, res.found)
167
+ require.NoError(t, res.parseErr)
168
+
169
+ txn := findTxn(res.transactions, "process1")
170
+ require.NotNil(t, txn)
171
+ assert.Equal(t, "WAITING", txn.lockStatus)
172
+ assert.Equal(t, "X", txn.lockMode)
173
+}
174
+
175
+func TestParseDeadlockGraph_NoDeadlock(t *testing.T) {
176
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
177
+ res := parseDeadlockGraph("", now)
178
+
179
+ assert.False(t, res.found)
180
+ assert.NoError(t, res.parseErr)
181
+ assert.Len(t, res.transactions, 0)
182
+}
183
+
184
+func TestParseDeadlockGraph_Malformed(t *testing.T) {
185
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
186
+ res := parseDeadlockGraph("<deadlock><broken>", now)
187
+
188
+ assert.True(t, res.found)
189
+ assert.Error(t, res.parseErr)
190
+}
191
+
192
+func TestCollectDeadlockInfo_ParseError(t *testing.T) {
193
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherRegexp))
194
+ require.NoError(t, err)
195
+ defer db.Close()
196
+
197
+ deadlockTime := time.Date(2026, time.January, 25, 12, 34, 56, 0, time.UTC)
198
+ deadlockRows := sqlmock.NewRows([]string{"deadlock_time", "deadlock_xml"}).
199
+ AddRow(deadlockTime, "<deadlock><broken>")
200
+ mock.ExpectQuery("WITH xevents").WillReturnRows(deadlockRows)
201
+
202
+ dbNameRows := sqlmock.NewRows([]string{"database_id", "name"})
203
+ mock.ExpectQuery("SELECT\\s+database_id").WillReturnRows(dbNameRows)
204
+
205
+ c := New()
206
+ c.db = db
207
+
208
+ resp := c.collectDeadlockInfo(context.Background())
209
+ require.Equal(t, deadlockParseErrorStatus, resp.Status)
210
+ assert.Contains(t, strings.ToLower(resp.Message), "could not be parsed")
211
+ require.NoError(t, mock.ExpectationsWereMet())
212
+}
213
+
214
+func TestCollectDeadlockInfo_QueryError(t *testing.T) {
215
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherRegexp))
216
+ require.NoError(t, err)
217
+ defer db.Close()
218
+
219
+ mock.ExpectQuery("WITH xevents").
220
+ WillReturnError(errors.New("boom"))
221
+
222
+ c := New()
223
+ c.db = db
224
+
225
+ resp := c.collectDeadlockInfo(context.Background())
226
+ require.Equal(t, 500, resp.Status)
227
+ assert.Contains(t, strings.ToLower(resp.Message), "deadlock query failed")
228
+ require.NoError(t, mock.ExpectationsWereMet())
229
+}
230
+
231
+func TestBuildDeadlockRows(t *testing.T) {
232
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 654321000, time.UTC)
233
+ res := parseDeadlockGraph(sampleDeadlockGraph, now)
234
+ require.NoError(t, res.parseErr)
235
+
236
+ deadlockID := generateDeadlockID(now)
237
+ dbNames := map[int]string{5: "netdata"}
238
+ rows := buildDeadlockRows(res, deadlockID, dbNames)
239
+
240
+ require.Len(t, rows, 2)
241
+
242
+ row := rows[0]
243
+ assert.Equal(t, deadlockID, row[deadlockIdxDeadlockID])
244
+ assert.Equal(t, deadlockID+":"+row[deadlockIdxProcessID].(string), row[deadlockIdxRowID])
245
+ assert.Equal(t, "netdata", row[deadlockIdxDatabase])
246
+}
247
+
248
+func TestDeadlockPermissionErrorDetection(t *testing.T) {
249
+ err := mssqlDriver.Error{Number: 297, Message: "VIEW SERVER STATE permission was denied"}
250
+ assert.True(t, isDeadlockPermissionError(err))
251
+}
252
+
253
+func TestCollectDeadlockInfo_PermissionDenied(t *testing.T) {
254
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherRegexp))
255
+ require.NoError(t, err)
256
+ defer db.Close()
257
+
258
+ mock.ExpectQuery("WITH xevents").
259
+ WillReturnError(mssqlDriver.Error{Number: 297, Message: "VIEW SERVER STATE permission was denied"})
260
+
261
+ c := New()
262
+ c.db = db
263
+
264
+ resp := c.collectDeadlockInfo(context.Background())
265
+ require.Equal(t, 403, resp.Status)
266
+ assert.Contains(t, strings.ToLower(resp.Message), "view server state")
267
+ require.NoError(t, mock.ExpectationsWereMet())
268
+}
269
+
270
+func TestCollectDeadlockInfo_Disabled(t *testing.T) {
271
+ c := New()
272
+ c.Config.DeadlockInfoFunctionEnabled = boolPtr(false)
273
+
274
+ resp := c.collectDeadlockInfo(context.Background())
275
+ require.Equal(t, 503, resp.Status)
276
+ assert.Contains(t, strings.ToLower(resp.Message), "disabled")
277
+}
278
+
279
+func TestCollectDeadlockInfo_Timeout(t *testing.T) {
280
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherRegexp))
281
+ require.NoError(t, err)
282
+ defer db.Close()
283
+
284
+ mock.ExpectQuery("WITH xevents").
285
+ WillReturnError(context.DeadlineExceeded)
286
+
287
+ c := New()
288
+ c.db = db
289
+
290
+ resp := c.collectDeadlockInfo(context.Background())
291
+ require.Equal(t, 504, resp.Status)
292
+ assert.Contains(t, strings.ToLower(resp.Message), "timed out")
293
+ require.NoError(t, mock.ExpectationsWereMet())
294
+}
295
+
296
+func TestCollectDeadlockInfo_Success(t *testing.T) {
297
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherRegexp))
298
+ require.NoError(t, err)
299
+ defer db.Close()
300
+
301
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
302
+
303
+ mock.ExpectQuery("WITH xevents").
304
+ WillReturnRows(
305
+ sqlmock.NewRows([]string{"deadlock_time", "deadlock_xml"}).
306
+ AddRow(now, sampleDeadlockGraph),
307
+ )
308
+
309
+ mock.ExpectQuery("SELECT database_id, name").
310
+ WillReturnRows(
311
+ sqlmock.NewRows([]string{"database_id", "name"}).
312
+ AddRow(5, "netdata"),
313
+ )
314
+
315
+ c := New()
316
+ c.db = db
317
+
318
+ resp := c.collectDeadlockInfo(context.Background())
319
+ require.Equal(t, 200, resp.Status)
320
+ require.NotEmpty(t, resp.Data)
321
+ require.NoError(t, mock.ExpectationsWereMet())
322
+}
323
+
324
+func findTxn(txns []*mssqlDeadlockTxn, id string) *mssqlDeadlockTxn {
325
+ for _, txn := range txns {
326
+ if txn.processID == id {
327
+ return txn
328
+ }
329
+ }
330
+ return nil
331
+}
332
+
333
+func boolPtr(v bool) *bool { return &v }
334
+
335
+const sampleDeadlockGraph = `
336
+<deadlock>
337
+ <victim-list>
338
+ <victimProcess id="process1" />
339
+ </victim-list>
340
+ <process-list>
341
+ <process id="process1" spid="62" ecid="0" dbid="5" lockMode="S" waitresource="KEY: 5:111">
342
+ <inputbuf>UPDATE dbo.deadlock_a SET value = value + 1 WHERE id = 1</inputbuf>
343
+ </process>
344
+ <process id="process2" spid="63" ecid="0" dbid="5" lockMode="X" waitresource="KEY: 5:222">
345
+ <inputbuf>UPDATE dbo.deadlock_b SET value = value + 1 WHERE id = 1</inputbuf>
346
+ </process>
347
+ </process-list>
348
+ <resource-list>
349
+ <keylock dbid="5">
350
+ <owner-list>
351
+ <owner id="process1" mode="S" />
352
+ </owner-list>
353
+ <waiter-list>
354
+ <waiter id="process2" mode="X" />
355
+ </waiter-list>
356
+ </keylock>
357
+ <keylock dbid="5">
358
+ <owner-list>
359
+ <owner id="process2" mode="S" />
360
+ </owner-list>
361
+ <waiter-list>
362
+ <waiter id="process1" mode="X" />
363
+ </waiter-list>
364
+ </keylock>
365
+ </resource-list>
366
+</deadlock>
367
+`
368
+
369
+const sampleDeadlockOwnerAfterWaiter = `
370
+<deadlock>
371
+ <victim-list>
372
+ <victimProcess id="process1" />
373
+ </victim-list>
374
+ <process-list>
375
+ <process id="process1" spid="62" ecid="0" dbid="5" lockMode="S" waitresource="KEY: 5:111">
376
+ <inputbuf>UPDATE dbo.deadlock_a SET value = value + 1 WHERE id = 1</inputbuf>
377
+ </process>
378
+ </process-list>
379
+ <resource-list>
380
+ <keylock dbid="5">
381
+ <owner-list>
382
+ <owner id="process2" mode="S" />
383
+ </owner-list>
384
+ <waiter-list>
385
+ <waiter id="process1" mode="X" />
386
+ </waiter-list>
387
+ </keylock>
388
+ <keylock dbid="5">
389
+ <owner-list>
390
+ <owner id="process1" mode="S" />
391
+ </owner-list>
392
+ </keylock>
393
+ </resource-list>
394
+</deadlock>
395
+`
396
+
397
+const sampleDeadlockGraphThreeWay = `
398
+<deadlock>
399
+ <victim-list>
400
+ <victimProcess id="process2" />
401
+ </victim-list>
402
+ <process-list>
403
+ <process id="process1" spid="62" ecid="0" dbid="5" lockMode="S" waitresource="KEY: 5:111">
404
+ <inputbuf>UPDATE dbo.deadlock_a SET value = value + 1 WHERE id = 1</inputbuf>
405
+ </process>
406
+ <process id="process2" spid="63" ecid="0" dbid="5" lockMode="X" waitresource="KEY: 5:222">
407
+ <inputbuf>UPDATE dbo.deadlock_b SET value = value + 1 WHERE id = 1</inputbuf>
408
+ </process>
409
+ <process id="process3" spid="64" ecid="0" dbid="5" lockMode="X" waitresource="KEY: 5:333">
410
+ <inputbuf>UPDATE dbo.deadlock_c SET value = value + 1 WHERE id = 1</inputbuf>
411
+ </process>
412
+ </process-list>
413
+ <resource-list>
414
+ <keylock dbid="5">
415
+ <owner-list>
416
+ <owner id="process1" mode="S" />
417
+ </owner-list>
418
+ <waiter-list>
419
+ <waiter id="process2" mode="X" />
420
+ </waiter-list>
421
+ </keylock>
422
+ <keylock dbid="5">
423
+ <owner-list>
424
+ <owner id="process2" mode="S" />
425
+ </owner-list>
426
+ <waiter-list>
427
+ <waiter id="process3" mode="X" />
428
+ </waiter-list>
429
+ </keylock>
430
+ <keylock dbid="5">
431
+ <owner-list>
432
+ <owner id="process3" mode="S" />
433
+ </owner-list>
434
+ <waiter-list>
435
+ <waiter id="process1" mode="X" />
436
+ </waiter-list>
437
+ </keylock>
438
+ </resource-list>
439
+</deadlock>
440
+`
src/go/plugin/go.d/collector/mssql/error_info.go
new
+606
@@ -0,0 +1,606 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+package mssql
4
+
5
+import (
6
+ "context"
7
+ "database/sql"
8
+ "encoding/xml"
9
+ "fmt"
10
+ "strings"
11
+ "time"
12
+
13
+ "github.com/netdata/netdata/go/plugins/pkg/funcapi"
14
+)
15
+
16
+const (
17
+ mssqlErrorAttrEnabled = "enabled"
18
+ mssqlErrorAttrNotEnabled = "not_enabled"
19
+ mssqlErrorAttrNotSupported = "not_supported"
20
+ mssqlErrorAttrNoData = "no_data"
21
+)
22
+
23
+const errorInfoMethodID = "error-info"
24
+
25
+func errorInfoMethodConfig() funcapi.MethodConfig {
26
+ return funcapi.MethodConfig{
27
+ ID: errorInfoMethodID,
28
+ Name: "Error Info",
29
+ UpdateEvery: 10,
30
+ Help: "Recent SQL errors from Extended Events error_reported",
31
+ RequireCloud: true,
32
+ RequiredParams: []funcapi.ParamConfig{},
33
+ }
34
+}
35
+
36
+// funcErrorInfo handles the error-info function.
37
+type funcErrorInfo struct {
38
+ router *funcRouter
39
+}
40
+
41
+func newFuncErrorInfo(r *funcRouter) *funcErrorInfo {
42
+ return &funcErrorInfo{router: r}
43
+}
44
+
45
+// Compile-time interface check.
46
+var _ funcapi.MethodHandler = (*funcErrorInfo)(nil)
47
+
48
+func (f *funcErrorInfo) MethodParams(ctx context.Context, method string) ([]funcapi.ParamConfig, error) {
49
+ if !f.router.collector.Config.GetErrorInfoFunctionEnabled() {
50
+ return nil, fmt.Errorf("error-info function disabled in configuration")
51
+ }
52
+ return []funcapi.ParamConfig{}, nil
53
+}
54
+
55
+func (f *funcErrorInfo) Handle(ctx context.Context, method string, params funcapi.ResolvedParams) *funcapi.FunctionResponse {
56
+ if f.router.collector.db == nil {
57
+ db, err := f.router.collector.openConnection()
58
+ if err != nil {
59
+ return funcapi.UnavailableResponse("collector is still initializing, please retry in a few seconds")
60
+ }
61
+ f.router.collector.db = db
62
+ }
63
+ return f.router.collector.collectErrorInfo(ctx)
64
+}
65
+
66
+func (f *funcErrorInfo) Cleanup(ctx context.Context) {}
67
+
68
+type mssqlErrorRow struct {
69
+ Time time.Time
70
+ ErrorNumber *int64
71
+ ErrorState *int64
72
+ Message string
73
+ Query string
74
+ QueryHash string
75
+}
76
+
77
+type mssqlPlanOps struct {
78
+ HashMatch int64
79
+ MergeJoin int64
80
+ NestedLoops int64
81
+ Sorts int64
82
+}
83
+
84
+func (c *Collector) errorInfoParams(context.Context) ([]funcapi.ParamConfig, error) {
85
+ if !c.Config.GetErrorInfoFunctionEnabled() {
86
+ return nil, fmt.Errorf("error-info function disabled in configuration")
87
+ }
88
+ return []funcapi.ParamConfig{}, nil
89
+}
90
+
91
+func (c *Collector) collectErrorInfo(ctx context.Context) *funcapi.FunctionResponse {
92
+ if !c.Config.GetErrorInfoFunctionEnabled() {
93
+ return &funcapi.FunctionResponse{
94
+ Status: 503,
95
+ Message: "error-info not enabled: function disabled in configuration. " +
96
+ "To enable, set error_info_function_enabled: true in the MSSQL collector config.",
97
+ }
98
+ }
99
+
100
+ sessionName := c.Config.GetErrorInfoSessionName()
101
+ status, rows, err := c.fetchMSSQLErrorRows(ctx, sessionName, c.TopQueriesLimit)
102
+ if err != nil {
103
+ if isDeadlockPermissionError(err) {
104
+ return &funcapi.FunctionResponse{Status: 403, Message: errorInfoPermissionMessage()}
105
+ }
106
+ if status == mssqlErrorAttrNotEnabled {
107
+ return &funcapi.FunctionResponse{Status: 503, Message: "error-info not enabled: Extended Events session not found or ring_buffer target missing"}
108
+ }
109
+ return &funcapi.FunctionResponse{Status: 500, Message: fmt.Sprintf("error-info query failed: %v", err)}
110
+ }
111
+
112
+ data := make([][]any, 0, len(rows))
113
+ for _, row := range rows {
114
+ var errNo any
115
+ var errState any
116
+ if row.ErrorNumber != nil {
117
+ errNo = *row.ErrorNumber
118
+ }
119
+ if row.ErrorState != nil {
120
+ errState = *row.ErrorState
121
+ }
122
+ data = append(data, []any{
123
+ row.Time,
124
+ errNo,
125
+ errState,
126
+ row.Message,
127
+ row.Query,
128
+ row.QueryHash,
129
+ })
130
+ }
131
+
132
+ return &funcapi.FunctionResponse{
133
+ Status: 200,
134
+ Help: "Recent SQL errors from Extended Events error_reported",
135
+ Columns: buildMSSQLErrorInfoColumns(),
136
+ Data: data,
137
+ DefaultSortColumn: "timestamp",
138
+ }
139
+}
140
+
141
+func buildMSSQLErrorInfoColumns() map[string]any {
142
+ columns := map[string]any{
143
+ "timestamp": funcapi.Column{
144
+ Index: 0,
145
+ Name: "Timestamp",
146
+ Type: funcapi.FieldTypeTimestamp,
147
+ Sortable: true,
148
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformDatetime},
149
+ }.BuildColumn(),
150
+ "errorNumber": funcapi.Column{
151
+ Index: 1,
152
+ Name: "Error Number",
153
+ Type: funcapi.FieldTypeInteger,
154
+ Sortable: true,
155
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNumber},
156
+ }.BuildColumn(),
157
+ "errorState": funcapi.Column{
158
+ Index: 2,
159
+ Name: "Error State",
160
+ Type: funcapi.FieldTypeInteger,
161
+ Sortable: true,
162
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNumber},
163
+ }.BuildColumn(),
164
+ "errorMessage": funcapi.Column{
165
+ Index: 3,
166
+ Name: "Error Message",
167
+ Type: funcapi.FieldTypeString,
168
+ Sortable: false,
169
+ FullWidth: true,
170
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
171
+ }.BuildColumn(),
172
+ "query": funcapi.Column{
173
+ Index: 4,
174
+ Name: "Query",
175
+ Type: funcapi.FieldTypeString,
176
+ Sortable: false,
177
+ FullWidth: true,
178
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
179
+ }.BuildColumn(),
180
+ "queryHash": funcapi.Column{
181
+ Index: 5,
182
+ Name: "Query Hash",
183
+ Type: funcapi.FieldTypeString,
184
+ Sortable: true,
185
+ Visible: false,
186
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
187
+ }.BuildColumn(),
188
+ }
189
+ return columns
190
+}
191
+
192
+func errorInfoPermissionMessage() string {
193
+ return "error-info requires VIEW SERVER STATE permission. Grant with: GRANT VIEW SERVER STATE TO [netdata_user];"
194
+}
195
+
196
+func mssqlErrorAttributionColumns() []topQueriesColumn {
197
+ return []topQueriesColumn{
198
+ {
199
+ ColumnMeta: funcapi.ColumnMeta{
200
+ Name: "errorAttribution",
201
+ Tooltip: "Error Attribution",
202
+ Type: funcapi.FieldTypeString,
203
+ Visible: true,
204
+ Transform: funcapi.FieldTransformNone,
205
+ Sort: funcapi.FieldSortAscending,
206
+ Summary: funcapi.FieldSummaryCount,
207
+ Filter: funcapi.FieldFilterMultiselect,
208
+ },
209
+ },
210
+ {
211
+ ColumnMeta: funcapi.ColumnMeta{
212
+ Name: "errorNumber",
213
+ Tooltip: "Error Number",
214
+ Type: funcapi.FieldTypeInteger,
215
+ Visible: true,
216
+ Transform: funcapi.FieldTransformNumber,
217
+ Sort: funcapi.FieldSortDescending,
218
+ Summary: funcapi.FieldSummaryMax,
219
+ Filter: funcapi.FieldFilterRange,
220
+ },
221
+ },
222
+ {
223
+ ColumnMeta: funcapi.ColumnMeta{
224
+ Name: "errorState",
225
+ Tooltip: "Error State",
226
+ Type: funcapi.FieldTypeInteger,
227
+ Visible: false,
228
+ Transform: funcapi.FieldTransformNumber,
229
+ Sort: funcapi.FieldSortDescending,
230
+ Summary: funcapi.FieldSummaryMax,
231
+ Filter: funcapi.FieldFilterRange,
232
+ },
233
+ },
234
+ {
235
+ ColumnMeta: funcapi.ColumnMeta{
236
+ Name: "errorMessage",
237
+ Tooltip: "Error Message",
238
+ Type: funcapi.FieldTypeString,
239
+ Visible: true,
240
+ Transform: funcapi.FieldTransformNone,
241
+ Sort: funcapi.FieldSortAscending,
242
+ Summary: funcapi.FieldSummaryCount,
243
+ Filter: funcapi.FieldFilterMultiselect,
244
+ FullWidth: true,
245
+ },
246
+ },
247
+ }
248
+}
249
+
250
+func mssqlPlanAttributionColumns() []topQueriesColumn {
251
+ return []topQueriesColumn{
252
+ {
253
+ ColumnMeta: funcapi.ColumnMeta{
254
+ Name: "hashMatch",
255
+ Tooltip: "Hash Match Joins",
256
+ Type: funcapi.FieldTypeInteger,
257
+ Visible: true,
258
+ Transform: funcapi.FieldTransformNumber,
259
+ Sort: funcapi.FieldSortDescending,
260
+ Summary: funcapi.FieldSummarySum,
261
+ Filter: funcapi.FieldFilterRange,
262
+ },
263
+ },
264
+ {
265
+ ColumnMeta: funcapi.ColumnMeta{
266
+ Name: "mergeJoin",
267
+ Tooltip: "Merge Joins",
268
+ Type: funcapi.FieldTypeInteger,
269
+ Visible: true,
270
+ Transform: funcapi.FieldTransformNumber,
271
+ Sort: funcapi.FieldSortDescending,
272
+ Summary: funcapi.FieldSummarySum,
273
+ Filter: funcapi.FieldFilterRange,
274
+ },
275
+ },
276
+ {
277
+ ColumnMeta: funcapi.ColumnMeta{
278
+ Name: "nestedLoops",
279
+ Tooltip: "Nested Loops",
280
+ Type: funcapi.FieldTypeInteger,
281
+ Visible: true,
282
+ Transform: funcapi.FieldTransformNumber,
283
+ Sort: funcapi.FieldSortDescending,
284
+ Summary: funcapi.FieldSummarySum,
285
+ Filter: funcapi.FieldFilterRange,
286
+ },
287
+ },
288
+ {
289
+ ColumnMeta: funcapi.ColumnMeta{
290
+ Name: "sorts",
291
+ Tooltip: "Sorts",
292
+ Type: funcapi.FieldTypeInteger,
293
+ Visible: true,
294
+ Transform: funcapi.FieldTransformNumber,
295
+ Sort: funcapi.FieldSortDescending,
296
+ Summary: funcapi.FieldSummarySum,
297
+ Filter: funcapi.FieldFilterRange,
298
+ },
299
+ },
300
+ }
301
+}
302
+
303
+func normalizeSQLText(text string) string {
304
+ if strings.TrimSpace(text) == "" {
305
+ return ""
306
+ }
307
+ fields := strings.Fields(text)
308
+ normalized := strings.Join(fields, " ")
309
+ normalized = strings.TrimSpace(normalized)
310
+ normalized = strings.TrimRight(normalized, ";")
311
+ return strings.TrimSpace(normalized)
312
+}
313
+
314
+func rowString(value any) string {
315
+ switch v := value.(type) {
316
+ case nil:
317
+ return ""
318
+ case string:
319
+ return v
320
+ case []byte:
321
+ return string(v)
322
+ default:
323
+ return fmt.Sprint(v)
324
+ }
325
+}
326
+
327
+func nullableString(value string) any {
328
+ if strings.TrimSpace(value) == "" {
329
+ return nil
330
+ }
331
+ return value
332
+}
333
+
334
+func (c *Collector) collectMSSQLErrorDetails(ctx context.Context) (string, map[string]mssqlErrorRow) {
335
+ status, rows, err := c.fetchMSSQLErrorRows(ctx, c.Config.GetErrorInfoSessionName(), c.TopQueriesLimit)
336
+ if err != nil {
337
+ if status == mssqlErrorAttrNotEnabled {
338
+ return mssqlErrorAttrNotEnabled, nil
339
+ }
340
+ mapped := classifyMSSQLErrorAttrError(err)
341
+ c.Debugf("error attribution query failed: %v (status=%s)", err, mapped)
342
+ return mapped, nil
343
+ }
344
+
345
+ if len(rows) == 0 {
346
+ return mssqlErrorAttrNoData, nil
347
+ }
348
+
349
+ out := make(map[string]mssqlErrorRow, len(rows)*2)
350
+ for _, row := range rows {
351
+ if row.QueryHash != "" {
352
+ if _, ok := out[row.QueryHash]; !ok {
353
+ out[row.QueryHash] = row
354
+ }
355
+ }
356
+ key := normalizeSQLText(row.Query)
357
+ if key != "" {
358
+ if _, ok := out[key]; !ok {
359
+ out[key] = row
360
+ }
361
+ }
362
+ }
363
+ return mssqlErrorAttrEnabled, out
364
+}
365
+
366
+func classifyMSSQLErrorAttrError(err error) string {
367
+ if err == nil {
368
+ return mssqlErrorAttrNotEnabled
369
+ }
370
+ if isDeadlockPermissionError(err) {
371
+ return mssqlErrorAttrNotEnabled
372
+ }
373
+ msg := strings.ToLower(err.Error())
374
+ if strings.Contains(msg, "permission") || strings.Contains(msg, "denied") {
375
+ return mssqlErrorAttrNotEnabled
376
+ }
377
+ if strings.Contains(msg, "invalid column") ||
378
+ strings.Contains(msg, "invalid object") ||
379
+ strings.Contains(msg, "could not find") {
380
+ return mssqlErrorAttrNotSupported
381
+ }
382
+ return mssqlErrorAttrNotEnabled
383
+}
384
+
385
+func (c *Collector) collectMSSQLPlanOps(ctx context.Context, data [][]any, cols []topQueriesColumn) map[string]map[string]mssqlPlanOps {
386
+ dbIdx := -1
387
+ hashIdx := -1
388
+ for i, col := range cols {
389
+ switch col.Name {
390
+ case "database":
391
+ dbIdx = i
392
+ case "queryHash":
393
+ hashIdx = i
394
+ }
395
+ }
396
+ if dbIdx < 0 || hashIdx < 0 {
397
+ return map[string]map[string]mssqlPlanOps{}
398
+ }
399
+
400
+ hashesByDB := make(map[string][]string)
401
+ seen := make(map[string]map[string]bool)
402
+ for _, row := range data {
403
+ if dbIdx >= len(row) || hashIdx >= len(row) {
404
+ continue
405
+ }
406
+ dbName := rowString(row[dbIdx])
407
+ queryHash := rowString(row[hashIdx])
408
+ if dbName == "" || queryHash == "" {
409
+ continue
410
+ }
411
+ if seen[dbName] == nil {
412
+ seen[dbName] = make(map[string]bool)
413
+ }
414
+ if seen[dbName][queryHash] {
415
+ continue
416
+ }
417
+ seen[dbName][queryHash] = true
418
+ hashesByDB[dbName] = append(hashesByDB[dbName], queryHash)
419
+ }
420
+
421
+ out := make(map[string]map[string]mssqlPlanOps)
422
+ for dbName, hashes := range hashesByDB {
423
+ ops, err := c.fetchMSSQLPlanOpsForDB(ctx, dbName, hashes)
424
+ if err != nil {
425
+ c.Debugf("plan attribution query failed for %s: %v", dbName, err)
426
+ continue
427
+ }
428
+ out[dbName] = ops
429
+ }
430
+ return out
431
+}
432
+
433
+func (c *Collector) fetchMSSQLPlanOpsForDB(ctx context.Context, dbName string, hashes []string) (map[string]mssqlPlanOps, error) {
434
+ if len(hashes) == 0 {
435
+ return map[string]mssqlPlanOps{}, nil
436
+ }
437
+
438
+ validHashes := make([]string, 0, len(hashes))
439
+ for _, hash := range hashes {
440
+ if strings.HasPrefix(hash, "0x") {
441
+ validHashes = append(validHashes, hash)
442
+ }
443
+ }
444
+ if len(validHashes) == 0 {
445
+ return map[string]mssqlPlanOps{}, nil
446
+ }
447
+
448
+ escapedDB := strings.ReplaceAll(dbName, "]", "]]")
449
+ query := fmt.Sprintf(`
450
+SELECT
451
+ CONVERT(VARCHAR(64), q.query_hash, 1) AS query_hash,
452
+ CAST(p.query_plan AS NVARCHAR(MAX)) AS query_plan
453
+FROM [%s].sys.query_store_query q
454
+INNER JOIN [%s].sys.query_store_plan p ON q.query_id = p.query_id
455
+WHERE q.query_hash IN (%s);
456
+`, escapedDB, escapedDB, strings.Join(validHashes, ","))
457
+
458
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
459
+ defer cancel()
460
+
461
+ rows, err := c.db.QueryContext(qctx, query)
462
+ if err != nil {
463
+ return nil, err
464
+ }
465
+ defer rows.Close()
466
+
467
+ out := make(map[string]mssqlPlanOps)
468
+ for rows.Next() {
469
+ var hash sql.NullString
470
+ var plan sql.NullString
471
+ if err := rows.Scan(&hash, &plan); err != nil {
472
+ return nil, err
473
+ }
474
+ if !hash.Valid || !plan.Valid {
475
+ continue
476
+ }
477
+ ops := countPlanOperators(plan.String)
478
+ current := out[hash.String]
479
+ current.HashMatch += ops.HashMatch
480
+ current.MergeJoin += ops.MergeJoin
481
+ current.NestedLoops += ops.NestedLoops
482
+ current.Sorts += ops.Sorts
483
+ out[hash.String] = current
484
+ }
485
+ if err := rows.Err(); err != nil {
486
+ return nil, err
487
+ }
488
+
489
+ return out, nil
490
+}
491
+
492
+func (c *Collector) fetchMSSQLErrorRows(ctx context.Context, sessionName string, limit int) (string, []mssqlErrorRow, error) {
493
+ if limit <= 0 {
494
+ limit = 500
495
+ }
496
+
497
+ sessionExists, err := c.mssqlErrorSessionAvailable(ctx, sessionName)
498
+ if err != nil {
499
+ return mssqlErrorAttrNotEnabled, nil, err
500
+ }
501
+ if !sessionExists {
502
+ return mssqlErrorAttrNotEnabled, nil, fmt.Errorf("session not found")
503
+ }
504
+
505
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
506
+ defer cancel()
507
+
508
+ rows, err := c.db.QueryContext(qctx, queryMSSQLErrorInfo, sql.Named("sessionName", sessionName), sql.Named("limit", limit))
509
+ if err != nil {
510
+ return mssqlErrorAttrNotSupported, nil, err
511
+ }
512
+ defer rows.Close()
513
+
514
+ var results []mssqlErrorRow
515
+ for rows.Next() {
516
+ var (
517
+ ts sql.NullTime
518
+ errNo sql.NullInt64
519
+ errState sql.NullInt64
520
+ message sql.NullString
521
+ sqlText sql.NullString
522
+ queryHash sql.NullString
523
+ errNoPtr *int64
524
+ errStatePtr *int64
525
+ )
526
+ if err := rows.Scan(&ts, &errNo, &errState, &message, &sqlText, &queryHash); err != nil {
527
+ return mssqlErrorAttrNotSupported, nil, err
528
+ }
529
+ if errNo.Valid {
530
+ val := errNo.Int64
531
+ errNoPtr = &val
532
+ }
533
+ if errState.Valid {
534
+ val := errState.Int64
535
+ errStatePtr = &val
536
+ }
537
+ results = append(results, mssqlErrorRow{
538
+ Time: ts.Time,
539
+ ErrorNumber: errNoPtr,
540
+ ErrorState: errStatePtr,
541
+ Message: message.String,
542
+ Query: sqlText.String,
543
+ QueryHash: queryHash.String,
544
+ })
545
+ }
546
+ if err := rows.Err(); err != nil {
547
+ return mssqlErrorAttrNotSupported, nil, err
548
+ }
549
+
550
+ return mssqlErrorAttrEnabled, results, nil
551
+}
552
+
553
+func (c *Collector) mssqlErrorSessionAvailable(ctx context.Context, sessionName string) (bool, error) {
554
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
555
+ defer cancel()
556
+
557
+ var count int
558
+ err := c.db.QueryRowContext(qctx, queryMSSQLErrorSessionExists, sql.Named("sessionName", sessionName)).Scan(&count)
559
+ if err != nil {
560
+ return false, err
561
+ }
562
+ if count == 0 {
563
+ return false, nil
564
+ }
565
+
566
+ err = c.db.QueryRowContext(qctx, queryMSSQLErrorSessionHasRingBuffer, sql.Named("sessionName", sessionName)).Scan(&count)
567
+ if err != nil {
568
+ return false, err
569
+ }
570
+ return count > 0, nil
571
+}
572
+
573
+func countPlanOperators(planXML string) mssqlPlanOps {
574
+ var ops mssqlPlanOps
575
+ if strings.TrimSpace(planXML) == "" {
576
+ return ops
577
+ }
578
+ decoder := xml.NewDecoder(strings.NewReader(planXML))
579
+ for {
580
+ tok, err := decoder.Token()
581
+ if err != nil {
582
+ break
583
+ }
584
+ start, ok := tok.(xml.StartElement)
585
+ if !ok || start.Name.Local != "RelOp" {
586
+ continue
587
+ }
588
+ for _, attr := range start.Attr {
589
+ if attr.Name.Local != "PhysicalOp" {
590
+ continue
591
+ }
592
+ switch strings.ToLower(strings.TrimSpace(attr.Value)) {
593
+ case "hash match":
594
+ ops.HashMatch++
595
+ case "merge join":
596
+ ops.MergeJoin++
597
+ case "nested loops":
598
+ ops.NestedLoops++
599
+ case "sort":
600
+ ops.Sorts++
601
+ }
602
+ break
603
+ }
604
+ }
605
+ return ops
606
+}
src/go/plugin/go.d/collector/mssql/func_router.go
+4
@@ -23,6 +23,8 @@ func newFuncRouter(c *Collector) *funcRouter {
23
handlers: make(map[string]funcapi.MethodHandler),
24
}
25
r.handlers[topQueriesMethodID] = newFuncTopQueries(r)
26
+ r.handlers[deadlockInfoMethodID] = newFuncDeadlockInfo(r)
27
+ r.handlers[errorInfoMethodID] = newFuncErrorInfo(r)
28
return r
29
}
30
@@ -52,6 +54,8 @@ func (r *funcRouter) Cleanup(ctx context.Context) {
54
func mssqlMethods() []funcapi.MethodConfig {
55
return []funcapi.MethodConfig{
56
topQueriesMethodConfig(),
57
+ deadlockInfoMethodConfig(),
58
+ errorInfoMethodConfig(),
59
}
60
}
61
src/go/plugin/go.d/collector/mssql/func_top_queries.go
+87
@@ -288,6 +288,93 @@ func (f *funcTopQueries) collectData(ctx context.Context, sortColumn string) *fu
288
return &funcapi.FunctionResponse{Status: 500, Message: err.Error()}
289
}
290
291
+ errorStatus, errorDetails := f.router.collector.collectMSSQLErrorDetails(ctx)
292
+ planOpsByDB := f.router.collector.collectMSSQLPlanOps(ctx, data, cols)
293
+ extraCols := append(mssqlErrorAttributionColumns(), mssqlPlanAttributionColumns()...)
294
+
295
+ queryIdx := -1
296
+ queryHashIdx := -1
297
+ dbIdx := -1
298
+ for i, col := range cols {
299
+ switch col.Name {
300
+ case "query":
301
+ queryIdx = i
302
+ case "queryHash":
303
+ queryHashIdx = i
304
+ case "database":
305
+ dbIdx = i
306
+ }
307
+ }
308
+
309
+ if len(extraCols) > 0 {
310
+ for i := range data {
311
+ status := errorStatus
312
+ var errRow mssqlErrorRow
313
+ if errorStatus == mssqlErrorAttrEnabled {
314
+ found := false
315
+ if queryHashIdx >= 0 && queryHashIdx < len(data[i]) {
316
+ queryHash := rowString(data[i][queryHashIdx])
317
+ if queryHash != "" {
318
+ if row, ok := errorDetails[queryHash]; ok {
319
+ status = mssqlErrorAttrEnabled
320
+ errRow = row
321
+ found = true
322
+ }
323
+ }
324
+ }
325
+ if !found && queryIdx >= 0 && queryIdx < len(data[i]) {
326
+ queryText := normalizeSQLText(rowString(data[i][queryIdx]))
327
+ if queryText != "" {
328
+ if row, ok := errorDetails[queryText]; ok {
329
+ status = mssqlErrorAttrEnabled
330
+ errRow = row
331
+ found = true
332
+ }
333
+ }
334
+ }
335
+ if !found {
336
+ status = mssqlErrorAttrNoData
337
+ }
338
+ }
339
+
340
+ var hashMatch, mergeJoin, nestedLoops, sorts any
341
+ if dbIdx >= 0 && dbIdx < len(data[i]) && queryHashIdx >= 0 && queryHashIdx < len(data[i]) {
342
+ dbName := rowString(data[i][dbIdx])
343
+ queryHash := rowString(data[i][queryHashIdx])
344
+ if dbName != "" && queryHash != "" {
345
+ if opsByHash, ok := planOpsByDB[dbName]; ok {
346
+ if ops, ok := opsByHash[queryHash]; ok {
347
+ hashMatch = ops.HashMatch
348
+ mergeJoin = ops.MergeJoin
349
+ nestedLoops = ops.NestedLoops
350
+ sorts = ops.Sorts
351
+ }
352
+ }
353
+ }
354
+ }
355
+
356
+ var errNo any
357
+ if errRow.ErrorNumber != nil {
358
+ errNo = *errRow.ErrorNumber
359
+ }
360
+ var errState any
361
+ if errRow.ErrorState != nil {
362
+ errState = *errRow.ErrorState
363
+ }
364
+ data[i] = append(data[i],
365
+ status,
366
+ errNo,
367
+ errState,
368
+ nullableString(rowString(errRow.Message)),
369
+ hashMatch,
370
+ mergeJoin,
371
+ nestedLoops,
372
+ sorts,
373
+ )
374
+ }
375
+ cols = append(cols, extraCols...)
376
+ }
377
+
378
sortParam, sortOptions := f.buildSortParam(cols)
379
380
defaultSort := ""
src/go/plugin/go.d/collector/mssql/func_top_queries_test.go
+38
-10
@@ -7,26 +7,54 @@ import (
7
8
"github.com/netdata/netdata/go/plugins/pkg/funcapi"
9
"github.com/stretchr/testify/assert"
10
+ "github.com/stretchr/testify/require"
11
)
12
13
func TestMSSQLMethods(t *testing.T) {
14
methods := mssqlMethods()
15
15
- require := assert.New(t)
16
- require.Len(methods, 1)
17
- require.Equal("top-queries", methods[0].ID)
18
- require.Equal("Top Queries", methods[0].Name)
19
- require.NotEmpty(methods[0].RequiredParams)
16
+ req := require.New(t)
17
+ req.Len(methods, 3)
18
+
19
+ topIdx := -1
20
+ deadlockIdx := -1
21
+ errorIdx := -1
22
+ for i := range methods {
23
+ switch methods[i].ID {
24
+ case "top-queries":
25
+ topIdx = i
26
+ case "deadlock-info":
27
+ deadlockIdx = i
28
+ case "error-info":
29
+ errorIdx = i
30
+ }
31
+ }
32
+
33
+ req.NotEqual(-1, topIdx, "expected top-queries method")
34
+ req.NotEqual(-1, deadlockIdx, "expected deadlock-info method")
35
+ req.NotEqual(-1, errorIdx, "expected error-info method")
36
+
37
+ topMethod := methods[topIdx]
38
+ req.Equal("Top Queries", topMethod.Name)
39
+ req.NotEmpty(topMethod.RequiredParams)
40
+
41
+ deadlockMethod := methods[deadlockIdx]
42
+ req.Equal("Deadlock Info", deadlockMethod.Name)
43
+ req.Empty(deadlockMethod.RequiredParams)
44
+
45
+ errorMethod := methods[errorIdx]
46
+ req.Equal("Error Info", errorMethod.Name)
47
+ req.Empty(errorMethod.RequiredParams)
48
49
var sortParam *funcapi.ParamConfig
22
- for i := range methods[0].RequiredParams {
23
- if methods[0].RequiredParams[i].ID == "__sort" {
24
- sortParam = &methods[0].RequiredParams[i]
50
+ for i := range topMethod.RequiredParams {
51
+ if topMethod.RequiredParams[i].ID == "__sort" {
52
+ sortParam = &topMethod.RequiredParams[i]
53
break
54
}
55
}
28
- require.NotNil(sortParam, "expected __sort required param")
29
- require.NotEmpty(sortParam.Options)
56
+ req.NotNil(sortParam, "expected __sort required param")
57
+ req.NotEmpty(sortParam.Options)
58
}
59
60
func TestTopQueriesColumns_HasRequiredColumns(t *testing.T) {
src/go/plugin/go.d/collector/mssql/integrations/microsoft_sql_server.md
renamed
+134
-7
@@ -28,7 +28,7 @@ It collects metrics from:
28
- Performance counters (buffer manager, memory manager, SQL statistics)
29
- Dynamic management views (DMVs) for wait statistics, locks, and sessions
30
- Per-database transaction and lock statistics
31
-- SQL Server Agent job status (if permissions allow)
31
+- SQL Server Agent job status
32
33
34
It connects to the SQL Server instance via TCP using the go-mssqldb driver and executes queries against:
@@ -41,7 +41,7 @@ It connects to the SQL Server instance via TCP using the go-mssqldb driver and e
41
- `sys.dm_os_process_memory` - SQL Server process memory
42
- `sys.dm_os_sys_memory` - OS physical memory and page file
43
- `sys.master_files` - Database file sizes
44
-- `msdb.dbo.sysjobs` - SQL Agent job status (optional)
44
+- `msdb.dbo.sysjobs` - SQL Agent job status
45
46
47
This collector is supported on all platforms.
@@ -49,7 +49,8 @@ This collector is supported on all platforms.
49
This collector supports collecting metrics from multiple instances of this integration, including remote instances.
50
51
The monitoring user requires the VIEW SERVER STATE permission to access DMVs.
52
-For SQL Agent job monitoring, access to the msdb database is required.
52
+For SQL Agent job monitoring (queried during collector startup), access to
53
+`msdb.dbo.sysjobs` is required.
54
55
56
### Default Behavior
@@ -316,6 +317,14 @@ Aggregated query execution statistics from Query Store runtime views, providing
317
| Query | string | | | The SQL query text with literal values truncated at 4096 characters. Use this to identify the actual SQL being executed and spot parameterized queries or injection risks. |
318
| Database | string | | | Database name where the query was executed. Essential for multi-database analysis to identify which database is experiencing query load. |
319
| Calls | integer | | | Total number of times this query pattern has been executed. High values indicate frequently run queries that may impact server performance significantly. |
320
+| Error Attribution | string | | | Status of error detail attribution for this query. Values: enabled, no_data, not_enabled, not_supported. |
321
+| Error Number | integer | | | Most recent error number observed for this query (when error attribution is enabled). |
322
+| Error State | integer | | hidden | SQL Server error state for the most recent error (when error attribution is enabled). |
323
+| Error Message | string | | | Most recent error message for this query (when error attribution is enabled). |
324
+| Hash Match Joins | integer | | | Count of Hash Match join operators across all stored plans for this query. |
325
+| Merge Joins | integer | | | Count of Merge Join operators across all stored plans for this query. |
326
+| Nested Loops | integer | | | Count of Nested Loops operators across all stored plans for this query. |
327
+| Sorts | integer | | | Count of Sort operators across all stored plans for this query. |
328
| Total Time | duration | milliseconds | | Cumulative execution time across all query executions. This is a key metric for identifying the most resource-intensive queries in terms of total server time consumption. |
329
| Avg Time | duration | milliseconds | | Average execution time per query run, calculated as weighted average when execution count is greater than zero. Compare with Total Time to determine if individual executions or high frequency drives resource usage. |
330
| Last Time | duration | milliseconds | hidden | Execution time of the most recent execution for this query pattern. Useful for identifying recent performance changes or individual outlier executions. |
@@ -374,6 +383,126 @@ Aggregated query execution statistics from Query Store runtime views, providing
383
| StdDev TempDB | float | | hidden | Standard deviation of tempdb space usage. High variability suggests inconsistent temporary object usage patterns, potentially varying by query complexity, parameter types, or different data access patterns affecting temporary object creation. |
384
385
386
+### Deadlock Info
387
+
388
+Retrieves the most recent deadlock event from SQL Server's `system_health` Extended Events ring buffer (`xml_deadlock_report`).
389
+
390
+The deadlock graph XML is parsed to attribute the deadlock to the participating processes and their query text, lock mode, lock status, and wait resource.
391
+
392
+Use cases:
393
+- Identify which process was chosen as the deadlock victim
394
+- Inspect the waiting resource and lock mode involved in the deadlock
395
+- Correlate deadlocks with recent application changes or deployments
396
+
397
+Query text and wait resource strings are truncated at 4096 characters for display purposes.
398
+
399
+
400
+| Aspect | Description |
401
+|:-------|:------------|
402
+| Name | `Mssql:deadlock-info` |
403
+| Require Cloud | yes |
404
+| Performance | Executes on-demand queries against the `system_health` ring buffer:<br/>• Not part of regular metric collection<br/>• Overhead is limited to function execution time and XML parsing |
405
+| Security | Query text and wait resource strings may include unmasked literal values including sensitive data (PII/secrets):<br/>• SQL literals such as emails, IDs, or tokens<br/>• Schema and table names that may be sensitive in some environments<br/>• Restrict dashboard access to authorized personnel only |
406
+| Availability | Available when:<br/>• The collector has successfully connected to SQL Server<br/>• `deadlock_info_function_enabled` is true<br/>• The account has `VIEW SERVER STATE` permission<br/>• Returns HTTP 200 with empty data when no deadlock is found<br/>• Returns HTTP 403 when permission is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 561 when the deadlock graph cannot be parsed<br/>• Returns HTTP 503 if the collector is still initializing or the function is disabled<br/>• Returns HTTP 504 if the query times out |
407
+
408
+#### Prerequisites
409
+
410
+1. Ensure the account has the required permission:
411
+ ```sql
412
+ GRANT VIEW SERVER STATE TO [netdata];
413
+ ```
414
+2. Enable the function in Netdata collector config:
415
+ ```yaml
416
+ jobs:
417
+ - name: local
418
+ dsn: "sqlserver://user:pass@localhost:1433"
419
+ deadlock_info_function_enabled: true
420
+ ```
421
+3. Verify the deadlock source is accessible:
422
+ ```sql
423
+ SELECT name
424
+ FROM sys.dm_xe_sessions
425
+ WHERE name = 'system_health';
426
+ ```
427
+
428
+#### Parameters
429
+
430
+This function has no parameters.
431
+
432
+#### Returns
433
+
434
+Parsed deadlock participants from the latest detected deadlock event. Each row represents one process involved in the deadlock.
435
+
436
+| Column | Type | Unit | Visibility | Description |
437
+|:-------|:-----|:-----|:-----------|:------------|
438
+| Row ID | string | | hidden | Unique row identifier composed of deadlock ID and process ID. |
439
+| Deadlock ID | string | | | Identifier for the deadlock event, derived from the deadlock timestamp to group participating processes. |
440
+| Timestamp | timestamp | | | Timestamp of the deadlock event from the ring buffer when available; otherwise the function execution time. |
441
+| Process ID | string | | | Deadlock graph process identifier for the process involved in the deadlock. |
442
+| SPID | integer | | | SQL Server session ID (SPID) for the process when available. |
443
+| ECID | integer | | | Execution context ID (ECID) for parallel execution contexts when available. |
444
+| Victim | string | | | "true" when the process was chosen as the deadlock victim and rolled back; otherwise "false". |
445
+| Query | string | | | SQL query text for the process involved in the deadlock. Truncated to 4096 characters. |
446
+| Lock Mode | string | | | Lock mode reported for the process within the deadlock graph (for example X or S). |
447
+| Lock Status | string | | | Lock status for the process. WAITING indicates the process was waiting on a lock. |
448
+| Wait Resource | string | | | Lock resource identifier from the deadlock graph showing what the process was waiting on. |
449
+| Database | string | | | Database name mapped from the deadlock graph database ID when available. |
450
+
451
+### Error Info
452
+
453
+Retrieves recent SQL errors from a user-managed Extended Events session that captures `sqlserver.error_reported`
454
+with the `sql_text` and `query_hash` actions (query_hash enables reliable mapping to top-queries).
455
+
456
+| Aspect | Description |
457
+|:-------|:------------|
458
+| Name | `Mssql:error-info` |
459
+| Require Cloud | yes |
460
+| Performance | Executes on-demand queries against the configured Extended Events ring buffer:<br/>• Not part of regular metric collection<br/>• Overhead is limited to function execution time and XML parsing |
461
+| Security | Error messages and query text may include unmasked literal values including sensitive data (PII/secrets):<br/>• Restrict dashboard access to authorized personnel only |
462
+| Availability | Available when:<br/>• The collector has successfully connected to SQL Server<br/>• `error_info_function_enabled` is true<br/>• The Extended Events session exists and has a ring_buffer target<br/>• The account has `VIEW SERVER STATE` permission<br/>• Returns HTTP 200 with empty data when no errors are found<br/>• Returns HTTP 403 when permission is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 503 if the session is not enabled or the function is disabled<br/>• Returns HTTP 504 if the query times out |
463
+
464
+#### Prerequisites
465
+
466
+1. Create an Extended Events session (admin-controlled) that captures `sqlserver.error_reported` with `sql_text` and `query_hash`:
467
+ ```sql
468
+ CREATE EVENT SESSION [netdata_errors] ON SERVER
469
+ ADD EVENT sqlserver.error_reported(
470
+ ACTION(sqlserver.sql_text, sqlserver.query_hash)
471
+ )
472
+ ADD TARGET package0.ring_buffer;
473
+ GO
474
+ ALTER EVENT SESSION [netdata_errors] ON SERVER STATE = START;
475
+ ```
476
+2. Ensure the account has the required permission:
477
+ ```sql
478
+ GRANT VIEW SERVER STATE TO [netdata];
479
+ ```
480
+3. Enable the function and (optionally) set the session name in Netdata config:
481
+ ```yaml
482
+ jobs:
483
+ - name: local
484
+ dsn: "sqlserver://user:pass@localhost:1433"
485
+ error_info_function_enabled: true
486
+ error_info_session_name: netdata_errors
487
+ ```
488
+
489
+#### Parameters
490
+
491
+This function has no parameters.
492
+
493
+#### Returns
494
+
495
+Recent error events from the configured Extended Events session.
496
+
497
+| Column | Type | Unit | Visibility | Description |
498
+|:-------|:-----|:-----|:-----------|:------------|
499
+| Timestamp | timestamp | | | Timestamp of the error event. |
500
+| Error Number | integer | | | SQL Server error number. |
501
+| Error State | integer | | | SQL Server error state. |
502
+| Error Message | string | | | Error message text. |
503
+| Query | string | | | SQL text captured with the error event. |
504
+| Query Hash | string | | hidden | Query hash captured with the error event (used for mapping into top-queries). |
505
+
506
507
## Alerts
508
@@ -410,7 +539,7 @@ CREATE LOGIN netdata_user WITH PASSWORD = 'YourStrongPassword!';
539
-- Grant VIEW SERVER STATE (required for DMVs)
540
GRANT VIEW SERVER STATE TO netdata_user;
541
542
+-- Grant access to msdb for SQL Agent job monitoring (required)
543
USE msdb;
544
CREATE USER netdata_user FOR LOGIN netdata_user;
545
GRANT SELECT ON dbo.sysjobs TO netdata_user;
@@ -426,9 +555,9 @@ GRANT SELECT ON dbo.MSsubscriptions TO netdata_user;
555
556
**Required permissions:**
557
- `VIEW SERVER STATE` - Access to dynamic management views
558
+- `SELECT on msdb.dbo.sysjobs` - SQL Agent job status monitoring
559
560
**Optional permissions:**
431
-- `SELECT on msdb.dbo.sysjobs` - SQL Agent job status monitoring
561
- `SELECT on distribution.dbo.MSreplication_monitordata` - Replication monitoring
562
- `SELECT on distribution.dbo.MSpublications` - Publication information
563
- `SELECT on distribution.dbo.MSsubscriptions` - Subscription counts
@@ -678,6 +807,3 @@ Ensure SQL Server is configured for mixed mode authentication if using SQL login
807
808
The monitoring user needs VIEW SERVER STATE permission.
809
Grant it with: `GRANT VIEW SERVER STATE TO netdata_user;`
681
-
682
-
683
-
src/go/plugin/go.d/collector/mssql/metadata.yaml
+162
-5
@@ -37,7 +37,7 @@ modules:
37
- Performance counters (buffer manager, memory manager, SQL statistics)
38
- Dynamic management views (DMVs) for wait statistics, locks, and sessions
39
- Per-database transaction and lock statistics
40
- - SQL Server Agent job status (if permissions allow)
40
+ - SQL Server Agent job status
41
method_description: |
42
It connects to the SQL Server instance via TCP using the go-mssqldb driver and executes queries against:
43
@@ -49,7 +49,7 @@ modules:
49
- `sys.dm_os_process_memory` - SQL Server process memory
50
- `sys.dm_os_sys_memory` - OS physical memory and page file
51
- `sys.master_files` - Database file sizes
52
- - `msdb.dbo.sysjobs` - SQL Agent job status (optional)
52
+ - `msdb.dbo.sysjobs` - SQL Agent job status
53
default_behavior:
54
auto_detection:
55
description: |
@@ -64,7 +64,8 @@ modules:
64
additional_permissions:
65
description: |
66
The monitoring user requires the VIEW SERVER STATE permission to access DMVs.
67
- For SQL Agent job monitoring, access to the msdb database is required.
67
+ SQL Agent job monitoring is part of collector startup, so access to
68
+ `msdb.dbo.sysjobs` is required.
69
supported_platforms:
70
include: []
71
exclude: []
@@ -82,7 +83,7 @@ modules:
83
-- Grant VIEW SERVER STATE (required for DMVs)
84
GRANT VIEW SERVER STATE TO netdata_user;
85
85
- -- Optional: Grant access to msdb for SQL Agent job monitoring
86
+ -- Grant access to msdb for SQL Agent job monitoring (required)
87
USE msdb;
88
CREATE USER netdata_user FOR LOGIN netdata_user;
89
GRANT SELECT ON dbo.sysjobs TO netdata_user;
@@ -98,9 +99,9 @@ modules:
99
100
**Required permissions:**
101
- `VIEW SERVER STATE` - Access to dynamic management views
102
+ - `SELECT on msdb.dbo.sysjobs` - SQL Agent job status monitoring
103
104
**Optional permissions:**
103
- - `SELECT on msdb.dbo.sysjobs` - SQL Agent job status monitoring
105
- `SELECT on distribution.dbo.MSreplication_monitordata` - Replication monitoring
106
- `SELECT on distribution.dbo.MSpublications` - Publication information
107
- `SELECT on distribution.dbo.MSsubscriptions` - Subscription counts
@@ -271,6 +272,39 @@ modules:
272
type: integer
273
unit: ""
274
description: Total number of times this query pattern has been executed. High values indicate frequently run queries that may impact server performance significantly.
275
+ - name: Error Attribution
276
+ type: string
277
+ unit: ""
278
+ description: "Status of error detail attribution for this query. Values: enabled, no_data, not_enabled, not_supported."
279
+ - name: Error Number
280
+ type: integer
281
+ unit: ""
282
+ description: "Most recent error number observed for this query (when error attribution is enabled)."
283
+ - name: Error State
284
+ type: integer
285
+ unit: ""
286
+ visibility: hidden
287
+ description: "SQL Server error state for the most recent error (when error attribution is enabled)."
288
+ - name: Error Message
289
+ type: string
290
+ unit: ""
291
+ description: "Most recent error message for this query (when error attribution is enabled)."
292
+ - name: Hash Match Joins
293
+ type: integer
294
+ unit: ""
295
+ description: "Count of Hash Match join operators across all stored plans for this query."
296
+ - name: Merge Joins
297
+ type: integer
298
+ unit: ""
299
+ description: "Count of Merge Join operators across all stored plans for this query."
300
+ - name: Nested Loops
301
+ type: integer
302
+ unit: ""
303
+ description: "Count of Nested Loops operators across all stored plans for this query."
304
+ - name: Sorts
305
+ type: integer
306
+ unit: ""
307
+ description: "Count of Sort operators across all stored plans for this query."
308
- name: Total Time
309
type: duration
310
unit: "milliseconds"
@@ -582,6 +616,129 @@ modules:
616
availability: |
617
Available when:<br/>• The collector has successfully connected to SQL Server<br/>• Query Store is enabled on at least one user database<br/>• Returns HTTP 503 if collector is still initializing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out
618
require_cloud: true
619
+ - id: deadlock-info
620
+ name: Deadlock Info
621
+ description: |
622
+ Retrieves the most recent deadlock event from SQL Server's `system_health` Extended Events ring buffer (`xml_deadlock_report`).
623
+
624
+ The deadlock graph XML is parsed to attribute the deadlock to the participating processes and their query text, lock mode, lock status, and wait resource.
625
+
626
+ Use cases:
627
+ - Identify which process was chosen as the deadlock victim
628
+ - Inspect the waiting resource and lock mode involved in the deadlock
629
+ - Correlate deadlocks with recent application changes or deployments
630
+
631
+ Query text and wait resource strings are truncated at 4096 characters for display purposes.
632
+ parameters: []
633
+ returns:
634
+ description: Parsed deadlock participants from the latest detected deadlock event. Each row represents one process involved in the deadlock.
635
+ columns:
636
+ - name: Row ID
637
+ type: string
638
+ unit: ""
639
+ visibility: hidden
640
+ description: "Unique row identifier composed of deadlock ID and process ID."
641
+ - name: Deadlock ID
642
+ type: string
643
+ unit: ""
644
+ description: "Identifier for the deadlock event, derived from the deadlock timestamp to group participating processes."
645
+ - name: Timestamp
646
+ type: timestamp
647
+ unit: ""
648
+ description: "Timestamp of the deadlock event from the ring buffer when available; otherwise the function execution time."
649
+ - name: Process ID
650
+ type: string
651
+ unit: ""
652
+ description: "Deadlock graph process identifier for the process involved in the deadlock."
653
+ - name: SPID
654
+ type: integer
655
+ unit: ""
656
+ description: "SQL Server session ID (SPID) for the process when available."
657
+ - name: ECID
658
+ type: integer
659
+ unit: ""
660
+ description: "Execution context ID (ECID) for parallel execution contexts when available."
661
+ - name: Victim
662
+ type: string
663
+ unit: ""
664
+ description: "\"true\" when the process was chosen as the deadlock victim and rolled back; otherwise \"false\"."
665
+ - name: Query
666
+ type: string
667
+ unit: ""
668
+ description: "SQL query text for the process involved in the deadlock. Truncated to 4096 characters."
669
+ - name: Lock Mode
670
+ type: string
671
+ unit: ""
672
+ description: "Lock mode reported for the process within the deadlock graph (for example X or S)."
673
+ - name: Lock Status
674
+ type: string
675
+ unit: ""
676
+ description: "Lock status for the process. WAITING indicates the process was waiting on a lock."
677
+ - name: Wait Resource
678
+ type: string
679
+ unit: ""
680
+ description: "Lock resource identifier from the deadlock graph showing what the process was waiting on."
681
+ - name: Database
682
+ type: string
683
+ unit: ""
684
+ description: "Database name mapped from the deadlock graph database ID when available."
685
+ performance: |
686
+ Executes on-demand queries against the `system_health` ring buffer:<br/>• Not part of regular metric collection<br/>• Overhead is limited to function execution time and XML parsing
687
+ security: |
688
+ Query text and wait resource strings may include unmasked literal values including sensitive data (PII/secrets):<br/>• SQL literals such as emails, IDs, or tokens<br/>• Schema and table names that may be sensitive in some environments<br/>• Restrict dashboard access to authorized personnel only
689
+ availability: |
690
+ Available when:<br/>• The collector has successfully connected to SQL Server<br/>• `deadlock_info_function_enabled` is true<br/>• The account has `VIEW SERVER STATE` permission<br/>• Returns HTTP 200 with empty data when no deadlock is found<br/>• Returns HTTP 403 when permission is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 561 when the deadlock graph cannot be parsed<br/>• Returns HTTP 503 if the collector is still initializing or the function is disabled<br/>• Returns HTTP 504 if the query times out
691
+ require_cloud: true
692
+ - id: error-info
693
+ name: Error Info
694
+ description: |
695
+ Retrieves recent SQL errors from a user-managed Extended Events session that captures `sqlserver.error_reported`
696
+ with both the `sql_text` and `query_hash` actions.
697
+
698
+ The session must be created by an administrator and include a `ring_buffer` target. Netdata reads the ring buffer
699
+ and returns recent error events with error number, message, and SQL text. The `query_hash` action is required for
700
+ reliable mapping into `top-queries` (query text fallback is best-effort).
701
+
702
+ Use cases:
703
+ - Identify recent query errors and their messages
704
+ - Correlate errors to query text
705
+ - Validate error rates seen in top-queries
706
+ parameters: []
707
+ returns:
708
+ description: Recent error events from the configured Extended Events session.
709
+ columns:
710
+ - name: Timestamp
711
+ type: timestamp
712
+ unit: ""
713
+ description: "Timestamp of the error event."
714
+ - name: Error Number
715
+ type: integer
716
+ unit: ""
717
+ description: "SQL Server error number."
718
+ - name: Error State
719
+ type: integer
720
+ unit: ""
721
+ description: "SQL Server error state."
722
+ - name: Error Message
723
+ type: string
724
+ unit: ""
725
+ description: "Error message text."
726
+ - name: Query
727
+ type: string
728
+ unit: ""
729
+ description: "SQL text captured with the error event."
730
+ - name: Query Hash
731
+ type: string
732
+ unit: ""
733
+ visibility: hidden
734
+ description: "Query hash captured with the error event (used for mapping into top-queries)."
735
+ performance: |
736
+ Executes on-demand queries against the configured Extended Events ring buffer:<br/>• Not part of regular metric collection<br/>• Overhead is limited to function execution time and XML parsing
737
+ security: |
738
+ Error messages and query text may include unmasked literal values including sensitive data (PII/secrets):<br/>• Restrict dashboard access to authorized personnel only
739
+ availability: |
740
+ Available when:<br/>• The collector has successfully connected to SQL Server<br/>• `error_info_function_enabled` is true<br/>• The Extended Events session exists and has a ring_buffer target<br/>• The account has `VIEW SERVER STATE` permission<br/>• Returns HTTP 200 with empty data when no errors are found<br/>• Returns HTTP 403 when permission is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 503 if the session is not enabled or the function is disabled<br/>• Returns HTTP 504 if the query times out
741
+ require_cloud: true
742
metrics:
743
folding:
744
title: Metrics
src/go/plugin/go.d/collector/mssql/queries.go
+61
@@ -201,12 +201,73 @@ WHERE object_name LIKE '%SQL Errors%'
201
AND counter_name = 'Errors/sec';
202
`
203
204
+// querySystemHealthLatestDeadlock retrieves the latest xml_deadlock_report event
205
+// from the system_health Extended Events ring buffer.
206
+const querySystemHealthLatestDeadlock = `
207
+WITH xevents AS (
208
+ SELECT CAST(xet.target_data AS XML) AS target_data
209
+ FROM sys.dm_xe_session_targets AS xet
210
+ JOIN sys.dm_xe_sessions AS xs ON xs.address = xet.event_session_address
211
+ WHERE xs.name = 'system_health'
212
+ AND xet.target_name = 'ring_buffer'
213
+)
214
+SELECT TOP (1)
215
+ xevent.value('@timestamp', 'datetime2(7)') AS deadlock_time,
216
+ CONVERT(nvarchar(max), xevent.query('(data/value/deadlock)[1]')) AS deadlock_xml
217
+FROM xevents
218
+CROSS APPLY target_data.nodes('RingBufferTarget/event[@name="xml_deadlock_report"]') AS T(xevent)
219
+ORDER BY deadlock_time DESC;
220
+`
221
+
222
// queryDatabaseStatus gets database state and read-only status
223
const queryDatabaseStatus = `
224
SELECT name, state, is_read_only
225
FROM sys.databases;
226
`
227
228
+// queryDatabaseNamesByID retrieves database_id to name mappings.
229
+const queryDatabaseNamesByID = `
230
+SELECT database_id, name
231
+FROM sys.databases;
232
+`
233
+
234
+// queryMSSQLErrorSessionExists checks for the configured Extended Events session.
235
+const queryMSSQLErrorSessionExists = `
236
+SELECT COUNT(*)
237
+FROM sys.dm_xe_sessions
238
+WHERE name = @sessionName;
239
+`
240
+
241
+// queryMSSQLErrorSessionHasRingBuffer verifies that the session has a ring_buffer target.
242
+const queryMSSQLErrorSessionHasRingBuffer = `
243
+SELECT COUNT(*)
244
+FROM sys.dm_xe_session_targets AS xet
245
+JOIN sys.dm_xe_sessions AS xs ON xs.address = xet.event_session_address
246
+WHERE xs.name = @sessionName
247
+ AND xet.target_name = 'ring_buffer';
248
+`
249
+
250
+// queryMSSQLErrorInfo reads recent error_reported events from the ring_buffer target.
251
+const queryMSSQLErrorInfo = `
252
+WITH xevents AS (
253
+ SELECT CAST(xet.target_data AS XML) AS target_data
254
+ FROM sys.dm_xe_session_targets AS xet
255
+ JOIN sys.dm_xe_sessions AS xs ON xs.address = xet.event_session_address
256
+ WHERE xs.name = @sessionName
257
+ AND xet.target_name = 'ring_buffer'
258
+)
259
+SELECT TOP (@limit)
260
+ xevent.value('@timestamp', 'datetime2(7)') AS event_time,
261
+ xevent.value('(data[@name="error_number"]/value)[1]', 'int') AS error_number,
262
+ xevent.value('(data[@name="state"]/value)[1]', 'int') AS error_state,
263
+ xevent.value('(data[@name="message"]/value)[1]', 'nvarchar(max)') AS message,
264
+ xevent.value('(action[@name="sql_text"]/value)[1]', 'nvarchar(max)') AS sql_text,
265
+ CONVERT(VARCHAR(64), xevent.value('(action[@name="query_hash"]/value)[1]', 'varbinary(8)'), 1) AS query_hash
266
+FROM xevents
267
+CROSS APPLY target_data.nodes('RingBufferTarget/event[@name="error_reported"]') AS T(xevent)
268
+ORDER BY event_time DESC;
269
+`
270
+
271
// queryReplicationStatus gets replication publication status (if configured)
272
// Groups by publication to aggregate across agent types and excludes 'ALL' placeholder
273
const queryReplicationStatus = `
src/go/plugin/go.d/collector/mysql/collector.go
+25
-7
@@ -63,13 +63,15 @@ func New() *Collector {
63
}
64
65
type Config struct {
66
- Vnode string `yaml:"vnode,omitempty" json:"vnode"`
67
- UpdateEvery int `yaml:"update_every,omitempty" json:"update_every"`
68
- AutoDetectionRetry int `yaml:"autodetection_retry,omitempty" json:"autodetection_retry"`
69
- DSN string `yaml:"dsn" json:"dsn"`
70
- MyCNF string `yaml:"my.cnf,omitempty" json:"my.cnf"`
71
- Timeout confopt.Duration `yaml:"timeout,omitempty" json:"timeout"`
72
- TopQueriesLimit int `yaml:"top_queries_limit,omitempty" json:"top_queries_limit,omitempty"`
66
+ Vnode string `yaml:"vnode,omitempty" json:"vnode"`
67
+ UpdateEvery int `yaml:"update_every,omitempty" json:"update_every"`
68
+ AutoDetectionRetry int `yaml:"autodetection_retry,omitempty" json:"autodetection_retry"`
69
+ DSN string `yaml:"dsn" json:"dsn"`
70
+ MyCNF string `yaml:"my.cnf,omitempty" json:"my.cnf"`
71
+ Timeout confopt.Duration `yaml:"timeout,omitempty" json:"timeout"`
72
+ TopQueriesLimit int `yaml:"top_queries_limit,omitempty" json:"top_queries_limit,omitempty"`
73
+ DeadlockInfoFunctionEnabled *bool `yaml:"deadlock_info_function_enabled,omitempty" json:"deadlock_info_function_enabled,omitempty"`
74
+ ErrorInfoFunctionEnabled *bool `yaml:"error_info_function_enabled,omitempty" json:"error_info_function_enabled,omitempty"`
75
}
76
77
type Collector struct {
@@ -120,6 +122,22 @@ func (c *Collector) Configuration() any {
122
return c.Config
123
}
124
125
+// GetDeadlockInfoFunctionEnabled returns whether the deadlock-info function is enabled (default: true).
126
+func (c *Config) GetDeadlockInfoFunctionEnabled() bool {
127
+ if c.DeadlockInfoFunctionEnabled == nil {
128
+ return true
129
+ }
130
+ return *c.DeadlockInfoFunctionEnabled
131
+}
132
+
133
+// GetErrorInfoFunctionEnabled returns whether the error-info function is enabled (default: true).
134
+func (c *Config) GetErrorInfoFunctionEnabled() bool {
135
+ if c.ErrorInfoFunctionEnabled == nil {
136
+ return true
137
+ }
138
+ return *c.ErrorInfoFunctionEnabled
139
+}
140
+
141
func (c *Collector) Init(context.Context) error {
142
if c.MyCNF != "" {
143
dsn, err := dsnFromFile(c.MyCNF)
src/go/plugin/go.d/collector/mysql/config_schema.json
+18
@@ -48,6 +48,18 @@
48
"minimum": 1,
49
"maximum": 5000,
50
"default": 500
51
+ },
52
+ "deadlock_info_function_enabled": {
53
+ "title": "Enable Deadlock Info Function",
54
+ "description": "Enable the deadlock-info function. WARNING: query text may contain unmasked sensitive literals (PII). Only enable after ensuring proper access controls to the Netdata dashboard. This function reads SHOW ENGINE INNODB STATUS and may require PROCESS privilege.",
55
+ "type": "boolean",
56
+ "default": true
57
+ },
58
+ "error_info_function_enabled": {
59
+ "title": "Enable Error Info Function",
60
+ "description": "Enable the error-info function. WARNING: error messages and query text may contain unmasked sensitive literals (PII). This function reads Performance Schema statement history tables; ensure proper access controls to the Netdata dashboard.",
61
+ "type": "boolean",
62
+ "default": true
63
}
64
},
65
"required": [
@@ -69,6 +81,12 @@
81
},
82
"timeout": {
83
"ui:help": "Accepts decimals for precise control (e.g., type 1.5 for 1.5 seconds)."
84
+ },
85
+ "deadlock_info_function_enabled": {
86
+ "ui:help": "When enabled, the deadlock-info function becomes available in the Netdata dashboard. WARNING: query text may contain unmasked sensitive literals; restrict dashboard access."
87
+ },
88
+ "error_info_function_enabled": {
89
+ "ui:help": "When enabled, the error-info function becomes available in the Netdata dashboard. WARNING: error messages and query text may include sensitive literals."
90
}
91
}
92
}
src/go/plugin/go.d/collector/mysql/deadlock_info.go
new
+779
@@ -0,0 +1,779 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+package mysql
4
+
5
+import (
6
+ "bufio"
7
+ "context"
8
+ "database/sql"
9
+ "errors"
10
+ "fmt"
11
+ "regexp"
12
+ "sort"
13
+ "strconv"
14
+ "strings"
15
+ "time"
16
+
17
+ mysqlDriver "github.com/go-sql-driver/mysql"
18
+
19
+ "github.com/netdata/netdata/go/plugins/pkg/funcapi"
20
+ "github.com/netdata/netdata/go/plugins/plugin/go.d/pkg/strmutil"
21
+)
22
+
23
+const (
24
+ deadlockIdxRowID = iota
25
+ deadlockIdxDeadlockID
26
+ deadlockIdxTimestamp
27
+ deadlockIdxProcessID
28
+ deadlockIdxSpid
29
+ deadlockIdxEcid
30
+ deadlockIdxIsVictim
31
+ deadlockIdxQueryText
32
+ deadlockIdxLockMode
33
+ deadlockIdxLockStatus
34
+ deadlockIdxWaitResource
35
+ deadlockIdxDatabase
36
+ deadlockColumnCount
37
+)
38
+
39
+const deadlockInfoMethodID = "deadlock-info"
40
+
41
+const (
42
+ deadlockSectionWaiting = "waiting"
43
+ deadlockSectionHolds = "holds"
44
+)
45
+
46
+var (
47
+ reDeadlockHeader = regexp.MustCompile(`LATEST DETECTED DEADLOCK`)
48
+ reDeadlockTxn = regexp.MustCompile(`(?i)^\*\*\* \((\d+)\) TRANSACTION:?`)
49
+ reDeadlockWait = regexp.MustCompile(`(?i)^\*\*\* \((\d+)\) WAITING FOR THIS LOCK TO BE GRANTED:?`)
50
+ reDeadlockHolds = regexp.MustCompile(`(?i)^\*\*\* \((\d+)\) HOLDS THE LOCK\(S\):?`)
51
+ reDeadlockWaitNoTxn = regexp.MustCompile(`(?i)^\*\*\*\s*WAITING FOR THIS LOCK TO BE GRANTED:?`)
52
+ reDeadlockHoldsNoTxn = regexp.MustCompile(`(?i)^\*\*\*\s*HOLDS THE LOCK\(S\):?`)
53
+ reDeadlockVictim = regexp.MustCompile(`(?i)^\*\*\* WE ROLL BACK TRANSACTION \((\d+)\)`)
54
+ reDeadlockThread = regexp.MustCompile(`(?i)\b(?:mysql|mariadb)?\s*thread id\s+(\d+)`)
55
+ reDeadlockMode = regexp.MustCompile(`(?i)lock[_ ]mode\s+([A-Z0-9_-]+)`)
56
+ reDeadlockTS = regexp.MustCompile(`\b\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\b`)
57
+ reDeadlockTable = regexp.MustCompile(`(?i)\bof\s+table\s+` + "`?" + `([-\w$]+)` + "`?" + `\.` + "`?" + `([-\w$]+)` + "`?")
58
+ reQueryTableRef = regexp.MustCompile(`(?i)\b(?:from|update|into|join)\s+` + "`?" + `([-\w$]+)` + "`?" + `\.` + "`?" + `([-\w$]+)` + "`?")
59
+ reSQLStatement = regexp.MustCompile(`(?i)^(?:/\*.*\*/\s*)?(SELECT|UPDATE|INSERT|DELETE|REPLACE|WITH|ALTER|CREATE|DROP|TRUNCATE|LOCK|UNLOCK|SET|SHOW|CALL|EXEC|EXECUTE|DO|BEGIN|COMMIT|ROLLBACK|MERGE)\b`)
60
+)
61
+
62
+const (
63
+ deadlockInfoHelp = "Latest detected deadlock from SHOW ENGINE INNODB STATUS. WARNING: query text may include unmasked sensitive literals; restrict dashboard access."
64
+ deadlockParseErrorStatus = 561
65
+)
66
+
67
+func deadlockInfoMethodConfig() funcapi.MethodConfig {
68
+ return funcapi.MethodConfig{
69
+ ID: deadlockInfoMethodID,
70
+ Name: "Deadlock Info",
71
+ UpdateEvery: 10,
72
+ Help: deadlockInfoHelp,
73
+ RequireCloud: true,
74
+ RequiredParams: []funcapi.ParamConfig{},
75
+ }
76
+}
77
+
78
+// funcDeadlockInfo handles the deadlock-info function.
79
+type funcDeadlockInfo struct {
80
+ router *funcRouter
81
+}
82
+
83
+func newFuncDeadlockInfo(r *funcRouter) *funcDeadlockInfo {
84
+ return &funcDeadlockInfo{router: r}
85
+}
86
+
87
+// Compile-time interface check.
88
+var _ funcapi.MethodHandler = (*funcDeadlockInfo)(nil)
89
+
90
+func (f *funcDeadlockInfo) MethodParams(ctx context.Context, method string) ([]funcapi.ParamConfig, error) {
91
+ if !f.router.collector.Config.GetDeadlockInfoFunctionEnabled() {
92
+ return nil, fmt.Errorf("deadlock-info function disabled in configuration")
93
+ }
94
+ return []funcapi.ParamConfig{}, nil
95
+}
96
+
97
+func (f *funcDeadlockInfo) Handle(ctx context.Context, method string, params funcapi.ResolvedParams) *funcapi.FunctionResponse {
98
+ if f.router.collector.db == nil {
99
+ if err := f.router.collector.openConnection(); err != nil {
100
+ return funcapi.UnavailableResponse("collector is still initializing, please retry in a few seconds")
101
+ }
102
+ }
103
+ return f.router.collector.collectDeadlockInfo(ctx)
104
+}
105
+
106
+func (f *funcDeadlockInfo) Cleanup(ctx context.Context) {}
107
+
108
+type mysqlDeadlockTxn struct {
109
+ txnNum int
110
+ threadID string
111
+ queryText string
112
+ lockMode string
113
+ lockStatus string
114
+ waitResource string
115
+}
116
+
117
+type mysqlDeadlockParseResult struct {
118
+ found bool
119
+ deadlockTime time.Time
120
+ victimTxnNum int
121
+ transactions []*mysqlDeadlockTxn
122
+ parseErr error
123
+}
124
+
125
+func (c *Collector) deadlockInfoParams(context.Context) ([]funcapi.ParamConfig, error) {
126
+ if !c.Config.GetDeadlockInfoFunctionEnabled() {
127
+ return nil, fmt.Errorf("deadlock-info function disabled in configuration")
128
+ }
129
+ return []funcapi.ParamConfig{}, nil
130
+}
131
+
132
+func (c *Collector) collectDeadlockInfo(ctx context.Context) *funcapi.FunctionResponse {
133
+ if !c.Config.GetDeadlockInfoFunctionEnabled() {
134
+ return &funcapi.FunctionResponse{
135
+ Status: 503,
136
+ Message: "deadlock-info function has been disabled in configuration. " +
137
+ "To enable, set deadlock_info_function_enabled: true in the MySQL collector config.",
138
+ }
139
+ }
140
+
141
+ statusText, err := c.queryInnoDBStatus(ctx)
142
+ if err != nil {
143
+ if errors.Is(err, context.DeadlineExceeded) {
144
+ return c.deadlockInfoResponse(504, "deadlock query timed out", nil)
145
+ }
146
+ if isMySQLPermissionError(err) {
147
+ return c.deadlockInfoResponse(
148
+ 403,
149
+ "Deadlock info requires permission to run SHOW ENGINE INNODB STATUS. "+
150
+ "Grant with: GRANT USAGE, REPLICATION CLIENT, PROCESS ON *.* TO 'netdata'@'%';",
151
+ nil,
152
+ )
153
+ }
154
+ c.Warningf("deadlock-info: query failed: %v", err)
155
+ return c.deadlockInfoResponse(500, fmt.Sprintf("deadlock query failed: %v", err), nil)
156
+ }
157
+
158
+ parseRes := parseInnoDBDeadlock(statusText, time.Now().UTC())
159
+ if parseRes.parseErr != nil {
160
+ c.Warningf("deadlock-info: parse failed: %v", parseRes.parseErr)
161
+ return c.deadlockInfoResponse(deadlockParseErrorStatus, "deadlock section could not be parsed", nil)
162
+ }
163
+ if !parseRes.found {
164
+ return c.deadlockInfoResponse(200, "no deadlock found in SHOW ENGINE INNODB STATUS", nil)
165
+ }
166
+
167
+ deadlockID := generateDeadlockID(parseRes.deadlockTime)
168
+ rows := buildDeadlockRows(parseRes, deadlockID)
169
+ if len(rows) == 0 {
170
+ return c.deadlockInfoResponse(200, "deadlock detected but no transactions could be parsed", nil)
171
+ }
172
+
173
+ return c.deadlockInfoResponse(200, "latest detected deadlock", rows)
174
+}
175
+
176
+func (c *Collector) deadlockInfoResponse(status int, message string, data [][]any) *funcapi.FunctionResponse {
177
+ if data == nil {
178
+ data = make([][]any, 0)
179
+ }
180
+ return &funcapi.FunctionResponse{
181
+ Status: status,
182
+ Help: deadlockInfoHelp,
183
+ Message: message,
184
+ Columns: c.buildDeadlockColumns(),
185
+ Data: data,
186
+ DefaultSortColumn: "timestamp",
187
+ }
188
+}
189
+
190
+func (c *Collector) queryInnoDBStatus(ctx context.Context) (string, error) {
191
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
192
+ defer cancel()
193
+
194
+ var typ, name, status sql.NullString
195
+ if err := c.db.QueryRowContext(qctx, queryShowEngineInnoDBStatus).Scan(&typ, &name, &status); err != nil {
196
+ return "", err
197
+ }
198
+ if !status.Valid {
199
+ return "", fmt.Errorf("innodb status response was empty")
200
+ }
201
+ return status.String, nil
202
+}
203
+
204
+func (c *Collector) buildDeadlockColumns() map[string]any {
205
+ const (
206
+ ftString = funcapi.FieldTypeString
207
+ ftInteger = funcapi.FieldTypeInteger
208
+ ftTimestamp = funcapi.FieldTypeTimestamp
209
+
210
+ trNone = funcapi.FieldTransformNone
211
+ trNumber = funcapi.FieldTransformNumber
212
+ trDatetime = funcapi.FieldTransformDatetime
213
+
214
+ visValue = funcapi.FieldVisualValue
215
+ visPill = funcapi.FieldVisualPill
216
+
217
+ sortAsc = funcapi.FieldSortAscending
218
+ sortDesc = funcapi.FieldSortDescending
219
+
220
+ summaryCount = funcapi.FieldSummaryCount
221
+ summaryMax = funcapi.FieldSummaryMax
222
+
223
+ filterMulti = funcapi.FieldFilterMultiselect
224
+ filterRange = funcapi.FieldFilterRange
225
+ )
226
+
227
+ columns := map[string]any{
228
+ "row_id": funcapi.Column{
229
+ Index: deadlockIdxRowID,
230
+ Name: "Row ID",
231
+ Type: ftString,
232
+ Visualization: visValue,
233
+ Sort: sortAsc,
234
+ Sortable: true,
235
+ Sticky: false,
236
+ Summary: summaryCount,
237
+ Filter: filterMulti,
238
+ FullWidth: false,
239
+ Wrap: false,
240
+ UniqueKey: true,
241
+ Visible: false,
242
+ ValueOptions: funcapi.ValueOptions{
243
+ Transform: trNone,
244
+ DecimalPoints: 0,
245
+ },
246
+ }.BuildColumn(),
247
+ "deadlock_id": funcapi.Column{
248
+ Index: deadlockIdxDeadlockID,
249
+ Name: "Deadlock ID",
250
+ Type: ftString,
251
+ Visualization: visValue,
252
+ Sort: sortAsc,
253
+ Sortable: true,
254
+ Summary: summaryCount,
255
+ Filter: filterMulti,
256
+ Visible: true,
257
+ ValueOptions: funcapi.ValueOptions{
258
+ Transform: trNone,
259
+ },
260
+ }.BuildColumn(),
261
+ "timestamp": funcapi.Column{
262
+ Index: deadlockIdxTimestamp,
263
+ Name: "Timestamp",
264
+ Type: ftTimestamp,
265
+ Visualization: visValue,
266
+ Sort: sortDesc,
267
+ Sortable: true,
268
+ Summary: summaryMax,
269
+ Filter: filterRange,
270
+ Visible: true,
271
+ ValueOptions: funcapi.ValueOptions{
272
+ Transform: trDatetime,
273
+ },
274
+ }.BuildColumn(),
275
+ "process_id": funcapi.Column{
276
+ Index: deadlockIdxProcessID,
277
+ Name: "Process ID",
278
+ Type: ftString,
279
+ Visualization: visValue,
280
+ Sort: sortAsc,
281
+ Sortable: true,
282
+ Summary: summaryCount,
283
+ Filter: filterMulti,
284
+ Visible: true,
285
+ ValueOptions: funcapi.ValueOptions{
286
+ Transform: trNone,
287
+ },
288
+ }.BuildColumn(),
289
+ "spid": funcapi.Column{
290
+ Index: deadlockIdxSpid,
291
+ Name: "Connection ID",
292
+ Type: ftInteger,
293
+ Visualization: visValue,
294
+ Sort: sortAsc,
295
+ Sortable: true,
296
+ Summary: summaryCount,
297
+ Filter: filterRange,
298
+ Visible: true,
299
+ ValueOptions: funcapi.ValueOptions{
300
+ Transform: trNumber,
301
+ },
302
+ }.BuildColumn(),
303
+ "ecid": funcapi.Column{
304
+ Index: deadlockIdxEcid,
305
+ Name: "ECID",
306
+ Type: ftInteger,
307
+ Visualization: visValue,
308
+ Sort: sortAsc,
309
+ Sortable: true,
310
+ Summary: summaryCount,
311
+ Filter: filterRange,
312
+ Visible: true,
313
+ ValueOptions: funcapi.ValueOptions{
314
+ Transform: trNumber,
315
+ },
316
+ }.BuildColumn(),
317
+ "is_victim": funcapi.Column{
318
+ Index: deadlockIdxIsVictim,
319
+ Name: "Victim",
320
+ Type: ftString,
321
+ Visualization: visPill,
322
+ Sort: sortAsc,
323
+ Sortable: true,
324
+ Summary: summaryCount,
325
+ Filter: filterMulti,
326
+ Visible: true,
327
+ ValueOptions: funcapi.ValueOptions{
328
+ Transform: trNone,
329
+ },
330
+ }.BuildColumn(),
331
+ "query_text": funcapi.Column{
332
+ Index: deadlockIdxQueryText,
333
+ Name: "Query",
334
+ Type: ftString,
335
+ Visualization: visValue,
336
+ Sort: sortAsc,
337
+ Sortable: false,
338
+ Sticky: true,
339
+ Summary: summaryCount,
340
+ Filter: filterMulti,
341
+ FullWidth: true,
342
+ Wrap: true,
343
+ Visible: true,
344
+ ValueOptions: funcapi.ValueOptions{
345
+ Transform: trNone,
346
+ },
347
+ }.BuildColumn(),
348
+ "lock_mode": funcapi.Column{
349
+ Index: deadlockIdxLockMode,
350
+ Name: "Lock Mode",
351
+ Type: ftString,
352
+ Visualization: visValue,
353
+ Sort: sortAsc,
354
+ Sortable: true,
355
+ Summary: summaryCount,
356
+ Filter: filterMulti,
357
+ Visible: true,
358
+ ValueOptions: funcapi.ValueOptions{
359
+ Transform: trNone,
360
+ },
361
+ }.BuildColumn(),
362
+ "lock_status": funcapi.Column{
363
+ Index: deadlockIdxLockStatus,
364
+ Name: "Lock Status",
365
+ Type: ftString,
366
+ Visualization: visPill,
367
+ Sort: sortAsc,
368
+ Sortable: true,
369
+ Summary: summaryCount,
370
+ Filter: filterMulti,
371
+ Visible: true,
372
+ ValueOptions: funcapi.ValueOptions{
373
+ Transform: trNone,
374
+ },
375
+ }.BuildColumn(),
376
+ "wait_resource": funcapi.Column{
377
+ Index: deadlockIdxWaitResource,
378
+ Name: "Wait Resource",
379
+ Type: ftString,
380
+ Visualization: visValue,
381
+ Sort: sortAsc,
382
+ Sortable: false,
383
+ Summary: summaryCount,
384
+ Filter: filterMulti,
385
+ Visible: true,
386
+ ValueOptions: funcapi.ValueOptions{
387
+ Transform: trNone,
388
+ },
389
+ }.BuildColumn(),
390
+ "database": funcapi.Column{
391
+ Index: deadlockIdxDatabase,
392
+ Name: "Database",
393
+ Type: ftString,
394
+ Visualization: visValue,
395
+ Sort: sortAsc,
396
+ Sortable: true,
397
+ Summary: summaryCount,
398
+ Filter: filterMulti,
399
+ Visible: true,
400
+ ValueOptions: funcapi.ValueOptions{
401
+ Transform: trNone,
402
+ },
403
+ }.BuildColumn(),
404
+ }
405
+ return columns
406
+}
407
+
408
+func parseInnoDBDeadlock(status string, now time.Time) mysqlDeadlockParseResult {
409
+ result := mysqlDeadlockParseResult{
410
+ found: false,
411
+ deadlockTime: now.UTC(),
412
+ }
413
+
414
+ section, ok := extractDeadlockSection(status)
415
+ if !ok {
416
+ return result
417
+ }
418
+ result.found = true
419
+
420
+ if ts, ok := parseDeadlockTimestamp(section); ok {
421
+ result.deadlockTime = ts.UTC()
422
+ }
423
+
424
+ scanner := bufio.NewScanner(strings.NewReader(section))
425
+ scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
426
+
427
+ txnByNum := make(map[int]*mysqlDeadlockTxn)
428
+ txnOrder := make([]int, 0, 4)
429
+
430
+ currentTxnNum := 0
431
+ currentSection := ""
432
+ expectingQueryTxn := 0
433
+ victimTxnNum := 0
434
+
435
+ ensureTxn := func(num int) *mysqlDeadlockTxn {
436
+ if txn, ok := txnByNum[num]; ok {
437
+ return txn
438
+ }
439
+ txn := &mysqlDeadlockTxn{txnNum: num}
440
+ txnByNum[num] = txn
441
+ txnOrder = append(txnOrder, num)
442
+ return txn
443
+ }
444
+
445
+ for scanner.Scan() {
446
+ line := strings.TrimSpace(scanner.Text())
447
+ if line == "" {
448
+ continue
449
+ }
450
+
451
+ if num, ok := parseDeadlockTxnHeader(line); ok {
452
+ currentTxnNum = num
453
+ currentSection = ""
454
+ expectingQueryTxn = 0
455
+ ensureTxn(num)
456
+ continue
457
+ }
458
+
459
+ if num, sectionType, ok := parseDeadlockTxnSection(line); ok {
460
+ currentTxnNum = num
461
+ currentSection = sectionType
462
+ expectingQueryTxn = 0
463
+ ensureTxn(num)
464
+ continue
465
+ }
466
+
467
+ if currentTxnNum != 0 {
468
+ if isDeadlockWaitNoTxn(line) {
469
+ currentSection = deadlockSectionWaiting
470
+ expectingQueryTxn = 0
471
+ ensureTxn(currentTxnNum)
472
+ continue
473
+ }
474
+ if isDeadlockHoldsNoTxn(line) {
475
+ currentSection = deadlockSectionHolds
476
+ expectingQueryTxn = 0
477
+ ensureTxn(currentTxnNum)
478
+ continue
479
+ }
480
+ }
481
+
482
+ if num, ok := parseDeadlockVictim(line); ok {
483
+ victimTxnNum = num
484
+ continue
485
+ }
486
+
487
+ if currentTxnNum == 0 {
488
+ continue
489
+ }
490
+
491
+ txn := ensureTxn(currentTxnNum)
492
+
493
+ if threadID, ok := parseDeadlockThreadID(line); ok {
494
+ txn.threadID = threadID
495
+ expectingQueryTxn = currentTxnNum
496
+ continue
497
+ }
498
+
499
+ if expectingQueryTxn == currentTxnNum && txn.queryText == "" && isSQLStatementLine(line) {
500
+ txn.queryText = line
501
+ expectingQueryTxn = 0
502
+ continue
503
+ }
504
+
505
+ if expectingQueryTxn == 0 && txn.queryText == "" && isSQLStatementLine(line) {
506
+ txn.queryText = line
507
+ continue
508
+ }
509
+
510
+ switch currentSection {
511
+ case deadlockSectionWaiting:
512
+ // WAITING must win even if HOLDS was seen first in the output.
513
+ txn.lockStatus = "WAITING"
514
+ if txn.waitResource == "" && isLockResourceLine(line) {
515
+ txn.waitResource = strmutil.TruncateText(line, topQueriesMaxTextLength)
516
+ }
517
+ if mode, ok := parseDeadlockLockMode(line); ok {
518
+ // WAITING lock mode should override any mode captured from HOLDS.
519
+ txn.lockMode = mode
520
+ }
521
+ case deadlockSectionHolds:
522
+ if txn.lockStatus == "" {
523
+ txn.lockStatus = "GRANTED"
524
+ }
525
+ if txn.lockMode == "" {
526
+ if mode, ok := parseDeadlockLockMode(line); ok {
527
+ txn.lockMode = mode
528
+ }
529
+ }
530
+ }
531
+ }
532
+
533
+ if err := scanner.Err(); err != nil {
534
+ result.parseErr = err
535
+ return result
536
+ }
537
+
538
+ result.victimTxnNum = victimTxnNum
539
+ result.transactions = make([]*mysqlDeadlockTxn, 0, len(txnOrder))
540
+ for _, num := range txnOrder {
541
+ txn := txnByNum[num]
542
+ if txn == nil {
543
+ continue
544
+ }
545
+ if txn.threadID == "" {
546
+ txn.threadID = fmt.Sprintf("txn-%d", num)
547
+ }
548
+ if txn.lockStatus == "" {
549
+ if num == victimTxnNum {
550
+ txn.lockStatus = "WAITING"
551
+ } else {
552
+ txn.lockStatus = "GRANTED"
553
+ }
554
+ }
555
+ result.transactions = append(result.transactions, txn)
556
+ }
557
+
558
+ if len(result.transactions) == 0 {
559
+ result.parseErr = fmt.Errorf("deadlock section detected but no transactions could be parsed")
560
+ return result
561
+ }
562
+
563
+ sort.Slice(result.transactions, func(i, j int) bool {
564
+ return result.transactions[i].txnNum < result.transactions[j].txnNum
565
+ })
566
+
567
+ return result
568
+}
569
+
570
+func buildDeadlockRows(parseRes mysqlDeadlockParseResult, deadlockID string) [][]any {
571
+ rows := make([][]any, 0, len(parseRes.transactions))
572
+ timestamp := parseRes.deadlockTime.UTC().Format(time.RFC3339Nano)
573
+
574
+ for _, txn := range parseRes.transactions {
575
+ if txn == nil {
576
+ continue
577
+ }
578
+
579
+ processID := strings.TrimSpace(txn.threadID)
580
+ if processID == "" {
581
+ processID = fmt.Sprintf("txn-%d", txn.txnNum)
582
+ }
583
+
584
+ var spid any
585
+ if id, err := strconv.Atoi(processID); err == nil {
586
+ spid = id
587
+ } else {
588
+ spid = nil
589
+ }
590
+
591
+ isVictim := "false"
592
+ if parseRes.victimTxnNum != 0 && txn.txnNum == parseRes.victimTxnNum {
593
+ isVictim = "true"
594
+ }
595
+
596
+ queryText := strmutil.TruncateText(strings.TrimSpace(txn.queryText), topQueriesMaxTextLength)
597
+ lockMode := strings.TrimSpace(txn.lockMode)
598
+ lockStatus := strings.TrimSpace(txn.lockStatus)
599
+ waitResource := strmutil.TruncateText(strings.TrimSpace(txn.waitResource), topQueriesMaxTextLength)
600
+ database := extractDeadlockDatabase(waitResource, queryText)
601
+
602
+ var databaseValue any
603
+ if database != "" {
604
+ databaseValue = database
605
+ }
606
+
607
+ row := make([]any, deadlockColumnCount)
608
+ row[deadlockIdxRowID] = fmt.Sprintf("%s:%s", deadlockID, processID)
609
+ row[deadlockIdxDeadlockID] = deadlockID
610
+ row[deadlockIdxTimestamp] = timestamp
611
+ row[deadlockIdxProcessID] = processID
612
+ row[deadlockIdxSpid] = spid
613
+ row[deadlockIdxEcid] = nil
614
+ row[deadlockIdxIsVictim] = isVictim
615
+ row[deadlockIdxQueryText] = queryText
616
+ row[deadlockIdxLockMode] = lockMode
617
+ row[deadlockIdxLockStatus] = lockStatus
618
+ row[deadlockIdxWaitResource] = waitResource
619
+ row[deadlockIdxDatabase] = databaseValue
620
+ rows = append(rows, row)
621
+ }
622
+
623
+ return rows
624
+}
625
+
626
+func extractDeadlockSection(status string) (string, bool) {
627
+ idx := reDeadlockHeader.FindStringIndex(status)
628
+ if idx == nil {
629
+ return "", false
630
+ }
631
+ return status[idx[0]:], true
632
+}
633
+
634
+func parseDeadlockTimestamp(section string) (time.Time, bool) {
635
+ match := reDeadlockTS.FindString(section)
636
+ if match == "" {
637
+ return time.Time{}, false
638
+ }
639
+ ts, err := time.ParseInLocation("2006-01-02 15:04:05", match, time.Local)
640
+ if err != nil {
641
+ return time.Time{}, false
642
+ }
643
+ return ts, true
644
+}
645
+
646
+func parseDeadlockTxnHeader(line string) (int, bool) {
647
+ m := reDeadlockTxn.FindStringSubmatch(line)
648
+ if len(m) != 2 {
649
+ return 0, false
650
+ }
651
+ n, err := strconv.Atoi(m[1])
652
+ if err != nil {
653
+ return 0, false
654
+ }
655
+ return n, true
656
+}
657
+
658
+func parseDeadlockTxnSection(line string) (int, string, bool) {
659
+ if m := reDeadlockWait.FindStringSubmatch(line); len(m) == 2 {
660
+ n, err := strconv.Atoi(m[1])
661
+ if err != nil {
662
+ return 0, "", false
663
+ }
664
+ return n, deadlockSectionWaiting, true
665
+ }
666
+ if m := reDeadlockHolds.FindStringSubmatch(line); len(m) == 2 {
667
+ n, err := strconv.Atoi(m[1])
668
+ if err != nil {
669
+ return 0, "", false
670
+ }
671
+ return n, deadlockSectionHolds, true
672
+ }
673
+ return 0, "", false
674
+}
675
+
676
+func isDeadlockWaitNoTxn(line string) bool {
677
+ return reDeadlockWaitNoTxn.MatchString(line)
678
+}
679
+
680
+func isDeadlockHoldsNoTxn(line string) bool {
681
+ return reDeadlockHoldsNoTxn.MatchString(line)
682
+}
683
+
684
+func parseDeadlockVictim(line string) (int, bool) {
685
+ m := reDeadlockVictim.FindStringSubmatch(line)
686
+ if len(m) != 2 {
687
+ return 0, false
688
+ }
689
+ n, err := strconv.Atoi(m[1])
690
+ if err != nil {
691
+ return 0, false
692
+ }
693
+ return n, true
694
+}
695
+
696
+func parseDeadlockThreadID(line string) (string, bool) {
697
+ m := reDeadlockThread.FindStringSubmatch(line)
698
+ if len(m) != 2 {
699
+ return "", false
700
+ }
701
+ return m[1], true
702
+}
703
+
704
+func parseDeadlockLockMode(line string) (string, bool) {
705
+ m := reDeadlockMode.FindStringSubmatch(line)
706
+ if len(m) != 2 {
707
+ return "", false
708
+ }
709
+ return strings.ToUpper(m[1]), true
710
+}
711
+
712
+func isLikelyQueryLine(line string) bool {
713
+ return isSQLStatementLine(line)
714
+}
715
+
716
+func isSQLStatementLine(line string) bool {
717
+ trimmed := strings.TrimSpace(line)
718
+ if trimmed == "" {
719
+ return false
720
+ }
721
+ upper := strings.ToUpper(trimmed)
722
+ if strings.HasPrefix(upper, "LOCK WAIT") {
723
+ return false
724
+ }
725
+ return reSQLStatement.MatchString(trimmed)
726
+}
727
+
728
+func isLockResourceLine(line string) bool {
729
+ upper := strings.ToUpper(strings.TrimSpace(line))
730
+ return strings.HasPrefix(upper, "RECORD LOCKS") || strings.HasPrefix(upper, "TABLE LOCK")
731
+}
732
+
733
+func extractDeadlockDatabase(waitResource, queryText string) string {
734
+ if db := extractDatabaseFromLock(waitResource); db != "" {
735
+ return db
736
+ }
737
+ if db := extractDatabaseFromQuery(queryText); db != "" {
738
+ return db
739
+ }
740
+ return ""
741
+}
742
+
743
+func extractDatabaseFromLock(line string) string {
744
+ m := reDeadlockTable.FindStringSubmatch(line)
745
+ if len(m) >= 2 {
746
+ return m[1]
747
+ }
748
+ return ""
749
+}
750
+
751
+func extractDatabaseFromQuery(queryText string) string {
752
+ m := reQueryTableRef.FindStringSubmatch(queryText)
753
+ if len(m) >= 2 {
754
+ return m[1]
755
+ }
756
+ return ""
757
+}
758
+
759
+func generateDeadlockID(t time.Time) string {
760
+ if t.IsZero() {
761
+ t = time.Now().UTC()
762
+ }
763
+ t = t.UTC()
764
+ micros := t.Nanosecond() / 1000
765
+ return t.Format("20060102150405") + fmt.Sprintf("%06d", micros)
766
+}
767
+
768
+func isMySQLPermissionError(err error) bool {
769
+ var mysqlErr *mysqlDriver.MySQLError
770
+ if errors.As(err, &mysqlErr) {
771
+ if mysqlErr.Number == 1045 || mysqlErr.Number == 1227 {
772
+ return true
773
+ }
774
+ }
775
+ msg := strings.ToLower(err.Error())
776
+ return strings.Contains(msg, "access denied") ||
777
+ strings.Contains(msg, "permission denied") ||
778
+ strings.Contains(msg, "process privilege")
779
+}
src/go/plugin/go.d/collector/mysql/deadlock_info_test.go
new
+523
@@ -0,0 +1,523 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+package mysql
4
+
5
+import (
6
+ "context"
7
+ "errors"
8
+ "testing"
9
+ "time"
10
+
11
+ mysqlDriver "github.com/go-sql-driver/mysql"
12
+
13
+ "github.com/DATA-DOG/go-sqlmock"
14
+ "github.com/stretchr/testify/assert"
15
+ "github.com/stretchr/testify/require"
16
+)
17
+
18
+func TestConfig_GetDeadlockInfoFunctionEnabled(t *testing.T) {
19
+ tests := []struct {
20
+ name string
21
+ cfg Config
22
+ expected bool
23
+ }{
24
+ {
25
+ name: "default nil pointer enables function",
26
+ cfg: Config{},
27
+ expected: true,
28
+ },
29
+ {
30
+ name: "explicit true enables function",
31
+ cfg: Config{
32
+ DeadlockInfoFunctionEnabled: boolPtr(true),
33
+ },
34
+ expected: true,
35
+ },
36
+ {
37
+ name: "explicit false disables function",
38
+ cfg: Config{
39
+ DeadlockInfoFunctionEnabled: boolPtr(false),
40
+ },
41
+ expected: false,
42
+ },
43
+ }
44
+
45
+ for _, tt := range tests {
46
+ t.Run(tt.name, func(t *testing.T) {
47
+ assert.Equal(t, tt.expected, tt.cfg.GetDeadlockInfoFunctionEnabled())
48
+ })
49
+ }
50
+}
51
+
52
+func TestConfig_GetErrorInfoFunctionEnabled(t *testing.T) {
53
+ tests := []struct {
54
+ name string
55
+ cfg Config
56
+ expected bool
57
+ }{
58
+ {
59
+ name: "default nil pointer enables function",
60
+ cfg: Config{},
61
+ expected: true,
62
+ },
63
+ {
64
+ name: "explicit true enables function",
65
+ cfg: Config{
66
+ ErrorInfoFunctionEnabled: boolPtr(true),
67
+ },
68
+ expected: true,
69
+ },
70
+ {
71
+ name: "explicit false disables function",
72
+ cfg: Config{
73
+ ErrorInfoFunctionEnabled: boolPtr(false),
74
+ },
75
+ expected: false,
76
+ },
77
+ }
78
+
79
+ for _, tt := range tests {
80
+ t.Run(tt.name, func(t *testing.T) {
81
+ assert.Equal(t, tt.expected, tt.cfg.GetErrorInfoFunctionEnabled())
82
+ })
83
+ }
84
+}
85
+
86
+func TestParseInnoDBDeadlock_WithDeadlock(t *testing.T) {
87
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 123456000, time.UTC)
88
+ res := parseInnoDBDeadlock(sampleDeadlockStatus, now)
89
+
90
+ assert.True(t, res.found)
91
+ assert.NoError(t, res.parseErr)
92
+ assert.Len(t, res.transactions, 2)
93
+ assert.Equal(t, 2, res.victimTxnNum)
94
+ assert.Equal(t, now.UTC(), res.deadlockTime)
95
+
96
+ assert.Equal(t, "10", res.transactions[0].threadID)
97
+ assert.Equal(t, "11", res.transactions[1].threadID)
98
+ assert.Equal(t, "WAITING", res.transactions[0].lockStatus)
99
+ assert.Equal(t, "GRANTED", res.transactions[1].lockStatus)
100
+}
101
+
102
+func TestParseInnoDBDeadlock_MariaDBThreadID(t *testing.T) {
103
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
104
+ res := parseInnoDBDeadlock(sampleDeadlockStatusMariaDB, now)
105
+
106
+ require.True(t, res.found)
107
+ require.NoError(t, res.parseErr)
108
+ require.Len(t, res.transactions, 2)
109
+
110
+ txnByNum := make(map[int]*mysqlDeadlockTxn, len(res.transactions))
111
+ for _, txn := range res.transactions {
112
+ txnByNum[txn.txnNum] = txn
113
+ }
114
+
115
+ require.Contains(t, txnByNum, 1)
116
+ require.Contains(t, txnByNum, 2)
117
+
118
+ assert.Equal(t, "55", txnByNum[1].threadID)
119
+ assert.Contains(t, txnByNum[1].queryText, "deadlock_a")
120
+ assert.NotEmpty(t, txnByNum[1].waitResource)
121
+}
122
+
123
+func TestParseInnoDBDeadlock_SkipsLockWaitLine(t *testing.T) {
124
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
125
+ res := parseInnoDBDeadlock(sampleDeadlockStatusLockWaitLine, now)
126
+
127
+ require.True(t, res.found)
128
+ require.NoError(t, res.parseErr)
129
+ require.Len(t, res.transactions, 1)
130
+
131
+ txn := res.transactions[0]
132
+ require.NotNil(t, txn)
133
+ assert.Equal(t, "90", txn.threadID)
134
+ assert.Equal(t, "UPDATE deadlock_a SET value = value + 1 WHERE id = 1", txn.queryText)
135
+}
136
+
137
+func TestParseInnoDBDeadlock_WaitingHeaderWithoutTxn(t *testing.T) {
138
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
139
+ res := parseInnoDBDeadlock(sampleDeadlockStatusWaitNoTxn, now)
140
+
141
+ require.True(t, res.found)
142
+ require.NoError(t, res.parseErr)
143
+ require.Len(t, res.transactions, 1)
144
+
145
+ txn := res.transactions[0]
146
+ require.NotNil(t, txn)
147
+ assert.Equal(t, "99", txn.threadID)
148
+ assert.Equal(t, "WAITING", txn.lockStatus)
149
+ assert.Equal(t, "X", txn.lockMode)
150
+ assert.NotEmpty(t, txn.waitResource)
151
+}
152
+
153
+func TestParseInnoDBDeadlock_HoldsBeforeWaiting(t *testing.T) {
154
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
155
+ res := parseInnoDBDeadlock(sampleDeadlockStatusHoldsFirst, now)
156
+
157
+ require.True(t, res.found)
158
+ require.NoError(t, res.parseErr)
159
+ require.Len(t, res.transactions, 2)
160
+
161
+ var txn1 *mysqlDeadlockTxn
162
+ for _, txn := range res.transactions {
163
+ if txn.txnNum == 1 {
164
+ txn1 = txn
165
+ break
166
+ }
167
+ }
168
+
169
+ require.NotNil(t, txn1, "transaction (1) should be present")
170
+ assert.Equal(t, "WAITING", txn1.lockStatus)
171
+ assert.Equal(t, "AUTO-INC", txn1.lockMode)
172
+}
173
+
174
+func TestParseDeadlockLockMode_HyphenAndUnderscore(t *testing.T) {
175
+ mode, ok := parseDeadlockLockMode("lock mode AUTO-INC waiting")
176
+ require.True(t, ok)
177
+ assert.Equal(t, "AUTO-INC", mode)
178
+
179
+ mode, ok = parseDeadlockLockMode("lock_mode AUTO_INC")
180
+ require.True(t, ok)
181
+ assert.Equal(t, "AUTO_INC", mode)
182
+}
183
+
184
+func TestParseInnoDBDeadlock_WithTimestamp(t *testing.T) {
185
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
186
+ res := parseInnoDBDeadlock(sampleDeadlockStatusWithTimestamp, now)
187
+
188
+ assert.True(t, res.found)
189
+ assert.NoError(t, res.parseErr)
190
+ assert.Equal(t, "2026-01-25 12:34:56", res.deadlockTime.In(time.Local).Format("2006-01-02 15:04:05"))
191
+}
192
+
193
+func TestParseInnoDBDeadlock_ThreeWay(t *testing.T) {
194
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
195
+ res := parseInnoDBDeadlock(sampleDeadlockStatusThreeWay, now)
196
+
197
+ require.True(t, res.found)
198
+ require.NoError(t, res.parseErr)
199
+ require.Len(t, res.transactions, 3)
200
+ assert.Equal(t, 3, res.victimTxnNum)
201
+
202
+ txnByNum := make(map[int]*mysqlDeadlockTxn, len(res.transactions))
203
+ for _, txn := range res.transactions {
204
+ txnByNum[txn.txnNum] = txn
205
+ }
206
+
207
+ require.Contains(t, txnByNum, 1)
208
+ require.Contains(t, txnByNum, 2)
209
+ require.Contains(t, txnByNum, 3)
210
+
211
+ assert.Equal(t, "30", txnByNum[1].threadID)
212
+ assert.Equal(t, "WAITING", txnByNum[1].lockStatus)
213
+
214
+ assert.Equal(t, "31", txnByNum[2].threadID)
215
+ assert.Equal(t, "GRANTED", txnByNum[2].lockStatus)
216
+
217
+ assert.Equal(t, "32", txnByNum[3].threadID)
218
+ // Victim fallback should mark transaction (3) as WAITING even without WAITING/HOLDS sections.
219
+ assert.Equal(t, "WAITING", txnByNum[3].lockStatus)
220
+}
221
+
222
+func TestParseInnoDBDeadlock_WaitingWithoutLockMode(t *testing.T) {
223
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
224
+ res := parseInnoDBDeadlock(sampleDeadlockStatusWaitingNoLockMode, now)
225
+
226
+ require.True(t, res.found)
227
+ require.NoError(t, res.parseErr)
228
+ require.Len(t, res.transactions, 2)
229
+
230
+ var txn1 *mysqlDeadlockTxn
231
+ for _, txn := range res.transactions {
232
+ if txn.txnNum == 1 {
233
+ txn1 = txn
234
+ break
235
+ }
236
+ }
237
+
238
+ require.NotNil(t, txn1, "transaction (1) should be present")
239
+ assert.Equal(t, "WAITING", txn1.lockStatus)
240
+ assert.Empty(t, txn1.lockMode)
241
+ assert.NotEmpty(t, txn1.waitResource)
242
+}
243
+
244
+func TestParseInnoDBDeadlock_NoDeadlock(t *testing.T) {
245
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
246
+ res := parseInnoDBDeadlock("no deadlock here", now)
247
+
248
+ assert.False(t, res.found)
249
+ assert.NoError(t, res.parseErr)
250
+ assert.Len(t, res.transactions, 0)
251
+}
252
+
253
+func TestParseInnoDBDeadlock_MalformedSection(t *testing.T) {
254
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 0, time.UTC)
255
+ res := parseInnoDBDeadlock(sampleDeadlockStatusMalformed, now)
256
+
257
+ assert.True(t, res.found)
258
+ assert.Error(t, res.parseErr)
259
+ assert.Len(t, res.transactions, 0)
260
+}
261
+
262
+func TestCollector_collectDeadlockInfo_ParseError(t *testing.T) {
263
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
264
+ require.NoError(t, err)
265
+ defer func() { _ = db.Close() }()
266
+
267
+ rows := sqlmock.NewRows([]string{"Type", "Name", "Status"}).
268
+ AddRow("InnoDB", "Status", sampleDeadlockStatusMalformed)
269
+ mock.ExpectQuery(queryShowEngineInnoDBStatus).WillReturnRows(rows)
270
+
271
+ collr := New()
272
+ collr.db = db
273
+
274
+ resp := collr.collectDeadlockInfo(context.Background())
275
+ require.NotNil(t, resp)
276
+ assert.Equal(t, deadlockParseErrorStatus, resp.Status)
277
+ assert.Contains(t, resp.Message, "could not be parsed")
278
+ assert.NoError(t, mock.ExpectationsWereMet())
279
+}
280
+
281
+func TestCollector_collectDeadlockInfo_QueryError(t *testing.T) {
282
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
283
+ require.NoError(t, err)
284
+ defer func() { _ = db.Close() }()
285
+
286
+ mock.ExpectQuery(queryShowEngineInnoDBStatus).
287
+ WillReturnError(errors.New("boom"))
288
+
289
+ collr := New()
290
+ collr.db = db
291
+
292
+ resp := collr.collectDeadlockInfo(context.Background())
293
+ require.NotNil(t, resp)
294
+ assert.Equal(t, 500, resp.Status)
295
+ assert.Contains(t, resp.Message, "deadlock query failed")
296
+ assert.NoError(t, mock.ExpectationsWereMet())
297
+}
298
+
299
+func TestCollector_collectDeadlockInfo_Timeout(t *testing.T) {
300
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
301
+ require.NoError(t, err)
302
+ defer func() { _ = db.Close() }()
303
+
304
+ mock.ExpectQuery(queryShowEngineInnoDBStatus).
305
+ WillReturnError(context.DeadlineExceeded)
306
+
307
+ collr := New()
308
+ collr.db = db
309
+
310
+ resp := collr.collectDeadlockInfo(context.Background())
311
+ require.NotNil(t, resp)
312
+ assert.Equal(t, 504, resp.Status)
313
+ assert.Contains(t, resp.Message, "timed out")
314
+ assert.NoError(t, mock.ExpectationsWereMet())
315
+}
316
+
317
+func TestCollector_collectDeadlockInfo_PermissionDenied(t *testing.T) {
318
+ db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherEqual))
319
+ require.NoError(t, err)
320
+ defer func() { _ = db.Close() }()
321
+
322
+ collr := New()
323
+ collr.db = db
324
+
325
+ permErr := &mysqlDriver.MySQLError{
326
+ Number: 1227,
327
+ Message: "Access denied; you need (at least one of) the PROCESS privilege(s) for this operation",
328
+ }
329
+ mock.ExpectQuery(queryShowEngineInnoDBStatus).WillReturnError(permErr)
330
+
331
+ resp := collr.collectDeadlockInfo(context.Background())
332
+ require.NotNil(t, resp)
333
+ assert.Equal(t, 403, resp.Status)
334
+ assert.Contains(t, resp.Message, "PROCESS")
335
+ assert.NoError(t, mock.ExpectationsWereMet())
336
+}
337
+
338
+func TestCollector_collectDeadlockInfo_Disabled(t *testing.T) {
339
+ c := New()
340
+ c.Config.DeadlockInfoFunctionEnabled = boolPtr(false)
341
+
342
+ resp := c.collectDeadlockInfo(context.Background())
343
+ require.Equal(t, 503, resp.Status)
344
+ assert.Contains(t, resp.Message, "disabled")
345
+}
346
+
347
+func TestBuildDeadlockRows(t *testing.T) {
348
+ now := time.Date(2026, time.January, 25, 12, 0, 0, 123456000, time.UTC)
349
+ res := parseInnoDBDeadlock(sampleDeadlockStatus, now)
350
+ deadlockID := generateDeadlockID(now)
351
+ rows := buildDeadlockRows(res, deadlockID)
352
+
353
+ assert.Len(t, rows, 2)
354
+ assert.Equal(t, deadlockID, rows[0][deadlockIdxDeadlockID])
355
+ assert.Equal(t, deadlockID, rows[1][deadlockIdxDeadlockID])
356
+ assert.Equal(t, deadlockID+":10", rows[0][deadlockIdxRowID])
357
+ assert.Equal(t, deadlockID+":11", rows[1][deadlockIdxRowID])
358
+
359
+ hasDatabase := false
360
+ for _, row := range rows {
361
+ if row[deadlockIdxDatabase] == "netdata" {
362
+ hasDatabase = true
363
+ break
364
+ }
365
+ }
366
+ assert.True(t, hasDatabase, "expected at least one row with database populated")
367
+}
368
+
369
+func boolPtr(v bool) *bool {
370
+ return &v
371
+}
372
+
373
+const sampleDeadlockStatus = `
374
+------------------------
375
+LATEST DETECTED DEADLOCK
376
+------------------------
377
+*** (1) TRANSACTION:
378
+TRANSACTION 100, ACTIVE 0 sec
379
+MySQL thread id 10, OS thread handle 1, query id 100 localhost root updating
380
+UPDATE Animals SET value = value + 1 WHERE name='Aardvark'
381
+*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
382
+RECORD LOCKS space id 1 page no 2 n bits 72 index PRIMARY of table netdata.Birds trx id 100 lock mode X waiting
383
+*** (2) TRANSACTION:
384
+TRANSACTION 101, ACTIVE 0 sec
385
+MySQL thread id 11, OS thread handle 2, query id 101 localhost root updating
386
+UPDATE Birds SET value = value + 1 WHERE name='Buzzard'
387
+*** (2) HOLDS THE LOCK(S):
388
+RECORD LOCKS space id 1 page no 3 n bits 72 index PRIMARY of table netdata.Animals trx id 101 lock mode X
389
+*** WE ROLL BACK TRANSACTION (2)
390
+`
391
+
392
+const sampleDeadlockStatusMariaDB = `
393
+------------------------
394
+LATEST DETECTED DEADLOCK
395
+------------------------
396
+*** (1) TRANSACTION:
397
+TRANSACTION 500, ACTIVE 1 sec
398
+MariaDB thread id 55, OS thread handle 1, query id 500 localhost root updating
399
+mysql tables in use 1, locked 1
400
+UPDATE deadlock_a SET value = value + 1 WHERE id = 1
401
+*** (1) waiting for this lock to be granted:
402
+RECORD LOCKS space id 5 page no 6 n bits 72 index PRIMARY of table netdata.deadlock_a trx id 500 lock_mode X locks rec but not gap waiting
403
+*** (2) TRANSACTION:
404
+TRANSACTION 501, ACTIVE 1 sec
405
+MariaDB thread id 56, OS thread handle 1, query id 501 localhost root updating
406
+UPDATE deadlock_b SET value = value + 1 WHERE id = 1
407
+*** (2) HOLDS THE LOCK(S):
408
+RECORD LOCKS space id 5 page no 6 n bits 72 index PRIMARY of table netdata.deadlock_a trx id 501 lock_mode X locks rec but not gap
409
+*** WE ROLL BACK TRANSACTION (2)
410
+`
411
+
412
+const sampleDeadlockStatusLockWaitLine = `
413
+------------------------
414
+LATEST DETECTED DEADLOCK
415
+------------------------
416
+*** (1) TRANSACTION:
417
+TRANSACTION 900, ACTIVE 1 sec
418
+MySQL thread id 90, OS thread handle 1, query id 900 localhost root updating
419
+LOCK WAIT 4 lock struct(s), heap size 1128, 2 row lock(s), undo log entries 1
420
+UPDATE deadlock_a SET value = value + 1 WHERE id = 1
421
+*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
422
+RECORD LOCKS space id 3 page no 4 n bits 72 index PRIMARY of table netdata.deadlock_a trx id 900 lock_mode X locks rec but not gap waiting
423
+*** WE ROLL BACK TRANSACTION (1)
424
+`
425
+
426
+const sampleDeadlockStatusWaitNoTxn = `
427
+------------------------
428
+LATEST DETECTED DEADLOCK
429
+------------------------
430
+*** (1) TRANSACTION:
431
+TRANSACTION 990, ACTIVE 1 sec
432
+MariaDB thread id 99, OS thread handle 1, query id 990 localhost root Updating
433
+UPDATE deadlock_b SET value = value + 1 WHERE id = 1
434
+*** WAITING FOR THIS LOCK TO BE GRANTED:
435
+RECORD LOCKS space id 7 page no 3 n bits 320 index PRIMARY of table netdata.deadlock_b trx id 42 lock_mode X locks rec but not gap waiting
436
+*** WE ROLL BACK TRANSACTION (1)
437
+`
438
+
439
+const sampleDeadlockStatusHoldsFirst = `
440
+------------------------
441
+LATEST DETECTED DEADLOCK
442
+------------------------
443
+*** (1) TRANSACTION:
444
+TRANSACTION 200, ACTIVE 0 sec
445
+MySQL thread id 20, OS thread handle 1, query id 200 localhost root updating
446
+UPDATE deadlock_b SET value = value + 1 WHERE id = 1
447
+*** (1) HOLDS THE LOCK(S):
448
+RECORD LOCKS space id 3 page no 4 n bits 72 index PRIMARY of table netdata.deadlock_a trx id 200 lock mode S
449
+*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
450
+RECORD LOCKS space id 4 page no 4 n bits 72 index PRIMARY of table netdata.deadlock_b trx id 200 lock mode AUTO-INC waiting
451
+*** (2) TRANSACTION:
452
+TRANSACTION 201, ACTIVE 0 sec
453
+MySQL thread id 21, OS thread handle 2, query id 201 localhost root updating
454
+UPDATE deadlock_a SET value = value + 1 WHERE id = 1
455
+*** (2) HOLDS THE LOCK(S):
456
+RECORD LOCKS space id 4 page no 4 n bits 72 index PRIMARY of table netdata.deadlock_b trx id 201 lock mode X
457
+*** WE ROLL BACK TRANSACTION (2)
458
+`
459
+
460
+const sampleDeadlockStatusWithTimestamp = `
461
+------------------------
462
+LATEST DETECTED DEADLOCK
463
+------------------------
464
+2026-01-25 12:34:56
465
+*** (1) TRANSACTION:
466
+TRANSACTION 100, ACTIVE 0 sec
467
+MySQL thread id 10, OS thread handle 1, query id 100 localhost root updating
468
+UPDATE Animals SET value = value + 1 WHERE name='Aardvark'
469
+*** (2) TRANSACTION:
470
+TRANSACTION 101, ACTIVE 0 sec
471
+MySQL thread id 11, OS thread handle 2, query id 101 localhost root updating
472
+UPDATE Birds SET value = value + 1 WHERE name='Buzzard'
473
+*** WE ROLL BACK TRANSACTION (2)
474
+`
475
+
476
+const sampleDeadlockStatusThreeWay = `
477
+------------------------
478
+LATEST DETECTED DEADLOCK
479
+------------------------
480
+*** (1) TRANSACTION:
481
+TRANSACTION 300, ACTIVE 0 sec
482
+MySQL thread id 30, OS thread handle 1, query id 300 localhost root updating
483
+UPDATE alpha SET value = value + 1 WHERE id = 1
484
+*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
485
+RECORD LOCKS space id 7 page no 8 n bits 72 index PRIMARY of table netdata.beta trx id 300 lock mode X waiting
486
+*** (2) TRANSACTION:
487
+TRANSACTION 301, ACTIVE 0 sec
488
+MySQL thread id 31, OS thread handle 2, query id 301 localhost root updating
489
+UPDATE beta SET value = value + 1 WHERE id = 1
490
+*** (2) HOLDS THE LOCK(S):
491
+RECORD LOCKS space id 7 page no 9 n bits 72 index PRIMARY of table netdata.gamma trx id 301 lock mode S
492
+*** (3) TRANSACTION:
493
+TRANSACTION 302, ACTIVE 0 sec
494
+MySQL thread id 32, OS thread handle 3, query id 302 localhost root updating
495
+UPDATE gamma SET value = value + 1 WHERE id = 1
496
+*** WE ROLL BACK TRANSACTION (3)
497
+`
498
+
499
+const sampleDeadlockStatusWaitingNoLockMode = `
500
+------------------------
501
+LATEST DETECTED DEADLOCK
502
+------------------------
503
+*** (1) TRANSACTION:
504
+TRANSACTION 400, ACTIVE 0 sec
505
+MySQL thread id 40, OS thread handle 1, query id 400 localhost root updating
506
+UPDATE delta SET value = value + 1 WHERE id = 1
507
+*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
508
+RECORD LOCKS space id 10 page no 11 n bits 72 index PRIMARY of table netdata.epsilon trx id 400 waiting
509
+*** (2) TRANSACTION:
510
+TRANSACTION 401, ACTIVE 0 sec
511
+MySQL thread id 41, OS thread handle 2, query id 401 localhost root updating
512
+UPDATE epsilon SET value = value + 1 WHERE id = 1
513
+*** (2) HOLDS THE LOCK(S):
514
+RECORD LOCKS space id 10 page no 12 n bits 72 index PRIMARY of table netdata.delta trx id 401 lock mode X
515
+*** WE ROLL BACK TRANSACTION (1)
516
+`
517
+
518
+const sampleDeadlockStatusMalformed = `
519
+------------------------
520
+LATEST DETECTED DEADLOCK
521
+------------------------
522
+THIS IS NOT A VALID DEADLOCK SECTION
523
+`
src/go/plugin/go.d/collector/mysql/error_info.go
new
+568
@@ -0,0 +1,568 @@
1
+// SPDX-License-Identifier: GPL-3.0-or-later
2
+
3
+package mysql
4
+
5
+import (
6
+ "context"
7
+ "database/sql"
8
+ "fmt"
9
+ "strings"
10
+
11
+ "github.com/netdata/netdata/go/plugins/pkg/funcapi"
12
+)
13
+
14
+const (
15
+ mysqlErrorAttrEnabled = "enabled"
16
+ mysqlErrorAttrNotEnabled = "not_enabled"
17
+ mysqlErrorAttrNotSupported = "not_supported"
18
+ mysqlErrorAttrNoData = "no_data"
19
+)
20
+
21
+const errorInfoMethodID = "error-info"
22
+
23
+func errorInfoMethodConfig() funcapi.MethodConfig {
24
+ return funcapi.MethodConfig{
25
+ ID: errorInfoMethodID,
26
+ Name: "Error Info",
27
+ UpdateEvery: 10,
28
+ Help: "Recent SQL errors from performance_schema statement history tables",
29
+ RequireCloud: true,
30
+ RequiredParams: []funcapi.ParamConfig{},
31
+ }
32
+}
33
+
34
+// funcErrorInfo handles the error-info function.
35
+type funcErrorInfo struct {
36
+ router *funcRouter
37
+}
38
+
39
+func newFuncErrorInfo(r *funcRouter) *funcErrorInfo {
40
+ return &funcErrorInfo{router: r}
41
+}
42
+
43
+// Compile-time interface check.
44
+var _ funcapi.MethodHandler = (*funcErrorInfo)(nil)
45
+
46
+func (f *funcErrorInfo) MethodParams(ctx context.Context, method string) ([]funcapi.ParamConfig, error) {
47
+ if !f.router.collector.Config.GetErrorInfoFunctionEnabled() {
48
+ return nil, fmt.Errorf("error-info function disabled in configuration")
49
+ }
50
+ return []funcapi.ParamConfig{}, nil
51
+}
52
+
53
+func (f *funcErrorInfo) Handle(ctx context.Context, method string, params funcapi.ResolvedParams) *funcapi.FunctionResponse {
54
+ if f.router.collector.db == nil {
55
+ if err := f.router.collector.openConnection(); err != nil {
56
+ return funcapi.UnavailableResponse("collector is still initializing, please retry in a few seconds")
57
+ }
58
+ }
59
+ return f.router.collector.collectErrorInfo(ctx)
60
+}
61
+
62
+func (f *funcErrorInfo) Cleanup(ctx context.Context) {}
63
+
64
+type mysqlErrorSource struct {
65
+ table string
66
+ fallbackTable string
67
+ columns map[string]bool
68
+ status string
69
+ reason string
70
+}
71
+
72
+type mysqlErrorRow struct {
73
+ Digest string
74
+ Query string
75
+ Schema string
76
+ ErrorNumber *int64
77
+ SQLState string
78
+ Message string
79
+}
80
+
81
+func mysqlErrorAttributionColumns() []topQueriesColumn {
82
+ return []topQueriesColumn{
83
+ {
84
+ ColumnMeta: funcapi.ColumnMeta{
85
+ Name: "errorAttribution",
86
+ Tooltip: "Error Attribution",
87
+ Type: funcapi.FieldTypeString,
88
+ Visible: true,
89
+ Transform: funcapi.FieldTransformNone,
90
+ Sort: funcapi.FieldSortAscending,
91
+ Summary: funcapi.FieldSummaryCount,
92
+ Filter: funcapi.FieldFilterMultiselect,
93
+ },
94
+ },
95
+ {
96
+ ColumnMeta: funcapi.ColumnMeta{
97
+ Name: "errorNumber",
98
+ Tooltip: "Error Number",
99
+ Type: funcapi.FieldTypeInteger,
100
+ Visible: true,
101
+ Transform: funcapi.FieldTransformNumber,
102
+ Sort: funcapi.FieldSortDescending,
103
+ Summary: funcapi.FieldSummaryMax,
104
+ Filter: funcapi.FieldFilterRange,
105
+ },
106
+ },
107
+ {
108
+ ColumnMeta: funcapi.ColumnMeta{
109
+ Name: "sqlState",
110
+ Tooltip: "SQL State",
111
+ Type: funcapi.FieldTypeString,
112
+ Visible: false,
113
+ Transform: funcapi.FieldTransformNone,
114
+ Sort: funcapi.FieldSortAscending,
115
+ Summary: funcapi.FieldSummaryCount,
116
+ Filter: funcapi.FieldFilterMultiselect,
117
+ },
118
+ },
119
+ {
120
+ ColumnMeta: funcapi.ColumnMeta{
121
+ Name: "errorMessage",
122
+ Tooltip: "Error Message",
123
+ Type: funcapi.FieldTypeString,
124
+ Visible: true,
125
+ Transform: funcapi.FieldTransformNone,
126
+ Sort: funcapi.FieldSortAscending,
127
+ Summary: funcapi.FieldSummaryCount,
128
+ Filter: funcapi.FieldFilterMultiselect,
129
+ FullWidth: true,
130
+ },
131
+ },
132
+ }
133
+}
134
+
135
+func (c *Collector) collectMySQLErrorDetailsForDigests(ctx context.Context, digests []string) (string, map[string]mysqlErrorRow) {
136
+ source, err := c.detectMySQLErrorHistorySource(ctx)
137
+ if err != nil {
138
+ c.Debugf("error attribution: %v", err)
139
+ return mysqlErrorAttrNotEnabled, nil
140
+ }
141
+ if source.status != mysqlErrorAttrEnabled {
142
+ return source.status, nil
143
+ }
144
+
145
+ rows, err := c.fetchMySQLErrorRows(ctx, source, digests, len(digests))
146
+ if err != nil {
147
+ c.Debugf("error attribution query failed: %v", err)
148
+ return mysqlErrorAttrNotEnabled, nil
149
+ }
150
+ if len(rows) == 0 && source.fallbackTable != "" {
151
+ fallback, ferr := c.buildMySQLErrorSource(ctx, source.fallbackTable)
152
+ if ferr == nil && fallback.status == mysqlErrorAttrEnabled {
153
+ fallbackRows, ferr := c.fetchMySQLErrorRows(ctx, fallback, digests, len(digests))
154
+ if ferr == nil && len(fallbackRows) > 0 {
155
+ rows = fallbackRows
156
+ }
157
+ }
158
+ }
159
+
160
+ out := make(map[string]mysqlErrorRow, len(rows))
161
+ for _, row := range rows {
162
+ if row.Digest == "" {
163
+ continue
164
+ }
165
+ if _, ok := out[row.Digest]; ok {
166
+ continue
167
+ }
168
+ out[row.Digest] = row
169
+ }
170
+ return mysqlErrorAttrEnabled, out
171
+}
172
+
173
+func (c *Collector) errorInfoParams(context.Context) ([]funcapi.ParamConfig, error) {
174
+ if !c.Config.GetErrorInfoFunctionEnabled() {
175
+ return nil, fmt.Errorf("error-info function disabled in configuration")
176
+ }
177
+ return []funcapi.ParamConfig{}, nil
178
+}
179
+
180
+func (c *Collector) collectErrorInfo(ctx context.Context) *funcapi.FunctionResponse {
181
+ if !c.Config.GetErrorInfoFunctionEnabled() {
182
+ return &funcapi.FunctionResponse{
183
+ Status: 503,
184
+ Message: "error-info not enabled: function disabled in configuration. " +
185
+ "To enable, set error_info_function_enabled: true in the MySQL collector config.",
186
+ }
187
+ }
188
+
189
+ available, err := c.checkPerformanceSchema(ctx)
190
+ if err != nil {
191
+ return &funcapi.FunctionResponse{
192
+ Status: 500,
193
+ Message: fmt.Sprintf("failed to check performance_schema availability: %v", err),
194
+ }
195
+ }
196
+ if !available {
197
+ return &funcapi.FunctionResponse{Status: 503, Message: "performance_schema is not enabled"}
198
+ }
199
+
200
+ source, err := c.detectMySQLErrorHistorySource(ctx)
201
+ if err != nil {
202
+ return &funcapi.FunctionResponse{Status: 503, Message: fmt.Sprintf("error-info not enabled: %v", err)}
203
+ }
204
+ if source.status != mysqlErrorAttrEnabled {
205
+ msg := "error-info not enabled"
206
+ if source.reason != "" {
207
+ msg = fmt.Sprintf("%s: %s", msg, source.reason)
208
+ }
209
+ return &funcapi.FunctionResponse{Status: 503, Message: msg}
210
+ }
211
+
212
+ limit := c.TopQueriesLimit
213
+ if limit <= 0 {
214
+ limit = 500
215
+ }
216
+
217
+ rows, err := c.fetchMySQLErrorRows(ctx, source, nil, limit)
218
+ if err != nil {
219
+ return &funcapi.FunctionResponse{Status: 500, Message: fmt.Sprintf("error-info query failed: %v", err)}
220
+ }
221
+ if len(rows) == 0 && source.fallbackTable != "" {
222
+ fallback, ferr := c.buildMySQLErrorSource(ctx, source.fallbackTable)
223
+ if ferr == nil && fallback.status == mysqlErrorAttrEnabled {
224
+ fallbackRows, ferr := c.fetchMySQLErrorRows(ctx, fallback, nil, limit)
225
+ if ferr == nil && len(fallbackRows) > 0 {
226
+ rows = fallbackRows
227
+ }
228
+ }
229
+ }
230
+
231
+ data := make([][]any, 0, len(rows))
232
+ for _, row := range rows {
233
+ var errNo any
234
+ if row.ErrorNumber != nil {
235
+ errNo = *row.ErrorNumber
236
+ }
237
+ data = append(data, []any{
238
+ row.Digest,
239
+ row.Query,
240
+ row.Schema,
241
+ errNo,
242
+ row.SQLState,
243
+ row.Message,
244
+ })
245
+ }
246
+
247
+ return &funcapi.FunctionResponse{
248
+ Status: 200,
249
+ Help: "Recent SQL errors from performance_schema statement history tables",
250
+ Columns: buildMySQLErrorInfoColumns(),
251
+ Data: data,
252
+ DefaultSortColumn: "errorNumber",
253
+ }
254
+}
255
+
256
+func buildMySQLErrorInfoColumns() map[string]any {
257
+ columns := map[string]any{
258
+ "digest": funcapi.Column{
259
+ Index: 0,
260
+ Name: "Digest",
261
+ Type: funcapi.FieldTypeString,
262
+ Sortable: true,
263
+ Visible: false,
264
+ UniqueKey: true,
265
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
266
+ }.BuildColumn(),
267
+ "query": funcapi.Column{
268
+ Index: 1,
269
+ Name: "Query",
270
+ Type: funcapi.FieldTypeString,
271
+ Sortable: true,
272
+ Sticky: true,
273
+ FullWidth: true,
274
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
275
+ }.BuildColumn(),
276
+ "schema": funcapi.Column{
277
+ Index: 2,
278
+ Name: "Schema",
279
+ Type: funcapi.FieldTypeString,
280
+ Sortable: true,
281
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
282
+ }.BuildColumn(),
283
+ "errorNumber": funcapi.Column{
284
+ Index: 3,
285
+ Name: "Error Number",
286
+ Type: funcapi.FieldTypeInteger,
287
+ Sortable: true,
288
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNumber},
289
+ }.BuildColumn(),
290
+ "sqlState": funcapi.Column{
291
+ Index: 4,
292
+ Name: "SQL State",
293
+ Type: funcapi.FieldTypeString,
294
+ Sortable: true,
295
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
296
+ }.BuildColumn(),
297
+ "errorMessage": funcapi.Column{
298
+ Index: 5,
299
+ Name: "Error Message",
300
+ Type: funcapi.FieldTypeString,
301
+ Sortable: false,
302
+ FullWidth: true,
303
+ ValueOptions: funcapi.ValueOptions{Transform: funcapi.FieldTransformNone},
304
+ }.BuildColumn(),
305
+ }
306
+ return columns
307
+}
308
+
309
+func (c *Collector) checkPerformanceSchema(ctx context.Context) (bool, error) {
310
+ c.varPerfSchemaMu.RLock()
311
+ cached := c.varPerformanceSchema
312
+ c.varPerfSchemaMu.RUnlock()
313
+ if cached != "" {
314
+ return cached == "ON" || cached == "1", nil
315
+ }
316
+
317
+ c.varPerfSchemaMu.Lock()
318
+ defer c.varPerfSchemaMu.Unlock()
319
+
320
+ if c.varPerformanceSchema != "" {
321
+ return c.varPerformanceSchema == "ON" || c.varPerformanceSchema == "1", nil
322
+ }
323
+
324
+ var value string
325
+ query := "SELECT @@performance_schema"
326
+ if err := c.db.QueryRowContext(ctx, query).Scan(&value); err != nil {
327
+ return false, err
328
+ }
329
+
330
+ c.varPerformanceSchema = value
331
+ return value == "ON" || value == "1", nil
332
+}
333
+
334
+func (c *Collector) detectMySQLErrorHistorySource(ctx context.Context) (mysqlErrorSource, error) {
335
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
336
+ defer cancel()
337
+
338
+ rows, err := c.db.QueryContext(qctx, `
339
+SELECT NAME, ENABLED
340
+FROM performance_schema.setup_consumers
341
+WHERE NAME IN ('events_statements_history_long','events_statements_history','events_statements_current');
342
+`)
343
+ if err != nil {
344
+ return mysqlErrorSource{status: mysqlErrorAttrNotEnabled, reason: "unable to read performance_schema.setup_consumers"}, err
345
+ }
346
+ defer rows.Close()
347
+
348
+ enabled := map[string]bool{}
349
+ present := map[string]bool{}
350
+ for rows.Next() {
351
+ var name, enabledVal string
352
+ if err := rows.Scan(&name, &enabledVal); err != nil {
353
+ return mysqlErrorSource{status: mysqlErrorAttrNotEnabled, reason: "unable to read performance_schema.setup_consumers"}, err
354
+ }
355
+ key := strings.ToLower(name)
356
+ enabled[key] = strings.EqualFold(enabledVal, "YES")
357
+ present[key] = true
358
+ }
359
+ if err := rows.Err(); err != nil {
360
+ return mysqlErrorSource{status: mysqlErrorAttrNotEnabled, reason: "unable to read performance_schema.setup_consumers"}, err
361
+ }
362
+
363
+ table := ""
364
+ fallback := ""
365
+ switch {
366
+ case enabled["events_statements_history_long"]:
367
+ table = "events_statements_history_long"
368
+ if enabled["events_statements_history"] {
369
+ fallback = "events_statements_history"
370
+ }
371
+ case enabled["events_statements_history"]:
372
+ table = "events_statements_history"
373
+ default:
374
+ return mysqlErrorSource{status: mysqlErrorAttrNotEnabled, reason: "statement history consumers are disabled"}, nil
375
+ }
376
+
377
+ source, err := c.buildMySQLErrorSource(ctx, table)
378
+ if err != nil {
379
+ return source, err
380
+ }
381
+ if source.status != mysqlErrorAttrEnabled {
382
+ return source, nil
383
+ }
384
+ source.fallbackTable = fallback
385
+ return source, nil
386
+}
387
+
388
+func (c *Collector) buildMySQLErrorSource(ctx context.Context, table string) (mysqlErrorSource, error) {
389
+ cols, err := c.fetchMySQLTableColumns(ctx, table)
390
+ if err != nil {
391
+ return mysqlErrorSource{status: mysqlErrorAttrNotSupported, reason: "unable to read history table columns"}, err
392
+ }
393
+
394
+ required := []string{"DIGEST", "MYSQL_ERRNO", "MESSAGE_TEXT", "RETURNED_SQLSTATE"}
395
+ for _, key := range required {
396
+ if !cols[key] {
397
+ return mysqlErrorSource{status: mysqlErrorAttrNotSupported, reason: "required history columns are missing"}, nil
398
+ }
399
+ }
400
+
401
+ return mysqlErrorSource{
402
+ table: table,
403
+ columns: cols,
404
+ status: mysqlErrorAttrEnabled,
405
+ }, nil
406
+}
407
+
408
+func (c *Collector) fetchMySQLTableColumns(ctx context.Context, table string) (map[string]bool, error) {
409
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
410
+ defer cancel()
411
+
412
+ rows, err := c.db.QueryContext(qctx, `
413
+SELECT COLUMN_NAME
414
+FROM information_schema.COLUMNS
415
+WHERE TABLE_SCHEMA = 'performance_schema'
416
+ AND TABLE_NAME = ?;`, table)
417
+ if err != nil {
418
+ return nil, err
419
+ }
420
+ defer rows.Close()
421
+
422
+ cols := make(map[string]bool)
423
+ for rows.Next() {
424
+ var name string
425
+ if err := rows.Scan(&name); err != nil {
426
+ return nil, err
427
+ }
428
+ cols[strings.ToUpper(name)] = true
429
+ }
430
+ if err := rows.Err(); err != nil {
431
+ return nil, err
432
+ }
433
+ return cols, nil
434
+}
435
+
436
+func (c *Collector) fetchMySQLErrorRows(ctx context.Context, source mysqlErrorSource, digests []string, limit int) ([]mysqlErrorRow, error) {
437
+ if source.status != mysqlErrorAttrEnabled {
438
+ return nil, fmt.Errorf("error history not enabled")
439
+ }
440
+
441
+ selectCols := []string{"DIGEST", "MYSQL_ERRNO", "RETURNED_SQLSTATE", "MESSAGE_TEXT"}
442
+ if source.columns["DIGEST_TEXT"] {
443
+ selectCols = append(selectCols, "DIGEST_TEXT")
444
+ }
445
+ if source.columns["SCHEMA_NAME"] {
446
+ selectCols = append(selectCols, "SCHEMA_NAME")
447
+ }
448
+ if source.columns["SQL_TEXT"] {
449
+ selectCols = append(selectCols, "SQL_TEXT")
450
+ }
451
+
452
+ orderBy := ""
453
+ switch {
454
+ case source.columns["EVENT_ID"]:
455
+ orderBy = "EVENT_ID DESC"
456
+ case source.columns["TIMER_END"]:
457
+ orderBy = "TIMER_END DESC"
458
+ }
459
+
460
+ var args []any
461
+ var filters []string
462
+ filters = append(filters, "MYSQL_ERRNO <> 0")
463
+ if len(digests) > 0 {
464
+ placeholders := make([]string, 0, len(digests))
465
+ for _, digest := range digests {
466
+ placeholders = append(placeholders, "?")
467
+ args = append(args, digest)
468
+ }
469
+ filters = append(filters, fmt.Sprintf("DIGEST IN (%s)", strings.Join(placeholders, ",")))
470
+ }
471
+
472
+ query := fmt.Sprintf("SELECT %s FROM performance_schema.%s WHERE %s",
473
+ strings.Join(selectCols, ", "),
474
+ source.table,
475
+ strings.Join(filters, " AND "),
476
+ )
477
+ if orderBy != "" {
478
+ query = fmt.Sprintf("%s ORDER BY %s", query, orderBy)
479
+ }
480
+ if limit > 0 {
481
+ query = fmt.Sprintf("%s LIMIT %d", query, limit)
482
+ }
483
+
484
+ qctx, cancel := context.WithTimeout(ctx, c.Timeout.Duration())
485
+ defer cancel()
486
+
487
+ rows, err := c.db.QueryContext(qctx, query, args...)
488
+ if err != nil {
489
+ return nil, err
490
+ }
491
+ defer rows.Close()
492
+
493
+ var results []mysqlErrorRow
494
+ seen := make(map[string]bool)
495
+
496
+ for rows.Next() {
497
+ var (
498
+ digest sql.NullString
499
+ errno sql.NullInt64
500
+ sqlState sql.NullString
501
+ message sql.NullString
502
+ digestText sql.NullString
503
+ schemaName sql.NullString
504
+ sqlText sql.NullString
505
+ scanTargets []any
506
+ )
507
+
508
+ scanTargets = append(scanTargets, &digest, &errno, &sqlState, &message)
509
+ if source.columns["DIGEST_TEXT"] {
510
+ scanTargets = append(scanTargets, &digestText)
511
+ }
512
+ if source.columns["SCHEMA_NAME"] {
513
+ scanTargets = append(scanTargets, &schemaName)
514
+ }
515
+ if source.columns["SQL_TEXT"] {
516
+ scanTargets = append(scanTargets, &sqlText)
517
+ }
518
+
519
+ if err := rows.Scan(scanTargets...); err != nil {
520
+ return nil, err
521
+ }
522
+
523
+ if !digest.Valid || strings.TrimSpace(digest.String) == "" {
524
+ continue
525
+ }
526
+ if seen[digest.String] {
527
+ continue
528
+ }
529
+ seen[digest.String] = true
530
+
531
+ queryText := ""
532
+ switch {
533
+ case digestText.Valid && strings.TrimSpace(digestText.String) != "":
534
+ queryText = digestText.String
535
+ case sqlText.Valid && strings.TrimSpace(sqlText.String) != "":
536
+ queryText = sqlText.String
537
+ }
538
+
539
+ var errNoPtr *int64
540
+ if errno.Valid {
541
+ val := errno.Int64
542
+ errNoPtr = &val
543
+ }
544
+
545
+ row := mysqlErrorRow{
546
+ Digest: digest.String,
547
+ Query: queryText,
548
+ Schema: schemaName.String,
549
+ ErrorNumber: errNoPtr,
550
+ SQLState: sqlState.String,
551
+ Message: message.String,
552
+ }
553
+ results = append(results, row)
554
+ }
555
+
556
+ if err := rows.Err(); err != nil {
557
+ return nil, err
558
+ }
559
+
560
+ return results, nil
561
+}
562
+
563
+func nullableString(value string) any {
564
+ if strings.TrimSpace(value) == "" {
565
+ return nil
566
+ }
567
+ return value
568
+}
src/go/plugin/go.d/collector/mysql/func_router.go
+4
@@ -23,6 +23,8 @@ func newFuncRouter(c *Collector) *funcRouter {
23
handlers: make(map[string]funcapi.MethodHandler),
24
}
25
r.handlers[topQueriesMethodID] = newFuncTopQueries(r)
26
+ r.handlers[deadlockInfoMethodID] = newFuncDeadlockInfo(r)
27
+ r.handlers[errorInfoMethodID] = newFuncErrorInfo(r)
28
return r
29
}
30
@@ -52,6 +54,8 @@ func (r *funcRouter) Cleanup(ctx context.Context) {
54
func mysqlMethods() []funcapi.MethodConfig {
55
return []funcapi.MethodConfig{
56
topQueriesMethodConfig(),
57
+ deadlockInfoMethodConfig(),
58
+ errorInfoMethodConfig(),
59
}
60
}
61
src/go/plugin/go.d/collector/mysql/func_top_queries.go
+63
@@ -286,6 +286,69 @@ func (f *funcTopQueries) collectData(ctx context.Context, sortColumn string) *fu
286
return &funcapi.FunctionResponse{Status: 500, Message: err.Error()}
287
}
288
289
+ errorCols := mysqlErrorAttributionColumns()
290
+ errorStatus := mysqlErrorAttrNotSupported
291
+ errorDetails := map[string]mysqlErrorRow{}
292
+ digestIdx := -1
293
+ for i, col := range cols {
294
+ if col.Name == "digest" {
295
+ digestIdx = i
296
+ break
297
+ }
298
+ }
299
+ if digestIdx >= 0 {
300
+ digests := make([]string, 0, len(data))
301
+ seen := make(map[string]bool)
302
+ for _, row := range data {
303
+ if digestIdx >= len(row) {
304
+ continue
305
+ }
306
+ digest, ok := row[digestIdx].(string)
307
+ if !ok || digest == "" {
308
+ continue
309
+ }
310
+ if seen[digest] {
311
+ continue
312
+ }
313
+ seen[digest] = true
314
+ digests = append(digests, digest)
315
+ }
316
+ if len(digests) > 0 {
317
+ errorStatus, errorDetails = f.router.collector.collectMySQLErrorDetailsForDigests(ctx, digests)
318
+ } else {
319
+ errorStatus = mysqlErrorAttrNoData
320
+ }
321
+ }
322
+
323
+ if len(errorCols) > 0 {
324
+ for i := range data {
325
+ status := errorStatus
326
+ var errRow mysqlErrorRow
327
+ var errNo any
328
+ if digestIdx >= 0 && digestIdx < len(data[i]) {
329
+ if digest, ok := data[i][digestIdx].(string); ok && digest != "" {
330
+ if row, ok := errorDetails[digest]; ok {
331
+ status = mysqlErrorAttrEnabled
332
+ errRow = row
333
+ if errRow.ErrorNumber != nil {
334
+ errNo = *errRow.ErrorNumber
335
+ }
336
+ } else if status == mysqlErrorAttrEnabled {
337
+ status = mysqlErrorAttrNoData
338
+ }
339
+ }
340
+ }
341
+
342
+ data[i] = append(data[i],
343
+ status,
344
+ errNo,
345
+ nullableString(errRow.SQLState),
346
+ nullableString(errRow.Message),
347
+ )
348
+ }
349
+ cols = append(cols, errorCols...)
350
+ }
351
+
352
// Build dynamic sort options from available columns (only those actually detected)
353
sortParam := f.buildSortParam(cols)
354
sortOptions := sortParam.Options
src/go/plugin/go.d/collector/mysql/func_top_queries_test.go
+38
-10
@@ -7,26 +7,54 @@ import (
7
8
"github.com/netdata/netdata/go/plugins/pkg/funcapi"
9
"github.com/stretchr/testify/assert"
10
+ "github.com/stretchr/testify/require"
11
)
12
13
func TestMySQLMethods(t *testing.T) {
14
methods := mysqlMethods()
15
15
- require := assert.New(t)
16
- require.Len(methods, 1)
17
- require.Equal("top-queries", methods[0].ID)
18
- require.Equal("Top Queries", methods[0].Name)
19
- require.NotEmpty(methods[0].RequiredParams)
16
+ req := require.New(t)
17
+ req.Len(methods, 3)
18
+
19
+ topIdx := -1
20
+ deadlockIdx := -1
21
+ errorIdx := -1
22
+ for i := range methods {
23
+ switch methods[i].ID {
24
+ case "top-queries":
25
+ topIdx = i
26
+ case "deadlock-info":
27
+ deadlockIdx = i
28
+ case "error-info":
29
+ errorIdx = i
30
+ }
31
+ }
32
+
33
+ req.NotEqual(-1, topIdx, "expected top-queries method")
34
+ req.NotEqual(-1, deadlockIdx, "expected deadlock-info method")
35
+ req.NotEqual(-1, errorIdx, "expected error-info method")
36
+
37
+ topMethod := methods[topIdx]
38
+ req.Equal("Top Queries", topMethod.Name)
39
+ req.NotEmpty(topMethod.RequiredParams)
40
+
41
+ deadlockMethod := methods[deadlockIdx]
42
+ req.Equal("Deadlock Info", deadlockMethod.Name)
43
+ req.Empty(deadlockMethod.RequiredParams)
44
+
45
+ errorMethod := methods[errorIdx]
46
+ req.Equal("Error Info", errorMethod.Name)
47
+ req.Empty(errorMethod.RequiredParams)
48
49
var sortParam *funcapi.ParamConfig
22
- for i := range methods[0].RequiredParams {
23
- if methods[0].RequiredParams[i].ID == "__sort" {
24
- sortParam = &methods[0].RequiredParams[i]
50
+ for i := range topMethod.RequiredParams {
51
+ if topMethod.RequiredParams[i].ID == "__sort" {
52
+ sortParam = &topMethod.RequiredParams[i]
53
break
54
}
55
}
28
- require.NotNil(sortParam, "expected __sort required param")
29
- require.NotEmpty(sortParam.Options)
56
+ req.NotNil(sortParam, "expected __sort required param")
57
+ req.NotEmpty(sortParam.Options)
58
}
59
60
func TestTopQueriesColumns_HasRequiredColumns(t *testing.T) {
src/go/plugin/go.d/collector/mysql/integrations/mariadb.md
+122
-2
@@ -227,8 +227,7 @@ Performance Schema must be enabled and statement instrumentation must be configu
227
WHERE NAME LIKE '%statement%';
228
```
229
230
-3. The following consumers should be enabled:
231
- - `events_statements_current`
230
+3. The following consumer should be enabled:
231
- `events_statements_summary_by_digest`
232
233
4. Enable statement consumers if needed:
@@ -306,6 +305,10 @@ Aggregated statement statistics from Performance Schema, grouped by query digest
305
| Lock Time | duration | milliseconds | | Total time spent waiting for table locks across all executions. High lock time may indicate contention from concurrent transactions. |
306
| Errors | integer | | | Total number of times this query pattern resulted in an error. Non-zero values require investigation into the underlying issue. |
307
| Warnings | integer | | | Total number of times this query pattern generated warnings. Warnings may indicate data type conversions, NULL handling issues, or other non-critical problems. |
308
+| Error Attribution | string | | | Status of error detail attribution for this query. Values: enabled (error details available), no_data (no recent error for this digest), not_enabled (statement history consumers disabled), not_supported (required columns unavailable). |
309
+| Error Number | integer | | | Most recent error number observed for this query digest (when error attribution is enabled). |
310
+| SQL State | string | | hidden | SQLSTATE code for the most recent error (when error attribution is enabled). |
311
+| Error Message | string | | | Most recent error message for this query digest (when error attribution is enabled). |
312
| Rows Affected | integer | | | Total number of rows modified by INSERT, UPDATE, DELETE, or REPLACE statements. Useful for tracking write workloads. |
313
| Rows Sent | integer | | | Total number of rows returned to the client by SELECT statements. High values may indicate result sets that are too large. |
314
| Rows Examined | integer | | | Total number of rows read during query execution. A high ratio of rows examined to rows sent suggests missing or inefficient indexes. |
@@ -334,6 +337,123 @@ Aggregated statement statistics from Performance Schema, grouped by query digest
337
| Max Controlled Memory | integer | | | Maximum memory controlled by the query executor for this query pattern. Available in MySQL 8.0.31+. Helps identify memory-intensive operations. |
338
| Max Total Memory | integer | | | Maximum total memory used by this query pattern including both controlled and uncontrolled allocations. Available in MySQL 8.0.31+. |
339
340
+### Deadlock Info
341
+
342
+Retrieves the latest detected InnoDB deadlock from `SHOW ENGINE INNODB STATUS`.
343
+
344
+The output is parsed to attribute the deadlock to the participating transactions and their query text, lock mode, lock status, and wait resource.
345
+
346
+Use cases:
347
+- Identify which query was chosen as the deadlock victim
348
+- Inspect the waiting lock resource and lock mode
349
+- Correlate deadlocks with application changes or deployment events
350
+
351
+Query text is truncated at 4096 characters for display purposes.
352
+
353
+
354
+| Aspect | Description |
355
+|:-------|:------------|
356
+| Name | `Mysql:deadlock-info` |
357
+| Require Cloud | yes |
358
+| Performance | Executes `SHOW ENGINE INNODB STATUS` on demand:<br/>• Not part of regular collection<br/>• Query cost depends on server load and the size of the InnoDB status output |
359
+| Security | Query text and wait resource strings may include unmasked literal values including sensitive data (PII/secrets):<br/>• SQL literals such as emails, IDs, or tokens<br/>• Schema and table names that may be sensitive in some environments<br/>• Restrict dashboard access to authorized personnel only |
360
+| Availability | Available when:<br/>• The collector has successfully connected to MySQL<br/>• `deadlock_info_function_enabled` is true<br/>• The account can run `SHOW ENGINE INNODB STATUS` (PROCESS privilege)<br/>• Returns HTTP 200 with empty data when no deadlock is found<br/>• Returns HTTP 403 when PROCESS privilege is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out<br/>• Returns HTTP 561 when the deadlock section cannot be parsed<br/>• Returns HTTP 503 if the collector is still initializing or the function is disabled |
361
+
362
+#### Prerequisites
363
+
364
+##### Enable deadlock-info function in Netdata
365
+
366
+Set `deadlock_info_function_enabled: true` in the `go.d/mysql.conf` job.
367
+
368
+
369
+##### Grant PROCESS privilege
370
+
371
+The monitoring user must have PROCESS privilege to run `SHOW ENGINE INNODB STATUS`.
372
+
373
+
374
+
375
+#### Parameters
376
+
377
+This function has no parameters.
378
+
379
+#### Returns
380
+
381
+Parsed deadlock participants from the latest detected deadlock. Each row represents one transaction involved in the deadlock.
382
+
383
+| Column | Type | Unit | Visibility | Description |
384
+|:-------|:-----|:-----|:-----------|:------------|
385
+| Row ID | string | | hidden | Unique row identifier composed of deadlock ID and process ID. |
386
+| Deadlock ID | string | | | Identifier for the deadlock event, used to group participating transactions. |
387
+| Timestamp | timestamp | | | Timestamp of the deadlock event. Parsed from the deadlock section when available; otherwise the function execution time. |
388
+| Process ID | string | | | MySQL thread id of the transaction involved in the deadlock. |
389
+| Connection ID | integer | | | Numeric connection identifier when the process id is numeric. |
390
+| ECID | integer | | | Execution context id (engine-specific). This is typically null for MySQL and reserved for cross-engine consistency. |
391
+| Victim | string | | | "true" when the transaction was chosen as the deadlock victim and rolled back; otherwise "false". |
392
+| Query | string | | | SQL query text for the transaction involved in the deadlock. Truncated to 4096 characters. |
393
+| Lock Mode | string | | | Lock mode reported for the waiting lock (for example X or S). |
394
+| Lock Status | string | | | Lock status for the transaction. WAITING indicates the transaction was waiting on a lock. |
395
+| Wait Resource | string | | | Lock resource line from InnoDB status showing what the transaction was waiting on. |
396
+| Database | string | | | Database name when it can be inferred. This may be empty or null depending on the deadlock output. |
397
+
398
+### Error Info
399
+
400
+Retrieves recent SQL errors from Performance Schema statement history tables.
401
+
402
+This function reads `performance_schema.events_statements_history_long` when enabled,
403
+otherwise falls back to `performance_schema.events_statements_history`. It reports the
404
+most recent error per query digest, including error number, SQLSTATE, and message.
405
+
406
+Use cases:
407
+- Identify recent query errors and their messages
408
+- Correlate errors to query patterns (digest)
409
+- Validate error rates seen in top-queries
410
+
411
+Error messages are truncated by Performance Schema (usually 128 characters).
412
+
413
+
414
+| Aspect | Description |
415
+|:-------|:------------|
416
+| Name | `Mysql:error-info` |
417
+| Require Cloud | yes |
418
+| Performance | Reads Performance Schema statement history tables on demand:<br/>• Not part of regular collection<br/>• Query cost depends on history table size and server load |
419
+| Security | Error messages and query text may include unmasked literals (PII/secrets).<br/>• Restrict dashboard access to authorized personnel only |
420
+| Availability | Available when:<br/>• The collector has successfully connected to MySQL<br/>• `error_info_function_enabled` is true<br/>• Performance Schema statement history consumers are enabled (history and/or history_long)<br/>• Returns HTTP 200 with empty data when no errors are found<br/>• Returns HTTP 503 when required consumers are not enabled or function disabled<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out |
421
+
422
+#### Prerequisites
423
+
424
+##### Enable error-info function in Netdata
425
+
426
+Set `error_info_function_enabled: true` in the `go.d/mysql.conf` job.
427
+
428
+
429
+##### Enable statement history consumers
430
+
431
+Ensure `events_statements_history` and/or `events_statements_history_long` consumers are enabled.
432
+
433
+
434
+##### Grant SELECT on Performance Schema
435
+
436
+The monitoring user must have SELECT on `performance_schema.*` to read statement history tables.
437
+
438
+
439
+
440
+#### Parameters
441
+
442
+This function has no parameters.
443
+
444
+#### Returns
445
+
446
+Most recent error per query digest from Performance Schema history tables.
447
+
448
+| Column | Type | Unit | Visibility | Description |
449
+|:-------|:-----|:-----|:-----------|:------------|
450
+| Digest | string | | hidden | Unique hash identifier for the normalized query pattern. |
451
+| Query | string | | | Normalized query text when available (digest text or SQL text). |
452
+| Schema | string | | | Database schema name when available. |
453
+| Error Number | integer | | | MySQL error number for the most recent error of this digest. |
454
+| SQL State | string | | | SQLSTATE code for the most recent error. |
455
+| Error Message | string | | | Error message for the most recent error. |
456
+
457
458
459
## Alerts
src/go/plugin/go.d/collector/mysql/integrations/mysql.md
+122
-2
@@ -227,8 +227,7 @@ Performance Schema must be enabled and statement instrumentation must be configu
227
WHERE NAME LIKE '%statement%';
228
```
229
230
-3. The following consumers should be enabled:
231
- - `events_statements_current`
230
+3. The following consumer should be enabled:
231
- `events_statements_summary_by_digest`
232
233
4. Enable statement consumers if needed:
@@ -306,6 +305,10 @@ Aggregated statement statistics from Performance Schema, grouped by query digest
305
| Lock Time | duration | milliseconds | | Total time spent waiting for table locks across all executions. High lock time may indicate contention from concurrent transactions. |
306
| Errors | integer | | | Total number of times this query pattern resulted in an error. Non-zero values require investigation into the underlying issue. |
307
| Warnings | integer | | | Total number of times this query pattern generated warnings. Warnings may indicate data type conversions, NULL handling issues, or other non-critical problems. |
308
+| Error Attribution | string | | | Status of error detail attribution for this query. Values: enabled (error details available), no_data (no recent error for this digest), not_enabled (statement history consumers disabled), not_supported (required columns unavailable). |
309
+| Error Number | integer | | | Most recent error number observed for this query digest (when error attribution is enabled). |
310
+| SQL State | string | | hidden | SQLSTATE code for the most recent error (when error attribution is enabled). |
311
+| Error Message | string | | | Most recent error message for this query digest (when error attribution is enabled). |
312
| Rows Affected | integer | | | Total number of rows modified by INSERT, UPDATE, DELETE, or REPLACE statements. Useful for tracking write workloads. |
313
| Rows Sent | integer | | | Total number of rows returned to the client by SELECT statements. High values may indicate result sets that are too large. |
314
| Rows Examined | integer | | | Total number of rows read during query execution. A high ratio of rows examined to rows sent suggests missing or inefficient indexes. |
@@ -334,6 +337,123 @@ Aggregated statement statistics from Performance Schema, grouped by query digest
337
| Max Controlled Memory | integer | | | Maximum memory controlled by the query executor for this query pattern. Available in MySQL 8.0.31+. Helps identify memory-intensive operations. |
338
| Max Total Memory | integer | | | Maximum total memory used by this query pattern including both controlled and uncontrolled allocations. Available in MySQL 8.0.31+. |
339
340
+### Deadlock Info
341
+
342
+Retrieves the latest detected InnoDB deadlock from `SHOW ENGINE INNODB STATUS`.
343
+
344
+The output is parsed to attribute the deadlock to the participating transactions and their query text, lock mode, lock status, and wait resource.
345
+
346
+Use cases:
347
+- Identify which query was chosen as the deadlock victim
348
+- Inspect the waiting lock resource and lock mode
349
+- Correlate deadlocks with application changes or deployment events
350
+
351
+Query text is truncated at 4096 characters for display purposes.
352
+
353
+
354
+| Aspect | Description |
355
+|:-------|:------------|
356
+| Name | `Mysql:deadlock-info` |
357
+| Require Cloud | yes |
358
+| Performance | Executes `SHOW ENGINE INNODB STATUS` on demand:<br/>• Not part of regular collection<br/>• Query cost depends on server load and the size of the InnoDB status output |
359
+| Security | Query text and wait resource strings may include unmasked literal values including sensitive data (PII/secrets):<br/>• SQL literals such as emails, IDs, or tokens<br/>• Schema and table names that may be sensitive in some environments<br/>• Restrict dashboard access to authorized personnel only |
360
+| Availability | Available when:<br/>• The collector has successfully connected to MySQL<br/>• `deadlock_info_function_enabled` is true<br/>• The account can run `SHOW ENGINE INNODB STATUS` (PROCESS privilege)<br/>• Returns HTTP 200 with empty data when no deadlock is found<br/>• Returns HTTP 403 when PROCESS privilege is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out<br/>• Returns HTTP 561 when the deadlock section cannot be parsed<br/>• Returns HTTP 503 if the collector is still initializing or the function is disabled |
361
+
362
+#### Prerequisites
363
+
364
+##### Enable deadlock-info function in Netdata
365
+
366
+Set `deadlock_info_function_enabled: true` in the `go.d/mysql.conf` job.
367
+
368
+
369
+##### Grant PROCESS privilege
370
+
371
+The monitoring user must have PROCESS privilege to run `SHOW ENGINE INNODB STATUS`.
372
+
373
+
374
+
375
+#### Parameters
376
+
377
+This function has no parameters.
378
+
379
+#### Returns
380
+
381
+Parsed deadlock participants from the latest detected deadlock. Each row represents one transaction involved in the deadlock.
382
+
383
+| Column | Type | Unit | Visibility | Description |
384
+|:-------|:-----|:-----|:-----------|:------------|
385
+| Row ID | string | | hidden | Unique row identifier composed of deadlock ID and process ID. |
386
+| Deadlock ID | string | | | Identifier for the deadlock event, used to group participating transactions. |
387
+| Timestamp | timestamp | | | Timestamp of the deadlock event. Parsed from the deadlock section when available; otherwise the function execution time. |
388
+| Process ID | string | | | MySQL thread id of the transaction involved in the deadlock. |
389
+| Connection ID | integer | | | Numeric connection identifier when the process id is numeric. |
390
+| ECID | integer | | | Execution context id (engine-specific). This is typically null for MySQL and reserved for cross-engine consistency. |
391
+| Victim | string | | | "true" when the transaction was chosen as the deadlock victim and rolled back; otherwise "false". |
392
+| Query | string | | | SQL query text for the transaction involved in the deadlock. Truncated to 4096 characters. |
393
+| Lock Mode | string | | | Lock mode reported for the waiting lock (for example X or S). |
394
+| Lock Status | string | | | Lock status for the transaction. WAITING indicates the transaction was waiting on a lock. |
395
+| Wait Resource | string | | | Lock resource line from InnoDB status showing what the transaction was waiting on. |
396
+| Database | string | | | Database name when it can be inferred. This may be empty or null depending on the deadlock output. |
397
+
398
+### Error Info
399
+
400
+Retrieves recent SQL errors from Performance Schema statement history tables.
401
+
402
+This function reads `performance_schema.events_statements_history_long` when enabled,
403
+otherwise falls back to `performance_schema.events_statements_history`. It reports the
404
+most recent error per query digest, including error number, SQLSTATE, and message.
405
+
406
+Use cases:
407
+- Identify recent query errors and their messages
408
+- Correlate errors to query patterns (digest)
409
+- Validate error rates seen in top-queries
410
+
411
+Error messages are truncated by Performance Schema (usually 128 characters).
412
+
413
+
414
+| Aspect | Description |
415
+|:-------|:------------|
416
+| Name | `Mysql:error-info` |
417
+| Require Cloud | yes |
418
+| Performance | Reads Performance Schema statement history tables on demand:<br/>• Not part of regular collection<br/>• Query cost depends on history table size and server load |
419
+| Security | Error messages and query text may include unmasked literals (PII/secrets).<br/>• Restrict dashboard access to authorized personnel only |
420
+| Availability | Available when:<br/>• The collector has successfully connected to MySQL<br/>• `error_info_function_enabled` is true<br/>• Performance Schema statement history consumers are enabled (history and/or history_long)<br/>• Returns HTTP 200 with empty data when no errors are found<br/>• Returns HTTP 503 when required consumers are not enabled or function disabled<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out |
421
+
422
+#### Prerequisites
423
+
424
+##### Enable error-info function in Netdata
425
+
426
+Set `error_info_function_enabled: true` in the `go.d/mysql.conf` job.
427
+
428
+
429
+##### Enable statement history consumers
430
+
431
+Ensure `events_statements_history` and/or `events_statements_history_long` consumers are enabled.
432
+
433
+
434
+##### Grant SELECT on Performance Schema
435
+
436
+The monitoring user must have SELECT on `performance_schema.*` to read statement history tables.
437
+
438
+
439
+
440
+#### Parameters
441
+
442
+This function has no parameters.
443
+
444
+#### Returns
445
+
446
+Most recent error per query digest from Performance Schema history tables.
447
+
448
+| Column | Type | Unit | Visibility | Description |
449
+|:-------|:-----|:-----|:-----------|:------------|
450
+| Digest | string | | hidden | Unique hash identifier for the normalized query pattern. |
451
+| Query | string | | | Normalized query text when available (digest text or SQL text). |
452
+| Schema | string | | | Database schema name when available. |
453
+| Error Number | integer | | | MySQL error number for the most recent error of this digest. |
454
+| SQL State | string | | | SQLSTATE code for the most recent error. |
455
+| Error Message | string | | | Error message for the most recent error. |
456
+
457
458
459
## Alerts
src/go/plugin/go.d/collector/mysql/integrations/percona_mysql.md
+122
-2
@@ -227,8 +227,7 @@ Performance Schema must be enabled and statement instrumentation must be configu
227
WHERE NAME LIKE '%statement%';
228
```
229
230
-3. The following consumers should be enabled:
231
- - `events_statements_current`
230
+3. The following consumer should be enabled:
231
- `events_statements_summary_by_digest`
232
233
4. Enable statement consumers if needed:
@@ -306,6 +305,10 @@ Aggregated statement statistics from Performance Schema, grouped by query digest
305
| Lock Time | duration | milliseconds | | Total time spent waiting for table locks across all executions. High lock time may indicate contention from concurrent transactions. |
306
| Errors | integer | | | Total number of times this query pattern resulted in an error. Non-zero values require investigation into the underlying issue. |
307
| Warnings | integer | | | Total number of times this query pattern generated warnings. Warnings may indicate data type conversions, NULL handling issues, or other non-critical problems. |
308
+| Error Attribution | string | | | Status of error detail attribution for this query. Values: enabled (error details available), no_data (no recent error for this digest), not_enabled (statement history consumers disabled), not_supported (required columns unavailable). |
309
+| Error Number | integer | | | Most recent error number observed for this query digest (when error attribution is enabled). |
310
+| SQL State | string | | hidden | SQLSTATE code for the most recent error (when error attribution is enabled). |
311
+| Error Message | string | | | Most recent error message for this query digest (when error attribution is enabled). |
312
| Rows Affected | integer | | | Total number of rows modified by INSERT, UPDATE, DELETE, or REPLACE statements. Useful for tracking write workloads. |
313
| Rows Sent | integer | | | Total number of rows returned to the client by SELECT statements. High values may indicate result sets that are too large. |
314
| Rows Examined | integer | | | Total number of rows read during query execution. A high ratio of rows examined to rows sent suggests missing or inefficient indexes. |
@@ -334,6 +337,123 @@ Aggregated statement statistics from Performance Schema, grouped by query digest
337
| Max Controlled Memory | integer | | | Maximum memory controlled by the query executor for this query pattern. Available in MySQL 8.0.31+. Helps identify memory-intensive operations. |
338
| Max Total Memory | integer | | | Maximum total memory used by this query pattern including both controlled and uncontrolled allocations. Available in MySQL 8.0.31+. |
339
340
+### Deadlock Info
341
+
342
+Retrieves the latest detected InnoDB deadlock from `SHOW ENGINE INNODB STATUS`.
343
+
344
+The output is parsed to attribute the deadlock to the participating transactions and their query text, lock mode, lock status, and wait resource.
345
+
346
+Use cases:
347
+- Identify which query was chosen as the deadlock victim
348
+- Inspect the waiting lock resource and lock mode
349
+- Correlate deadlocks with application changes or deployment events
350
+
351
+Query text is truncated at 4096 characters for display purposes.
352
+
353
+
354
+| Aspect | Description |
355
+|:-------|:------------|
356
+| Name | `Mysql:deadlock-info` |
357
+| Require Cloud | yes |
358
+| Performance | Executes `SHOW ENGINE INNODB STATUS` on demand:<br/>• Not part of regular collection<br/>• Query cost depends on server load and the size of the InnoDB status output |
359
+| Security | Query text and wait resource strings may include unmasked literal values including sensitive data (PII/secrets):<br/>• SQL literals such as emails, IDs, or tokens<br/>• Schema and table names that may be sensitive in some environments<br/>• Restrict dashboard access to authorized personnel only |
360
+| Availability | Available when:<br/>• The collector has successfully connected to MySQL<br/>• `deadlock_info_function_enabled` is true<br/>• The account can run `SHOW ENGINE INNODB STATUS` (PROCESS privilege)<br/>• Returns HTTP 200 with empty data when no deadlock is found<br/>• Returns HTTP 403 when PROCESS privilege is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out<br/>• Returns HTTP 561 when the deadlock section cannot be parsed<br/>• Returns HTTP 503 if the collector is still initializing or the function is disabled |
361
+
362
+#### Prerequisites
363
+
364
+##### Enable deadlock-info function in Netdata
365
+
366
+Set `deadlock_info_function_enabled: true` in the `go.d/mysql.conf` job.
367
+
368
+
369
+##### Grant PROCESS privilege
370
+
371
+The monitoring user must have PROCESS privilege to run `SHOW ENGINE INNODB STATUS`.
372
+
373
+
374
+
375
+#### Parameters
376
+
377
+This function has no parameters.
378
+
379
+#### Returns
380
+
381
+Parsed deadlock participants from the latest detected deadlock. Each row represents one transaction involved in the deadlock.
382
+
383
+| Column | Type | Unit | Visibility | Description |
384
+|:-------|:-----|:-----|:-----------|:------------|
385
+| Row ID | string | | hidden | Unique row identifier composed of deadlock ID and process ID. |
386
+| Deadlock ID | string | | | Identifier for the deadlock event, used to group participating transactions. |
387
+| Timestamp | timestamp | | | Timestamp of the deadlock event. Parsed from the deadlock section when available; otherwise the function execution time. |
388
+| Process ID | string | | | MySQL thread id of the transaction involved in the deadlock. |
389
+| Connection ID | integer | | | Numeric connection identifier when the process id is numeric. |
390
+| ECID | integer | | | Execution context id (engine-specific). This is typically null for MySQL and reserved for cross-engine consistency. |
391
+| Victim | string | | | "true" when the transaction was chosen as the deadlock victim and rolled back; otherwise "false". |
392
+| Query | string | | | SQL query text for the transaction involved in the deadlock. Truncated to 4096 characters. |
393
+| Lock Mode | string | | | Lock mode reported for the waiting lock (for example X or S). |
394
+| Lock Status | string | | | Lock status for the transaction. WAITING indicates the transaction was waiting on a lock. |
395
+| Wait Resource | string | | | Lock resource line from InnoDB status showing what the transaction was waiting on. |
396
+| Database | string | | | Database name when it can be inferred. This may be empty or null depending on the deadlock output. |
397
+
398
+### Error Info
399
+
400
+Retrieves recent SQL errors from Performance Schema statement history tables.
401
+
402
+This function reads `performance_schema.events_statements_history_long` when enabled,
403
+otherwise falls back to `performance_schema.events_statements_history`. It reports the
404
+most recent error per query digest, including error number, SQLSTATE, and message.
405
+
406
+Use cases:
407
+- Identify recent query errors and their messages
408
+- Correlate errors to query patterns (digest)
409
+- Validate error rates seen in top-queries
410
+
411
+Error messages are truncated by Performance Schema (usually 128 characters).
412
+
413
+
414
+| Aspect | Description |
415
+|:-------|:------------|
416
+| Name | `Mysql:error-info` |
417
+| Require Cloud | yes |
418
+| Performance | Reads Performance Schema statement history tables on demand:<br/>• Not part of regular collection<br/>• Query cost depends on history table size and server load |
419
+| Security | Error messages and query text may include unmasked literals (PII/secrets).<br/>• Restrict dashboard access to authorized personnel only |
420
+| Availability | Available when:<br/>• The collector has successfully connected to MySQL<br/>• `error_info_function_enabled` is true<br/>• Performance Schema statement history consumers are enabled (history and/or history_long)<br/>• Returns HTTP 200 with empty data when no errors are found<br/>• Returns HTTP 503 when required consumers are not enabled or function disabled<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out |
421
+
422
+#### Prerequisites
423
+
424
+##### Enable error-info function in Netdata
425
+
426
+Set `error_info_function_enabled: true` in the `go.d/mysql.conf` job.
427
+
428
+
429
+##### Enable statement history consumers
430
+
431
+Ensure `events_statements_history` and/or `events_statements_history_long` consumers are enabled.
432
+
433
+
434
+##### Grant SELECT on Performance Schema
435
+
436
+The monitoring user must have SELECT on `performance_schema.*` to read statement history tables.
437
+
438
+
439
+
440
+#### Parameters
441
+
442
+This function has no parameters.
443
+
444
+#### Returns
445
+
446
+Most recent error per query digest from Performance Schema history tables.
447
+
448
+| Column | Type | Unit | Visibility | Description |
449
+|:-------|:-----|:-----|:-----------|:------------|
450
+| Digest | string | | hidden | Unique hash identifier for the normalized query pattern. |
451
+| Query | string | | | Normalized query text when available (digest text or SQL text). |
452
+| Schema | string | | | Database schema name when available. |
453
+| Error Number | integer | | | MySQL error number for the most recent error of this digest. |
454
+| SQL State | string | | | SQLSTATE code for the most recent error. |
455
+| Error Message | string | | | Error message for the most recent error. |
456
+
457
458
459
## Alerts
src/go/plugin/go.d/collector/mysql/metadata.yaml
+161
-2
@@ -304,6 +304,23 @@ modules:
304
type: integer
305
unit: ""
306
description: "Total number of times this query pattern generated warnings. Warnings may indicate data type conversions, NULL handling issues, or other non-critical problems."
307
+ - name: Error Attribution
308
+ type: string
309
+ unit: ""
310
+ description: "Status of error detail attribution for this query. Values: enabled (error details available), no_data (no recent error for this digest), not_enabled (statement history consumers disabled), not_supported (required columns unavailable)."
311
+ - name: Error Number
312
+ type: integer
313
+ unit: ""
314
+ description: "Most recent error number observed for this query digest (when error attribution is enabled)."
315
+ - name: SQL State
316
+ type: string
317
+ unit: ""
318
+ visibility: hidden
319
+ description: "SQLSTATE code for the most recent error (when error attribution is enabled)."
320
+ - name: Error Message
321
+ type: string
322
+ unit: ""
323
+ description: "Most recent error message for this query digest (when error attribution is enabled)."
324
- name: Rows Affected
325
type: integer
326
unit: ""
@@ -446,8 +463,7 @@ modules:
463
WHERE NAME LIKE '%statement%';
464
```
465
449
- 3. The following consumers should be enabled:
450
- - `events_statements_current`
466
+ 3. The following consumer should be enabled:
467
- `events_statements_summary_by_digest`
468
469
4. Enable statement consumers if needed:
@@ -500,6 +516,149 @@ modules:
516
availability: |
517
Available when:<br/>• The collector has successfully connected to MySQL<br/>• Performance Schema is enabled with statement digest collection<br/>• Returns HTTP 503 if collector is still initializing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out
518
require_cloud: true
519
+ - id: deadlock-info
520
+ name: Deadlock Info
521
+ description: |
522
+ Retrieves the latest detected InnoDB deadlock from `SHOW ENGINE INNODB STATUS`.
523
+
524
+ The output is parsed to attribute the deadlock to the participating transactions and their query text, lock mode, lock status, and wait resource.
525
+
526
+ Use cases:
527
+ - Identify which query was chosen as the deadlock victim
528
+ - Inspect the waiting lock resource and lock mode
529
+ - Correlate deadlocks with application changes or deployment events
530
+
531
+ Query text is truncated at 4096 characters for display purposes.
532
+ parameters: []
533
+ returns:
534
+ description: Parsed deadlock participants from the latest detected deadlock. Each row represents one transaction involved in the deadlock.
535
+ columns:
536
+ - name: Row ID
537
+ type: string
538
+ unit: ""
539
+ visibility: hidden
540
+ description: "Unique row identifier composed of deadlock ID and process ID."
541
+ - name: Deadlock ID
542
+ type: string
543
+ unit: ""
544
+ description: "Identifier for the deadlock event, used to group participating transactions."
545
+ - name: Timestamp
546
+ type: timestamp
547
+ unit: ""
548
+ description: "Timestamp of the deadlock event. Parsed from the deadlock section when available; otherwise the function execution time."
549
+ - name: Process ID
550
+ type: string
551
+ unit: ""
552
+ description: "MySQL thread id of the transaction involved in the deadlock."
553
+ - name: Connection ID
554
+ type: integer
555
+ unit: ""
556
+ description: "Numeric connection identifier when the process id is numeric."
557
+ - name: ECID
558
+ type: integer
559
+ unit: ""
560
+ description: "Execution context id (engine-specific). This is typically null for MySQL and reserved for cross-engine consistency."
561
+ - name: Victim
562
+ type: string
563
+ unit: ""
564
+ description: "\"true\" when the transaction was chosen as the deadlock victim and rolled back; otherwise \"false\"."
565
+ - name: Query
566
+ type: string
567
+ unit: ""
568
+ description: "SQL query text for the transaction involved in the deadlock. Truncated to 4096 characters."
569
+ - name: Lock Mode
570
+ type: string
571
+ unit: ""
572
+ description: "Lock mode reported for the waiting lock (for example X or S)."
573
+ - name: Lock Status
574
+ type: string
575
+ unit: ""
576
+ description: "Lock status for the transaction. WAITING indicates the transaction was waiting on a lock."
577
+ - name: Wait Resource
578
+ type: string
579
+ unit: ""
580
+ description: "Lock resource line from InnoDB status showing what the transaction was waiting on."
581
+ - name: Database
582
+ type: string
583
+ unit: ""
584
+ description: "Database name when it can be inferred. This may be empty or null depending on the deadlock output."
585
+ performance: |
586
+ Executes `SHOW ENGINE INNODB STATUS` on demand:<br/>• Not part of regular collection<br/>• Query cost depends on server load and the size of the InnoDB status output
587
+ security: |
588
+ Query text and wait resource strings may include unmasked literal values including sensitive data (PII/secrets):<br/>• SQL literals such as emails, IDs, or tokens<br/>• Schema and table names that may be sensitive in some environments<br/>• Restrict dashboard access to authorized personnel only
589
+ prerequisites:
590
+ list:
591
+ - title: Enable deadlock-info function in Netdata
592
+ description: |
593
+ Set `deadlock_info_function_enabled: true` in the `go.d/mysql.conf` job.
594
+ - title: Grant PROCESS privilege
595
+ description: |
596
+ The monitoring user must have PROCESS privilege to run `SHOW ENGINE INNODB STATUS`.
597
+ availability: |
598
+ Available when:<br/>• The collector has successfully connected to MySQL<br/>• `deadlock_info_function_enabled` is true<br/>• The account can run `SHOW ENGINE INNODB STATUS` (PROCESS privilege)<br/>• Returns HTTP 200 with empty data when no deadlock is found<br/>• Returns HTTP 403 when PROCESS privilege is missing<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out<br/>• Returns HTTP 561 when the deadlock section cannot be parsed<br/>• Returns HTTP 503 if the collector is still initializing or the function is disabled
599
+ require_cloud: true
600
+ - id: error-info
601
+ name: Error Info
602
+ description: |
603
+ Retrieves recent SQL errors from Performance Schema statement history tables.
604
+
605
+ This function reads `performance_schema.events_statements_history_long` when enabled,
606
+ otherwise falls back to `performance_schema.events_statements_history`. It reports the
607
+ most recent error per query digest, including error number, SQLSTATE, and message.
608
+
609
+ Use cases:
610
+ - Identify recent query errors and their messages
611
+ - Correlate errors to query patterns (digest)
612
+ - Validate error rates seen in top-queries
613
+
614
+ Error messages are truncated by Performance Schema (usually 128 characters).
615
+ parameters: []
616
+ returns:
617
+ description: Most recent error per query digest from Performance Schema history tables.
618
+ columns:
619
+ - name: Digest
620
+ type: string
621
+ unit: ""
622
+ visibility: hidden
623
+ description: "Unique hash identifier for the normalized query pattern."
624
+ - name: Query
625
+ type: string
626
+ unit: ""
627
+ description: "Normalized query text when available (digest text or SQL text)."
628
+ - name: Schema
629
+ type: string
630
+ unit: ""
631
+ description: "Database schema name when available."
632
+ - name: Error Number
633
+ type: integer
634
+ unit: ""
635
+ description: "MySQL error number for the most recent error of this digest."
636
+ - name: SQL State
637
+ type: string
638
+ unit: ""
639
+ description: "SQLSTATE code for the most recent error."
640
+ - name: Error Message
641
+ type: string
642
+ unit: ""
643
+ description: "Error message for the most recent error."
644
+ performance: |
645
+ Reads Performance Schema statement history tables on demand:<br/>• Not part of regular collection<br/>• Query cost depends on history table size and server load
646
+ security: |
647
+ Error messages and query text may include unmasked literals (PII/secrets).<br/>• Restrict dashboard access to authorized personnel only
648
+ prerequisites:
649
+ list:
650
+ - title: Enable error-info function in Netdata
651
+ description: |
652
+ Set `error_info_function_enabled: true` in the `go.d/mysql.conf` job.
653
+ - title: Enable statement history consumers
654
+ description: |
655
+ Ensure `events_statements_history` and/or `events_statements_history_long` consumers are enabled.
656
+ - title: Grant SELECT on Performance Schema
657
+ description: |
658
+ The monitoring user must have SELECT on `performance_schema.*` to read statement history tables.
659
+ availability: |
660
+ Available when:<br/>• The collector has successfully connected to MySQL<br/>• `error_info_function_enabled` is true<br/>• Performance Schema statement history consumers are enabled (history and/or history_long)<br/>• Returns HTTP 200 with empty data when no errors are found<br/>• Returns HTTP 503 when required consumers are not enabled or function disabled<br/>• Returns HTTP 500 if the query fails<br/>• Returns HTTP 504 if the query times out
661
+ require_cloud: true
662
metrics:
663
folding:
664
title: Metrics
src/go/tools/functions-validation/config/go.d/mssql.conf
+2
-1
@@ -1,4 +1,5 @@
1
jobs:
2
- name: local
3
- dsn: "sqlserver://sa:Netdata123!@127.0.0.1:1433?database=netdata"
3
+ dsn: "sqlserver://sa:Netdata123!@127.0.0.1:1433?database=netdata&encrypt=disable"
4
top_queries_limit: 100
5
+ deadlock_info_function_enabled: true
src/go/tools/functions-validation/config/go.d/mysql.conf
+1
@@ -2,3 +2,4 @@ jobs:
2
- name: local
3
dsn: "netdata:netdata@tcp(127.0.0.1:3306)/netdata"
4
top_queries_limit: 100
5
+ deadlock_info_function_enabled: true
src/go/tools/functions-validation/docker-compose.yml
+7
-3
@@ -17,25 +17,29 @@ services:
17
retries: 10
18
19
mysql:
20
- image: mysql:8.0
20
+ image: ${MYSQL_IMAGE:-mysql:8.0}
21
environment:
22
MYSQL_ROOT_PASSWORD: rootpw
23
MYSQL_DATABASE: netdata
24
MYSQL_USER: netdata
25
MYSQL_PASSWORD: netdata
26
+ MARIADB_ROOT_PASSWORD: rootpw
27
+ MARIADB_DATABASE: netdata
28
+ MARIADB_USER: netdata
29
+ MARIADB_PASSWORD: netdata
30
command: ["--performance_schema=ON"]
31
ports:
32
- "${MYSQL_PORT:-3306}:3306"
33
volumes:
34
- ./seed/mysql/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
35
healthcheck:
32
- test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -u root -p$$MYSQL_ROOT_PASSWORD > /dev/null"]
36
+ test: ["CMD-SHELL", "if command -v mysqladmin >/dev/null 2>&1; then mysqladmin ping -h 127.0.0.1 -u root -p$$MYSQL_ROOT_PASSWORD > /dev/null; elif command -v mariadb-admin >/dev/null 2>&1; then mariadb-admin ping -h 127.0.0.1 -u root -p$$MYSQL_ROOT_PASSWORD > /dev/null; else exit 127; fi"]
37
interval: 5s
38
timeout: 5s
39
retries: 10
40
41
mssql:
38
- image: mcr.microsoft.com/mssql/server:2022-latest
42
+ image: ${MSSQL_IMAGE:-mcr.microsoft.com/mssql/server:2022-latest}
43
environment:
44
ACCEPT_EULA: "Y"
45
MSSQL_SA_PASSWORD: "Netdata123!"
src/go/tools/functions-validation/e2e/lib.sh
+22
@@ -206,6 +206,28 @@ run_info() {
206
run_info_method "$module" "top-queries"
207
}
208
209
+run_function() {
210
+ local module="$1"
211
+ local method="$2"
212
+ local args="${3:-__job:local}"
213
+ local require_rows="${4:-true}"
214
+ local output="$WORKDIR/${module}-${method}.json"
215
+
216
+ run "$WORKDIR/go.d.plugin" \
217
+ --config-dir "$WORKDIR/config" \
218
+ --function "${module}:${method}" \
219
+ --function-args "$args" \
220
+ > "$output"
221
+
222
+ if [ "$require_rows" = "true" ]; then
223
+ validate "$output" --min-rows 1
224
+ else
225
+ validate "$output"
226
+ fi
227
+
228
+ echo "$output"
229
+}
230
+
231
run_top_queries() {
232
local module="$1"
233
local output="$WORKDIR/${module}-top-queries.json"
src/go/tools/functions-validation/e2e/mssql-matrix.sh
new
+16
@@ -0,0 +1,16 @@
1
+#!/usr/bin/env bash
2
+set -euo pipefail
3
+
4
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5
+
6
+MSSQL_VARIANTS=(
7
+ "mcr.microsoft.com/mssql/server:2017-latest|mssql-2017"
8
+ "mcr.microsoft.com/mssql/server:2019-latest|mssql-2019"
9
+ "mcr.microsoft.com/mssql/server:2022-latest|mssql-2022"
10
+)
11
+
12
+for entry in "${MSSQL_VARIANTS[@]}"; do
13
+ IFS='|' read -r image label <<< "$entry"
14
+ printf '\n=== Running MSSQL collector E2E for %s (%s) ===\n' "$label" "$image" >&2
15
+ MSSQL_IMAGE="$image" MSSQL_VARIANT="$label" bash "$SCRIPT_DIR/mssql.sh"
16
+done
src/go/tools/functions-validation/e2e/mssql.sh
+1505
-4
@@ -10,14 +10,1515 @@ trap cleanup EXIT
10
11
MSSQL_PORT="$(reserve_port)"
12
write_env "MSSQL_PORT" "$MSSQL_PORT"
13
-replace_in_file "$WORKDIR/config/go.d/mssql.conf" "127.0.0.1:1433" "127.0.0.1:${MSSQL_PORT}"
13
+MSSQL_CONF="$WORKDIR/config/go.d/mssql.conf"
14
+replace_in_file "$MSSQL_CONF" "127.0.0.1:1433" "127.0.0.1:${MSSQL_PORT}"
15
+
16
+MSSQL_VARIANT_LABEL="${MSSQL_VARIANT:-mssql}"
17
+if [ -n "${MSSQL_IMAGE:-}" ]; then
18
+ write_env "MSSQL_IMAGE" "$MSSQL_IMAGE"
19
+fi
20
21
compose_up mssql
22
wait_healthy mssql 120
23
compose_run mssql-init
24
25
build_plugin
20
-run_info mssql
21
-run_top_queries mssql
26
23
-echo "E2E checks passed for mssql." >&2
27
+MSSQL_JOB_RETRIES="${MSSQL_JOB_RETRIES:-6}"
28
+MSSQL_JOB_RETRY_DELAY="${MSSQL_JOB_RETRY_DELAY:-5}"
29
+
30
+mssql_is_no_jobs_started() {
31
+ local input="$1"
32
+ if [ ! -s "$input" ]; then
33
+ return 1
34
+ fi
35
+
36
+ if command -v python3 >/dev/null 2>&1; then
37
+ python3 - "$input" <<'PY'
38
+import json
39
+import sys
40
+
41
+path = sys.argv[1]
42
+try:
43
+ with open(path, "r", encoding="utf-8") as fh:
44
+ doc = json.load(fh)
45
+except Exception:
46
+ raise SystemExit(1)
47
+
48
+status = doc.get("status")
49
+msg = str(doc.get("errorMessage") or "").lower()
50
+if status == 503 and "no jobs started for module" in msg:
51
+ raise SystemExit(0)
52
+raise SystemExit(1)
53
+PY
54
+ return $?
55
+ fi
56
+
57
+ python - "$input" <<'PY'
58
+import json
59
+import sys
60
+
61
+path = sys.argv[1]
62
+try:
63
+ with open(path, "r") as fh:
64
+ doc = json.load(fh)
65
+except Exception:
66
+ raise SystemExit(1)
67
+
68
+status = doc.get("status")
69
+msg = str(doc.get("errorMessage") or "").lower()
70
+if status == 503 and "no jobs started for module" in msg:
71
+ raise SystemExit(0)
72
+raise SystemExit(1)
73
+PY
74
+}
75
+
76
+run_mssql_info_with_retry() {
77
+ local output="$WORKDIR/mssql-top-queries-info.json"
78
+ local attempt=1
79
+
80
+ while true; do
81
+ if run "$WORKDIR/go.d.plugin" \
82
+ --config-dir "$WORKDIR/config" \
83
+ --function "mssql:top-queries" \
84
+ --function-args info \
85
+ > "$output"; then
86
+ validate "$output"
87
+ return 0
88
+ fi
89
+
90
+ if mssql_is_no_jobs_started "$output"; then
91
+ if [ "$attempt" -ge "$MSSQL_JOB_RETRIES" ]; then
92
+ echo "Timed out waiting for mssql jobs to start (info)" >&2
93
+ return 1
94
+ fi
95
+ attempt=$((attempt + 1))
96
+ sleep "$MSSQL_JOB_RETRY_DELAY"
97
+ continue
98
+ fi
99
+
100
+ echo "Unexpected failure while running mssql top-queries info" >&2
101
+ cat "$output" >&2
102
+ return 1
103
+ done
104
+}
105
+
106
+run_mssql_top_queries_with_retry() {
107
+ local output="$WORKDIR/mssql-top-queries.json"
108
+ local attempt=1
109
+
110
+ while true; do
111
+ if run "$WORKDIR/go.d.plugin" \
112
+ --config-dir "$WORKDIR/config" \
113
+ --function "mssql:top-queries" \
114
+ --function-args __job:local \
115
+ > "$output"; then
116
+ validate "$output" --min-rows 1
117
+ return 0
118
+ fi
119
+
120
+ if mssql_is_no_jobs_started "$output"; then
121
+ if [ "$attempt" -ge "$MSSQL_JOB_RETRIES" ]; then
122
+ echo "Timed out waiting for mssql jobs to start (top-queries)" >&2
123
+ return 1
124
+ fi
125
+ attempt=$((attempt + 1))
126
+ sleep "$MSSQL_JOB_RETRY_DELAY"
127
+ continue
128
+ fi
129
+
130
+ echo "Unexpected failure while running mssql top-queries" >&2
131
+ cat "$output" >&2
132
+ return 1
133
+ done
134
+}
135
+
136
+run_mssql_function_with_retry() {
137
+ local method="$1"
138
+ local args="${2:-__job:local}"
139
+ local require_rows="${3:-true}"
140
+ local output="$WORKDIR/mssql-${method}.json"
141
+ local attempt=1
142
+
143
+ while true; do
144
+ if run "$WORKDIR/go.d.plugin" \
145
+ --config-dir "$WORKDIR/config" \
146
+ --function "mssql:${method}" \
147
+ --function-args "$args" \
148
+ > "$output"; then
149
+ if [ "$require_rows" = "true" ]; then
150
+ validate "$output" --min-rows 1
151
+ else
152
+ validate "$output"
153
+ fi
154
+ echo "$output"
155
+ return 0
156
+ fi
157
+
158
+ if mssql_is_no_jobs_started "$output"; then
159
+ if [ "$attempt" -ge "$MSSQL_JOB_RETRIES" ]; then
160
+ echo "Timed out waiting for mssql jobs to start (${method})" >&2
161
+ return 1
162
+ fi
163
+ attempt=$((attempt + 1))
164
+ sleep "$MSSQL_JOB_RETRY_DELAY"
165
+ continue
166
+ fi
167
+
168
+ echo "Unexpected failure while running mssql ${method}" >&2
169
+ cat "$output" >&2
170
+ return 1
171
+ done
172
+}
173
+
174
+run_mssql_info_with_retry
175
+run_mssql_top_queries_with_retry
176
+
177
+mssql_container_id() {
178
+ "${COMPOSE[@]}" ps -q mssql
179
+}
180
+
181
+mssql_sqlcmd_path() {
182
+ local cid
183
+ cid="$(mssql_container_id)"
184
+ if [ -z "$cid" ]; then
185
+ echo "MSSQL container ID not found" >&2
186
+ return 1
187
+ fi
188
+ docker exec -i "$cid" bash -lc 'if [ -x /opt/mssql-tools18/bin/sqlcmd ]; then echo /opt/mssql-tools18/bin/sqlcmd; elif [ -x /opt/mssql-tools/bin/sqlcmd ]; then echo /opt/mssql-tools/bin/sqlcmd; else exit 1; fi'
189
+}
190
+
191
+MSSQL_SQLCMD_PATH="$(mssql_sqlcmd_path)"
192
+echo "Using sqlcmd path: $MSSQL_SQLCMD_PATH" >&2
193
+
194
+mssql_sqlcmd_supports_c() {
195
+ local cid
196
+ cid="$(mssql_container_id)"
197
+ if [ -z "$cid" ]; then
198
+ return 1
199
+ fi
200
+ set +e
201
+ local help
202
+ help="$(docker exec -i "$cid" "$MSSQL_SQLCMD_PATH" -? 2>&1)"
203
+ local status=$?
204
+ set -e
205
+ if [ $status -ne 0 ] && [ -z "$help" ]; then
206
+ return 1
207
+ fi
208
+ echo "$help" | grep -q " -C"
209
+}
210
+
211
+MSSQL_SQLCMD_CFLAG=()
212
+case "$MSSQL_SQLCMD_PATH" in
213
+ *mssql-tools18*)
214
+ MSSQL_SQLCMD_CFLAG=(-C)
215
+ ;;
216
+ *)
217
+ if mssql_sqlcmd_supports_c; then
218
+ MSSQL_SQLCMD_CFLAG=(-C)
219
+ fi
220
+ ;;
221
+esac
222
+
223
+mssql_exec_sa() {
224
+ local sql="$1"
225
+ local cid
226
+ cid="$(mssql_container_id)"
227
+ if [ -z "$cid" ]; then
228
+ echo "MSSQL container ID not found" >&2
229
+ return 1
230
+ fi
231
+ run docker exec -i "$cid" "$MSSQL_SQLCMD_PATH" "${MSSQL_SQLCMD_CFLAG[@]}" -S localhost -U sa -P "Netdata123!" -d netdata -b -y 0 -Y 0 -Q "$sql"
232
+}
233
+
234
+mssql_exec_sa_allow_error() {
235
+ local sql="$1"
236
+ local cid
237
+ cid="$(mssql_container_id)"
238
+ if [ -z "$cid" ]; then
239
+ echo "MSSQL container ID not found" >&2
240
+ return 1
241
+ fi
242
+ set +e
243
+ docker exec -i "$cid" "$MSSQL_SQLCMD_PATH" "${MSSQL_SQLCMD_CFLAG[@]}" -S localhost -U sa -P "Netdata123!" -d netdata -b -y 0 -Y 0 -Q "$sql" >/dev/null 2>&1
244
+ set -e
245
+}
246
+
247
+induce_deadlock_once() {
248
+ local tx1
249
+ local tx2
250
+
251
+ tx1="$(cat <<'SQL'
252
+SET NOCOUNT ON;
253
+SET LOCK_TIMEOUT 5000;
254
+BEGIN TRAN;
255
+UPDATE dbo.deadlock_a SET value = value + 1 WHERE id = 1;
256
+WAITFOR DELAY '00:00:01';
257
+UPDATE dbo.deadlock_b SET value = value + 1 WHERE id = 1;
258
+COMMIT;
259
+SQL
260
+)"
261
+
262
+ tx2="$(cat <<'SQL'
263
+SET NOCOUNT ON;
264
+SET LOCK_TIMEOUT 5000;
265
+BEGIN TRAN;
266
+UPDATE dbo.deadlock_b SET value = value + 1 WHERE id = 1;
267
+WAITFOR DELAY '00:00:01';
268
+UPDATE dbo.deadlock_a SET value = value + 1 WHERE id = 1;
269
+COMMIT;
270
+SQL
271
+)"
272
+
273
+ mssql_exec_sa "$tx1" &
274
+ local pid1=$!
275
+ mssql_exec_sa "$tx2" &
276
+ local pid2=$!
277
+
278
+ set +e
279
+ wait "$pid1"
280
+ wait "$pid2"
281
+ set -e
282
+}
283
+
284
+assert_deadlock_info_content() {
285
+ local input="$1"
286
+ if command -v python3 >/dev/null 2>&1; then
287
+ python3 - "$input" <<'PY'
288
+import json
289
+import re
290
+import sys
291
+
292
+path = sys.argv[1]
293
+with open(path, "r", encoding="utf-8") as fh:
294
+ doc = json.load(fh)
295
+
296
+try:
297
+ status = int(doc.get("status"))
298
+except (TypeError, ValueError):
299
+ raise SystemExit(f"unexpected status value: {doc.get('status')!r}")
300
+
301
+if status != 200:
302
+ raise SystemExit(f"expected status 200, got {status}")
303
+
304
+if doc.get("errorMessage"):
305
+ raise SystemExit(f"unexpected errorMessage on status 200: {doc.get('errorMessage')!r}")
306
+
307
+columns = doc.get("columns") or {}
308
+field_to_idx = {}
309
+if isinstance(columns, dict):
310
+ for field, col in columns.items():
311
+ if not isinstance(col, dict):
312
+ continue
313
+ try:
314
+ field_to_idx[field] = int(col.get("index"))
315
+ except (TypeError, ValueError):
316
+ continue
317
+else:
318
+ for idx, col in enumerate(columns):
319
+ if not isinstance(col, dict):
320
+ continue
321
+ field = col.get("field")
322
+ if field:
323
+ field_to_idx[field] = idx
324
+
325
+for required in ("row_id", "deadlock_id", "process_id", "is_victim", "lock_mode", "lock_status", "query_text", "wait_resource", "database"):
326
+ if required not in field_to_idx:
327
+ raise SystemExit(f"missing expected column: {required}")
328
+
329
+data = doc.get("data") or []
330
+if not data:
331
+ raise SystemExit("deadlock-info returned no rows")
332
+
333
+def get_value(row, field):
334
+ idx = field_to_idx[field]
335
+ return row[idx] if idx < len(row) else None
336
+
337
+def norm(val):
338
+ return "" if val is None else str(val).strip()
339
+
340
+has_waiting = any(str(get_value(row, "lock_status")).upper() == "WAITING" for row in data)
341
+if not has_waiting:
342
+ raise SystemExit("no WAITING lock_status found in deadlock-info output")
343
+
344
+table_pattern = re.compile(r"deadlock_(a|b)", re.IGNORECASE)
345
+has_expected_query = any(table_pattern.search(str(get_value(row, "query_text"))) for row in data)
346
+if not has_expected_query:
347
+ raise SystemExit("query_text does not reference deadlock tables")
348
+
349
+waiting_rows = [row for row in data if str(get_value(row, "lock_status")).upper() == "WAITING"]
350
+if any(norm(get_value(row, "lock_mode")) == "" for row in waiting_rows):
351
+ raise SystemExit("WAITING rows must include lock_mode")
352
+if any(norm(get_value(row, "wait_resource")) == "" for row in waiting_rows):
353
+ raise SystemExit("WAITING rows must include wait_resource")
354
+
355
+lock_mode_re = re.compile(r"^[A-Za-z0-9_-]+$")
356
+if any(not lock_mode_re.match(norm(get_value(row, "lock_mode"))) for row in waiting_rows):
357
+ raise SystemExit("WAITING rows must include a valid lock_mode")
358
+
359
+victim_counts = {}
360
+expected_db = "netdata"
361
+has_database = False
362
+for row in data:
363
+ deadlock_id = norm(get_value(row, "deadlock_id"))
364
+ if deadlock_id == "":
365
+ raise SystemExit("deadlock_id missing from deadlock-info output")
366
+ process_id = norm(get_value(row, "process_id"))
367
+ if process_id == "":
368
+ raise SystemExit("process_id missing from deadlock-info output")
369
+ row_id = norm(get_value(row, "row_id"))
370
+ if row_id != f"{deadlock_id}:{process_id}":
371
+ raise SystemExit(f"row_id {row_id} does not match deadlock_id/process_id")
372
+ victim_counts.setdefault(deadlock_id, 0)
373
+ if str(get_value(row, "is_victim")).lower() == "true":
374
+ victim_counts[deadlock_id] += 1
375
+ db_val = norm(get_value(row, "database")).lower()
376
+ if db_val:
377
+ has_database = True
378
+ if db_val != expected_db:
379
+ raise SystemExit(f"unexpected database value {db_val!r}, expected {expected_db!r}")
380
+
381
+for deadlock_id, count in victim_counts.items():
382
+ if count != 1:
383
+ raise SystemExit(f"deadlock_id {deadlock_id} has victim count {count}, expected 1")
384
+if not has_database:
385
+ raise SystemExit("expected at least one row with database populated")
386
+PY
387
+ return
388
+ fi
389
+
390
+ python - "$input" <<'PY'
391
+import json
392
+import re
393
+import sys
394
+
395
+path = sys.argv[1]
396
+with open(path, "r") as fh:
397
+ doc = json.load(fh)
398
+
399
+try:
400
+ status = int(doc.get("status"))
401
+except (TypeError, ValueError):
402
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
403
+
404
+if status != 200:
405
+ raise SystemExit("expected status 200, got %s" % status)
406
+
407
+if doc.get("errorMessage"):
408
+ raise SystemExit("unexpected errorMessage on status 200: %r" % (doc.get("errorMessage"),))
409
+
410
+columns = doc.get("columns") or {}
411
+field_to_idx = {}
412
+if isinstance(columns, dict):
413
+ for field, col in columns.items():
414
+ if not isinstance(col, dict):
415
+ continue
416
+ try:
417
+ field_to_idx[field] = int(col.get("index"))
418
+ except (TypeError, ValueError):
419
+ continue
420
+else:
421
+ for idx, col in enumerate(columns):
422
+ if not isinstance(col, dict):
423
+ continue
424
+ field = col.get("field")
425
+ if field:
426
+ field_to_idx[field] = idx
427
+
428
+for required in ("row_id", "deadlock_id", "process_id", "is_victim", "lock_mode", "lock_status", "query_text", "wait_resource", "database"):
429
+ if required not in field_to_idx:
430
+ raise SystemExit("missing expected column: %s" % required)
431
+
432
+data = doc.get("data") or []
433
+if not data:
434
+ raise SystemExit("deadlock-info returned no rows")
435
+
436
+def get_value(row, field):
437
+ idx = field_to_idx[field]
438
+ return row[idx] if idx < len(row) else None
439
+
440
+def norm(val):
441
+ return "" if val is None else str(val).strip()
442
+
443
+has_waiting = any(str(get_value(row, "lock_status")).upper() == "WAITING" for row in data)
444
+if not has_waiting:
445
+ raise SystemExit("no WAITING lock_status found in deadlock-info output")
446
+
447
+table_pattern = re.compile(r"deadlock_(a|b)", re.IGNORECASE)
448
+has_expected_query = any(table_pattern.search(str(get_value(row, "query_text"))) for row in data)
449
+if not has_expected_query:
450
+ raise SystemExit("query_text does not reference deadlock tables")
451
+
452
+waiting_rows = [row for row in data if str(get_value(row, "lock_status")).upper() == "WAITING"]
453
+if any(norm(get_value(row, "lock_mode")) == "" for row in waiting_rows):
454
+ raise SystemExit("WAITING rows must include lock_mode")
455
+if any(norm(get_value(row, "wait_resource")) == "" for row in waiting_rows):
456
+ raise SystemExit("WAITING rows must include wait_resource")
457
+
458
+lock_mode_re = re.compile(r"^[A-Za-z0-9_-]+$")
459
+if any(not lock_mode_re.match(norm(get_value(row, "lock_mode"))) for row in waiting_rows):
460
+ raise SystemExit("WAITING rows must include a valid lock_mode")
461
+
462
+victim_counts = {}
463
+expected_db = "netdata"
464
+has_database = False
465
+for row in data:
466
+ deadlock_id = norm(get_value(row, "deadlock_id"))
467
+ if deadlock_id == "":
468
+ raise SystemExit("deadlock_id missing from deadlock-info output")
469
+ process_id = norm(get_value(row, "process_id"))
470
+ if process_id == "":
471
+ raise SystemExit("process_id missing from deadlock-info output")
472
+ row_id = norm(get_value(row, "row_id"))
473
+ if row_id != "%s:%s" % (deadlock_id, process_id):
474
+ raise SystemExit("row_id %s does not match deadlock_id/process_id" % row_id)
475
+ victim_counts.setdefault(deadlock_id, 0)
476
+ if str(get_value(row, "is_victim")).lower() == "true":
477
+ victim_counts[deadlock_id] += 1
478
+ db_val = norm(get_value(row, "database")).lower()
479
+ if db_val:
480
+ has_database = True
481
+ if db_val != expected_db:
482
+ raise SystemExit("unexpected database value %r, expected %r" % (db_val, expected_db))
483
+
484
+for deadlock_id, count in victim_counts.items():
485
+ if count != 1:
486
+ raise SystemExit("deadlock_id %s has victim count %s, expected 1" % (deadlock_id, count))
487
+if not has_database:
488
+ raise SystemExit("expected at least one row with database populated")
489
+PY
490
+}
491
+
492
+assert_deadlock_info_empty_success() {
493
+ local input="$1"
494
+
495
+ if command -v python3 >/dev/null 2>&1; then
496
+ python3 - "$input" <<'PY'
497
+import json
498
+import sys
499
+
500
+path = sys.argv[1]
501
+with open(path, "r", encoding="utf-8") as fh:
502
+ doc = json.load(fh)
503
+
504
+try:
505
+ status = int(doc.get("status"))
506
+except (TypeError, ValueError):
507
+ raise SystemExit(f"unexpected status value: {doc.get('status')!r}")
508
+
509
+if status != 200:
510
+ raise SystemExit(f"expected status 200, got {status}")
511
+
512
+if doc.get("errorMessage"):
513
+ raise SystemExit(f"unexpected errorMessage on status 200: {doc.get('errorMessage')!r}")
514
+
515
+columns = doc.get("columns") or {}
516
+field_to_idx = {}
517
+if isinstance(columns, dict):
518
+ for field, col in columns.items():
519
+ if not isinstance(col, dict):
520
+ continue
521
+ try:
522
+ field_to_idx[field] = int(col.get("index"))
523
+ except (TypeError, ValueError):
524
+ continue
525
+else:
526
+ for idx, col in enumerate(columns):
527
+ if not isinstance(col, dict):
528
+ continue
529
+ field = col.get("field")
530
+ if field:
531
+ field_to_idx[field] = idx
532
+
533
+data = doc.get("data") or []
534
+if len(data) == 0:
535
+ raise SystemExit(0)
536
+
537
+query_idx = field_to_idx.get("query_text", None)
538
+if query_idx is None:
539
+ raise SystemExit(f"expected no rows, got {len(data)}")
540
+
541
+for row in data:
542
+ if query_idx >= len(row):
543
+ continue
544
+ query = str(row[query_idx]).lower()
545
+ if "deadlock_a" in query or "deadlock_b" in query:
546
+ raise SystemExit(f"unexpected deadlock rows for test tables, got {len(data)} rows")
547
+PY
548
+ return
549
+ fi
550
+
551
+ python - "$input" <<'PY'
552
+import json
553
+import sys
554
+
555
+path = sys.argv[1]
556
+with open(path, "r") as fh:
557
+ doc = json.load(fh)
558
+
559
+try:
560
+ status = int(doc.get("status"))
561
+except (TypeError, ValueError):
562
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
563
+
564
+if status != 200:
565
+ raise SystemExit("expected status 200, got %s" % status)
566
+
567
+if doc.get("errorMessage"):
568
+ raise SystemExit("unexpected errorMessage on status 200: %r" % (doc.get("errorMessage"),))
569
+
570
+columns = doc.get("columns") or {}
571
+field_to_idx = {}
572
+if isinstance(columns, dict):
573
+ for field, col in columns.items():
574
+ if not isinstance(col, dict):
575
+ continue
576
+ try:
577
+ field_to_idx[field] = int(col.get("index"))
578
+ except (TypeError, ValueError):
579
+ continue
580
+else:
581
+ for idx, col in enumerate(columns):
582
+ if not isinstance(col, dict):
583
+ continue
584
+ field = col.get("field")
585
+ if field:
586
+ field_to_idx[field] = idx
587
+
588
+data = doc.get("data") or []
589
+if len(data) == 0:
590
+ raise SystemExit(0)
591
+
592
+query_idx = field_to_idx.get("query_text", None)
593
+if query_idx is None:
594
+ raise SystemExit("expected no rows, got %s" % len(data))
595
+
596
+for row in data:
597
+ if query_idx >= len(row):
598
+ continue
599
+ query = str(row[query_idx]).lower()
600
+ if "deadlock_a" in query or "deadlock_b" in query:
601
+ raise SystemExit("unexpected deadlock rows for test tables, got %s rows" % len(data))
602
+PY
603
+}
604
+
605
+assert_deadlock_info_error_contains() {
606
+ local input="$1"
607
+ local expected_status="$2"
608
+ local expected_substr="$3"
609
+
610
+ if command -v python3 >/dev/null 2>&1; then
611
+ python3 - "$input" "$expected_status" "$expected_substr" <<'PY'
612
+import json
613
+import sys
614
+
615
+path = sys.argv[1]
616
+expected_status = int(sys.argv[2])
617
+expected = sys.argv[3].strip().lower()
618
+with open(path, "r", encoding="utf-8") as fh:
619
+ doc = json.load(fh)
620
+
621
+try:
622
+ status = int(doc.get("status"))
623
+except (TypeError, ValueError):
624
+ raise SystemExit(f"unexpected status value: {doc.get('status')!r}")
625
+
626
+if status != expected_status:
627
+ raise SystemExit(f"expected status {expected_status}, got {status}")
628
+
629
+err = str(doc.get("errorMessage") or "").lower()
630
+if expected not in err:
631
+ raise SystemExit(f"expected errorMessage to contain {expected!r}, got {err!r}")
632
+PY
633
+ return
634
+ fi
635
+
636
+ python - "$input" "$expected_status" "$expected_substr" <<'PY'
637
+import json
638
+import sys
639
+
640
+path = sys.argv[1]
641
+expected_status = int(sys.argv[2])
642
+expected = sys.argv[3].strip().lower()
643
+with open(path, "r") as fh:
644
+ doc = json.load(fh)
645
+
646
+try:
647
+ status = int(doc.get("status"))
648
+except (TypeError, ValueError):
649
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
650
+
651
+if status != expected_status:
652
+ raise SystemExit("expected status %s, got %s" % (expected_status, status))
653
+
654
+err = str(doc.get("errorMessage") or "").lower()
655
+if expected not in err:
656
+ raise SystemExit("expected errorMessage to contain %r, got %r" % (expected, err))
657
+PY
658
+}
659
+
660
+assert_error_info_not_enabled() {
661
+ local input="$1"
662
+
663
+ if command -v python3 >/dev/null 2>&1; then
664
+ python3 - "$input" <<'PY'
665
+import json
666
+import sys
667
+
668
+path = sys.argv[1]
669
+with open(path, "r", encoding="utf-8") as fh:
670
+ doc = json.load(fh)
671
+
672
+try:
673
+ status = int(doc.get("status"))
674
+except (TypeError, ValueError):
675
+ raise SystemExit(f"unexpected status value: {doc.get('status')!r}")
676
+
677
+if status < 400:
678
+ raise SystemExit(f"expected error status, got {status}")
679
+
680
+err = str(doc.get("errorMessage") or "").lower()
681
+if "not enabled" not in err:
682
+ raise SystemExit(f"expected errorMessage to contain 'not enabled', got {err!r}")
683
+PY
684
+ return
685
+ fi
686
+
687
+ python - "$input" <<'PY'
688
+import json
689
+import sys
690
+
691
+path = sys.argv[1]
692
+with open(path, "r") as fh:
693
+ doc = json.load(fh)
694
+
695
+try:
696
+ status = int(doc.get("status"))
697
+except (TypeError, ValueError):
698
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
699
+
700
+if status < 400:
701
+ raise SystemExit("expected error status, got %s" % status)
702
+
703
+err = str(doc.get("errorMessage") or "").lower()
704
+if "not enabled" not in err:
705
+ raise SystemExit("expected errorMessage to contain 'not enabled', got %r" % err)
706
+PY
707
+}
708
+
709
+assert_error_info_has_errors() {
710
+ local input="$1"
711
+
712
+ if command -v python3 >/dev/null 2>&1; then
713
+ python3 - "$input" <<'PY'
714
+import json
715
+import sys
716
+
717
+path = sys.argv[1]
718
+with open(path, "r", encoding="utf-8") as fh:
719
+ doc = json.load(fh)
720
+
721
+try:
722
+ status = int(doc.get("status"))
723
+except (TypeError, ValueError):
724
+ raise SystemExit(f"unexpected status value: {doc.get('status')!r}")
725
+
726
+if status != 200:
727
+ raise SystemExit(f"expected status 200, got {status}")
728
+
729
+if doc.get("errorMessage"):
730
+ raise SystemExit(f"unexpected errorMessage on status 200: {doc.get('errorMessage')!r}")
731
+
732
+columns = doc.get("columns") or {}
733
+field_to_idx = {}
734
+if isinstance(columns, dict):
735
+ for field, col in columns.items():
736
+ if not isinstance(col, dict):
737
+ continue
738
+ try:
739
+ field_to_idx[field] = int(col.get("index"))
740
+ except (TypeError, ValueError):
741
+ continue
742
+else:
743
+ for idx, col in enumerate(columns):
744
+ if not isinstance(col, dict):
745
+ continue
746
+ field = col.get("field")
747
+ if field:
748
+ field_to_idx[field] = idx
749
+
750
+for required in ("errorNumber", "errorMessage", "query"):
751
+ if required not in field_to_idx:
752
+ raise SystemExit(f"missing expected column: {required}")
753
+
754
+data = doc.get("data") or []
755
+if not data:
756
+ raise SystemExit("error-info returned no rows")
757
+
758
+num_idx = field_to_idx["errorNumber"]
759
+msg_idx = field_to_idx["errorMessage"]
760
+query_idx = field_to_idx["query"]
761
+
762
+# Error categories to verify:
763
+# 208 - Invalid object name (table not found)
764
+# 102 - Syntax error
765
+# 2627 - Duplicate key / unique constraint violation
766
+# 245 - Data type conversion error
767
+# 8134 - Division by zero
768
+error_categories = {
769
+ "table_not_found": {"patterns": ["invalid object name", "netdata_error_map_e2e"], "found": False},
770
+ "syntax_error": {"patterns": ["incorrect syntax", "form"], "found": False},
771
+ "duplicate_key": {"patterns": ["duplicate key", "unique", "primary key", "error_test"], "found": False},
772
+ "data_type": {"patterns": ["conversion failed", "converting"], "found": False},
773
+ "divide_by_zero": {"patterns": ["divide by zero"], "found": False},
774
+}
775
+
776
+for row in data:
777
+ if num_idx >= len(row) or row[num_idx] is None:
778
+ continue
779
+ msg = str(row[msg_idx]).lower() if msg_idx < len(row) else ""
780
+ query = str(row[query_idx]).lower() if query_idx < len(row) else ""
781
+ combined = msg + " " + query
782
+ for cat, info in error_categories.items():
783
+ if info["found"]:
784
+ continue
785
+ for pattern in info["patterns"]:
786
+ if pattern in combined:
787
+ info["found"] = True
788
+ break
789
+
790
+missing = [cat for cat, info in error_categories.items() if not info["found"]]
791
+if missing:
792
+ raise SystemExit(f"error-info missing error categories: {', '.join(missing)}")
793
+PY
794
+ return
795
+ fi
796
+
797
+ python - "$input" <<'PY'
798
+import json
799
+import sys
800
+
801
+path = sys.argv[1]
802
+with open(path, "r") as fh:
803
+ doc = json.load(fh)
804
+
805
+try:
806
+ status = int(doc.get("status"))
807
+except (TypeError, ValueError):
808
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
809
+
810
+if status != 200:
811
+ raise SystemExit("expected status 200, got %s" % status)
812
+
813
+if doc.get("errorMessage"):
814
+ raise SystemExit("unexpected errorMessage on status 200: %r" % (doc.get("errorMessage"),))
815
+
816
+columns = doc.get("columns") or {}
817
+field_to_idx = {}
818
+if isinstance(columns, dict):
819
+ for field, col in columns.items():
820
+ if not isinstance(col, dict):
821
+ continue
822
+ try:
823
+ field_to_idx[field] = int(col.get("index"))
824
+ except (TypeError, ValueError):
825
+ continue
826
+else:
827
+ for idx, col in enumerate(columns):
828
+ if not isinstance(col, dict):
829
+ continue
830
+ field = col.get("field")
831
+ if field:
832
+ field_to_idx[field] = idx
833
+
834
+for required in ("errorNumber", "errorMessage", "query"):
835
+ if required not in field_to_idx:
836
+ raise SystemExit("missing expected column: %s" % required)
837
+
838
+data = doc.get("data") or []
839
+if not data:
840
+ raise SystemExit("error-info returned no rows")
841
+
842
+num_idx = field_to_idx["errorNumber"]
843
+msg_idx = field_to_idx["errorMessage"]
844
+query_idx = field_to_idx["query"]
845
+
846
+# Error categories to verify:
847
+# 208 - Invalid object name (table not found)
848
+# 102 - Syntax error
849
+# 2627 - Duplicate key / unique constraint violation
850
+# 245 - Data type conversion error
851
+# 8134 - Division by zero
852
+error_categories = {
853
+ "table_not_found": {"patterns": ["invalid object name", "netdata_error_map_e2e"], "found": False},
854
+ "syntax_error": {"patterns": ["incorrect syntax", "form"], "found": False},
855
+ "duplicate_key": {"patterns": ["duplicate key", "unique", "primary key", "error_test"], "found": False},
856
+ "data_type": {"patterns": ["conversion failed", "converting"], "found": False},
857
+ "divide_by_zero": {"patterns": ["divide by zero"], "found": False},
858
+}
859
+
860
+for row in data:
861
+ if num_idx >= len(row) or row[num_idx] is None:
862
+ continue
863
+ msg = str(row[msg_idx]).lower() if msg_idx < len(row) else ""
864
+ query = str(row[query_idx]).lower() if query_idx < len(row) else ""
865
+ combined = msg + " " + query
866
+ for cat, info in error_categories.items():
867
+ if info["found"]:
868
+ continue
869
+ for pattern in info["patterns"]:
870
+ if pattern in combined:
871
+ info["found"] = True
872
+ break
873
+
874
+missing = [cat for cat, info in error_categories.items() if not info["found"]]
875
+if missing:
876
+ raise SystemExit("error-info missing error categories: %s" % ", ".join(missing))
877
+PY
878
+}
879
+
880
+assert_top_queries_error_attribution_not_enabled() {
881
+ local input="$1"
882
+
883
+ if command -v python3 >/dev/null 2>&1; then
884
+ python3 - "$input" <<'PY'
885
+import json
886
+import sys
887
+
888
+path = sys.argv[1]
889
+with open(path, "r", encoding="utf-8") as fh:
890
+ doc = json.load(fh)
891
+
892
+columns = doc.get("columns") or {}
893
+field_to_idx = {}
894
+if isinstance(columns, dict):
895
+ for field, col in columns.items():
896
+ if not isinstance(col, dict):
897
+ continue
898
+ try:
899
+ field_to_idx[field] = int(col.get("index"))
900
+ except (TypeError, ValueError):
901
+ continue
902
+else:
903
+ for idx, col in enumerate(columns):
904
+ if not isinstance(col, dict):
905
+ continue
906
+ field = col.get("field")
907
+ if field:
908
+ field_to_idx[field] = idx
909
+
910
+if "errorAttribution" not in field_to_idx:
911
+ raise SystemExit("missing expected column: errorAttribution")
912
+
913
+data = doc.get("data") or []
914
+idx = field_to_idx["errorAttribution"]
915
+for row in data:
916
+ if idx >= len(row):
917
+ continue
918
+ if str(row[idx]) != "not_enabled":
919
+ raise SystemExit(f"expected errorAttribution 'not_enabled', got {row[idx]!r}")
920
+PY
921
+ return
922
+ fi
923
+
924
+ python - "$input" <<'PY'
925
+import json
926
+import sys
927
+
928
+path = sys.argv[1]
929
+with open(path, "r") as fh:
930
+ doc = json.load(fh)
931
+
932
+columns = doc.get("columns") or {}
933
+field_to_idx = {}
934
+if isinstance(columns, dict):
935
+ for field, col in columns.items():
936
+ if not isinstance(col, dict):
937
+ continue
938
+ try:
939
+ field_to_idx[field] = int(col.get("index"))
940
+ except (TypeError, ValueError):
941
+ continue
942
+else:
943
+ for idx, col in enumerate(columns):
944
+ if not isinstance(col, dict):
945
+ continue
946
+ field = col.get("field")
947
+ if field:
948
+ field_to_idx[field] = idx
949
+
950
+if "errorAttribution" not in field_to_idx:
951
+ raise SystemExit("missing expected column: errorAttribution")
952
+
953
+data = doc.get("data") or []
954
+idx = field_to_idx["errorAttribution"]
955
+for row in data:
956
+ if idx >= len(row):
957
+ continue
958
+ if str(row[idx]) != "not_enabled":
959
+ raise SystemExit("expected errorAttribution 'not_enabled', got %r" % row[idx])
960
+PY
961
+}
962
+
963
+assert_top_queries_error_attribution_active() {
964
+ local input="$1"
965
+
966
+ if command -v python3 >/dev/null 2>&1; then
967
+ python3 - "$input" <<'PY'
968
+import json
969
+import sys
970
+
971
+path = sys.argv[1]
972
+with open(path, "r", encoding="utf-8") as fh:
973
+ doc = json.load(fh)
974
+
975
+columns = doc.get("columns") or {}
976
+field_to_idx = {}
977
+if isinstance(columns, dict):
978
+ for field, col in columns.items():
979
+ if not isinstance(col, dict):
980
+ continue
981
+ try:
982
+ field_to_idx[field] = int(col.get("index"))
983
+ except (TypeError, ValueError):
984
+ continue
985
+else:
986
+ for idx, col in enumerate(columns):
987
+ if not isinstance(col, dict):
988
+ continue
989
+ field = col.get("field")
990
+ if field:
991
+ field_to_idx[field] = idx
992
+
993
+for required in ("errorAttribution",):
994
+ if required not in field_to_idx:
995
+ raise SystemExit(f"missing expected column: {required}")
996
+
997
+data = doc.get("data") or []
998
+status_idx = field_to_idx["errorAttribution"]
999
+
1000
+for row in data:
1001
+ if status_idx >= len(row):
1002
+ continue
1003
+ status = str(row[status_idx])
1004
+ if status not in ("enabled", "no_data"):
1005
+ raise SystemExit(f"unexpected errorAttribution status {status!r}")
1006
+PY
1007
+ return
1008
+ fi
1009
+
1010
+ python - "$input" <<'PY'
1011
+import json
1012
+import sys
1013
+
1014
+path = sys.argv[1]
1015
+with open(path, "r") as fh:
1016
+ doc = json.load(fh)
1017
+
1018
+columns = doc.get("columns") or {}
1019
+field_to_idx = {}
1020
+if isinstance(columns, dict):
1021
+ for field, col in columns.items():
1022
+ if not isinstance(col, dict):
1023
+ continue
1024
+ try:
1025
+ field_to_idx[field] = int(col.get("index"))
1026
+ except (TypeError, ValueError):
1027
+ continue
1028
+else:
1029
+ for idx, col in enumerate(columns):
1030
+ if not isinstance(col, dict):
1031
+ continue
1032
+ field = col.get("field")
1033
+ if field:
1034
+ field_to_idx[field] = idx
1035
+
1036
+for required in ("errorAttribution",):
1037
+ if required not in field_to_idx:
1038
+ raise SystemExit("missing expected column: %s" % required)
1039
+
1040
+data = doc.get("data") or []
1041
+status_idx = field_to_idx["errorAttribution"]
1042
+
1043
+for row in data:
1044
+ if status_idx >= len(row):
1045
+ continue
1046
+ status = str(row[status_idx])
1047
+ if status not in ("enabled", "no_data"):
1048
+ raise SystemExit("unexpected errorAttribution status %r" % status)
1049
+PY
1050
+}
1051
+
1052
+assert_top_queries_error_attribution_mapped() {
1053
+ local top_queries="$1"
1054
+ local error_info="$2"
1055
+
1056
+ if command -v python3 >/dev/null 2>&1; then
1057
+ python3 - "$top_queries" "$error_info" <<'PY'
1058
+import json
1059
+import sys
1060
+
1061
+top_path = sys.argv[1]
1062
+err_path = sys.argv[2]
1063
+
1064
+with open(err_path, "r", encoding="utf-8") as fh:
1065
+ err_doc = json.load(fh)
1066
+
1067
+err_cols = err_doc.get("columns") or {}
1068
+err_idx = {}
1069
+if isinstance(err_cols, dict):
1070
+ for field, col in err_cols.items():
1071
+ if not isinstance(col, dict):
1072
+ continue
1073
+ try:
1074
+ err_idx[field] = int(col.get("index"))
1075
+ except (TypeError, ValueError):
1076
+ continue
1077
+else:
1078
+ for idx, col in enumerate(err_cols):
1079
+ if not isinstance(col, dict):
1080
+ continue
1081
+ field = col.get("field")
1082
+ if field:
1083
+ err_idx[field] = idx
1084
+
1085
+for required in ("errorMessage", "errorNumber", "query", "queryHash"):
1086
+ if required not in err_idx:
1087
+ raise SystemExit(f"missing expected error-info column: {required}")
1088
+
1089
+def normalize(text: str) -> str:
1090
+ return " ".join(text.split()).strip().rstrip(";").strip()
1091
+
1092
+error_rows = err_doc.get("data") or []
1093
+candidates = []
1094
+for row in error_rows:
1095
+ msg = str(row[err_idx["errorMessage"]]).lower() if err_idx["errorMessage"] < len(row) else ""
1096
+ err_no = row[err_idx["errorNumber"]] if err_idx["errorNumber"] < len(row) else None
1097
+ query = str(row[err_idx["query"]]).lower() if err_idx["query"] < len(row) else ""
1098
+ qh = row[err_idx["queryHash"]] if err_idx["queryHash"] < len(row) else None
1099
+ try:
1100
+ err_no_val = int(err_no)
1101
+ except Exception:
1102
+ continue
1103
+ if err_no_val != 208:
1104
+ continue
1105
+ if "invalid object name" in msg and "netdata_error_map_e2e" in query:
1106
+ candidates.append((str(qh) if qh else "", normalize(query)))
1107
+
1108
+if not candidates:
1109
+ raise SystemExit("no error-info row contained invalid object name for netdata_error_map_e2e")
1110
+
1111
+with open(top_path, "r", encoding="utf-8") as fh:
1112
+ doc = json.load(fh)
1113
+
1114
+columns = doc.get("columns") or {}
1115
+field_to_idx = {}
1116
+if isinstance(columns, dict):
1117
+ for field, col in columns.items():
1118
+ if not isinstance(col, dict):
1119
+ continue
1120
+ try:
1121
+ field_to_idx[field] = int(col.get("index"))
1122
+ except (TypeError, ValueError):
1123
+ continue
1124
+else:
1125
+ for idx, col in enumerate(columns):
1126
+ if not isinstance(col, dict):
1127
+ continue
1128
+ field = col.get("field")
1129
+ if field:
1130
+ field_to_idx[field] = idx
1131
+
1132
+for required in ("query", "queryHash", "errorAttribution", "errorNumber", "errorMessage"):
1133
+ if required not in field_to_idx:
1134
+ raise SystemExit(f"missing expected column: {required}")
1135
+
1136
+data = doc.get("data") or []
1137
+status_idx = field_to_idx["errorAttribution"]
1138
+num_idx = field_to_idx["errorNumber"]
1139
+msg_idx = field_to_idx["errorMessage"]
1140
+hash_idx = field_to_idx["queryHash"]
1141
+
1142
+matched = False
1143
+for row in data:
1144
+ if status_idx >= len(row):
1145
+ continue
1146
+ if hash_idx >= len(row):
1147
+ continue
1148
+ status = str(row[status_idx]) if status_idx < len(row) else ""
1149
+ if status != "enabled":
1150
+ continue
1151
+ err_no = row[num_idx] if num_idx < len(row) else None
1152
+ try:
1153
+ err_no_val = int(err_no)
1154
+ except Exception:
1155
+ continue
1156
+ if err_no_val != 208:
1157
+ continue
1158
+ msg = str(row[msg_idx]).lower() if msg_idx < len(row) and row[msg_idx] is not None else ""
1159
+ if "invalid object name" not in msg:
1160
+ continue
1161
+ row_hash = str(row[hash_idx]) if hash_idx < len(row) and row[hash_idx] is not None else ""
1162
+ row_query = normalize(str(row[field_to_idx["query"]]).lower()) if field_to_idx["query"] < len(row) else ""
1163
+ for cand_hash, cand_query in candidates:
1164
+ if cand_hash and row_hash == cand_hash:
1165
+ matched = True
1166
+ break
1167
+ if cand_query and row_query == cand_query:
1168
+ matched = True
1169
+ break
1170
+ if matched:
1171
+ break
1172
+
1173
+if not matched:
1174
+ raise SystemExit("no top-queries row had enabled error attribution for netdata_error_map_e2e")
1175
+PY
1176
+ return
1177
+ fi
1178
+
1179
+ python - "$top_queries" "$error_info" <<'PY'
1180
+import json
1181
+import sys
1182
+
1183
+top_path = sys.argv[1]
1184
+err_path = sys.argv[2]
1185
+
1186
+with open(err_path, "r") as fh:
1187
+ err_doc = json.load(fh)
1188
+
1189
+err_cols = err_doc.get("columns") or {}
1190
+err_idx = {}
1191
+if isinstance(err_cols, dict):
1192
+ for field, col in err_cols.items():
1193
+ if not isinstance(col, dict):
1194
+ continue
1195
+ try:
1196
+ err_idx[field] = int(col.get("index"))
1197
+ except (TypeError, ValueError):
1198
+ continue
1199
+else:
1200
+ for idx, col in enumerate(err_cols):
1201
+ if not isinstance(col, dict):
1202
+ continue
1203
+ field = col.get("field")
1204
+ if field:
1205
+ err_idx[field] = idx
1206
+
1207
+for required in ("errorMessage", "errorNumber", "query", "queryHash"):
1208
+ if required not in err_idx:
1209
+ raise SystemExit("missing expected error-info column: %s" % required)
1210
+
1211
+def normalize(text):
1212
+ return " ".join(text.split()).strip().rstrip(";").strip()
1213
+
1214
+error_rows = err_doc.get("data") or []
1215
+candidates = []
1216
+for row in error_rows:
1217
+ msg = str(row[err_idx["errorMessage"]]).lower() if err_idx["errorMessage"] < len(row) else ""
1218
+ err_no = row[err_idx["errorNumber"]] if err_idx["errorNumber"] < len(row) else None
1219
+ query = str(row[err_idx["query"]]).lower() if err_idx["query"] < len(row) else ""
1220
+ qh = row[err_idx["queryHash"]] if err_idx["queryHash"] < len(row) else None
1221
+ try:
1222
+ err_no_val = int(err_no)
1223
+ except Exception:
1224
+ continue
1225
+ if err_no_val != 208:
1226
+ continue
1227
+ if "invalid object name" in msg and "netdata_error_map_e2e" in query:
1228
+ candidates.append((str(qh) if qh else "", normalize(query)))
1229
+
1230
+if not candidates:
1231
+ raise SystemExit("no error-info row contained invalid object name for netdata_error_map_e2e")
1232
+
1233
+with open(top_path, "r") as fh:
1234
+ doc = json.load(fh)
1235
+
1236
+columns = doc.get("columns") or {}
1237
+field_to_idx = {}
1238
+if isinstance(columns, dict):
1239
+ for field, col in columns.items():
1240
+ if not isinstance(col, dict):
1241
+ continue
1242
+ try:
1243
+ field_to_idx[field] = int(col.get("index"))
1244
+ except (TypeError, ValueError):
1245
+ continue
1246
+else:
1247
+ for idx, col in enumerate(columns):
1248
+ if not isinstance(col, dict):
1249
+ continue
1250
+ field = col.get("field")
1251
+ if field:
1252
+ field_to_idx[field] = idx
1253
+
1254
+for required in ("query", "queryHash", "errorAttribution", "errorNumber", "errorMessage"):
1255
+ if required not in field_to_idx:
1256
+ raise SystemExit("missing expected column: %s" % required)
1257
+
1258
+data = doc.get("data") or []
1259
+status_idx = field_to_idx["errorAttribution"]
1260
+num_idx = field_to_idx["errorNumber"]
1261
+msg_idx = field_to_idx["errorMessage"]
1262
+hash_idx = field_to_idx["queryHash"]
1263
+
1264
+matched = False
1265
+for row in data:
1266
+ if status_idx >= len(row):
1267
+ continue
1268
+ if hash_idx >= len(row):
1269
+ continue
1270
+ status = str(row[status_idx]) if status_idx < len(row) else ""
1271
+ if status != "enabled":
1272
+ continue
1273
+ err_no = row[num_idx] if num_idx < len(row) else None
1274
+ try:
1275
+ err_no_val = int(err_no)
1276
+ except Exception:
1277
+ continue
1278
+ if err_no_val != 208:
1279
+ continue
1280
+ msg = str(row[msg_idx]).lower() if msg_idx < len(row) and row[msg_idx] is not None else ""
1281
+ if "invalid object name" not in msg:
1282
+ continue
1283
+ row_hash = str(row[hash_idx]) if hash_idx < len(row) and row[hash_idx] is not None else ""
1284
+ row_query = normalize(str(row[field_to_idx["query"]]).lower()) if field_to_idx["query"] < len(row) else ""
1285
+ for cand_hash, cand_query in candidates:
1286
+ if cand_hash and row_hash == cand_hash:
1287
+ matched = True
1288
+ break
1289
+ if cand_query and row_query == cand_query:
1290
+ matched = True
1291
+ break
1292
+ if matched:
1293
+ break
1294
+
1295
+if not matched:
1296
+ raise SystemExit("no top-queries row had enabled error attribution for netdata_error_map_e2e")
1297
+PY
1298
+}
1299
+
1300
+assert_top_queries_plan_ops() {
1301
+ local input="$1"
1302
+
1303
+ if command -v python3 >/dev/null 2>&1; then
1304
+ python3 - "$input" <<'PY'
1305
+import json
1306
+import sys
1307
+
1308
+path = sys.argv[1]
1309
+with open(path, "r", encoding="utf-8") as fh:
1310
+ doc = json.load(fh)
1311
+
1312
+columns = doc.get("columns") or {}
1313
+field_to_idx = {}
1314
+if isinstance(columns, dict):
1315
+ for field, col in columns.items():
1316
+ if not isinstance(col, dict):
1317
+ continue
1318
+ try:
1319
+ field_to_idx[field] = int(col.get("index"))
1320
+ except (TypeError, ValueError):
1321
+ continue
1322
+else:
1323
+ for idx, col in enumerate(columns):
1324
+ if not isinstance(col, dict):
1325
+ continue
1326
+ field = col.get("field")
1327
+ if field:
1328
+ field_to_idx[field] = idx
1329
+
1330
+for required in ("query", "hashMatch", "sorts"):
1331
+ if required not in field_to_idx:
1332
+ raise SystemExit(f"missing expected column: {required}")
1333
+
1334
+data = doc.get("data") or []
1335
+query_idx = field_to_idx["query"]
1336
+hash_idx = field_to_idx["hashMatch"]
1337
+sort_idx = field_to_idx["sorts"]
1338
+
1339
+matched = False
1340
+for row in data:
1341
+ if query_idx >= len(row):
1342
+ continue
1343
+ query = str(row[query_idx]).lower()
1344
+ if "join" not in query or "sample" not in query:
1345
+ continue
1346
+ hash_val = row[hash_idx] if hash_idx < len(row) else 0
1347
+ sort_val = row[sort_idx] if sort_idx < len(row) else 0
1348
+ try:
1349
+ hash_val = int(hash_val)
1350
+ except Exception:
1351
+ hash_val = 0
1352
+ try:
1353
+ sort_val = int(sort_val)
1354
+ except Exception:
1355
+ sort_val = 0
1356
+ if hash_val > 0 and sort_val > 0:
1357
+ matched = True
1358
+ break
1359
+
1360
+if not matched:
1361
+ raise SystemExit("no top-queries row had hashMatch and sorts counts for the join query")
1362
+PY
1363
+ return
1364
+ fi
1365
+
1366
+ python - "$input" <<'PY'
1367
+import json
1368
+import sys
1369
+
1370
+path = sys.argv[1]
1371
+with open(path, "r") as fh:
1372
+ doc = json.load(fh)
1373
+
1374
+columns = doc.get("columns") or {}
1375
+field_to_idx = {}
1376
+if isinstance(columns, dict):
1377
+ for field, col in columns.items():
1378
+ if not isinstance(col, dict):
1379
+ continue
1380
+ try:
1381
+ field_to_idx[field] = int(col.get("index"))
1382
+ except (TypeError, ValueError):
1383
+ continue
1384
+else:
1385
+ for idx, col in enumerate(columns):
1386
+ if not isinstance(col, dict):
1387
+ continue
1388
+ field = col.get("field")
1389
+ if field:
1390
+ field_to_idx[field] = idx
1391
+
1392
+for required in ("query", "hashMatch", "sorts"):
1393
+ if required not in field_to_idx:
1394
+ raise SystemExit("missing expected column: %s" % required)
1395
+
1396
+data = doc.get("data") or []
1397
+query_idx = field_to_idx["query"]
1398
+hash_idx = field_to_idx["hashMatch"]
1399
+sort_idx = field_to_idx["sorts"]
1400
+
1401
+matched = False
1402
+for row in data:
1403
+ if query_idx >= len(row):
1404
+ continue
1405
+ query = str(row[query_idx]).lower()
1406
+ if "join" not in query or "sample" not in query:
1407
+ continue
1408
+ hash_val = row[hash_idx] if hash_idx < len(row) else 0
1409
+ sort_val = row[sort_idx] if sort_idx < len(row) else 0
1410
+ try:
1411
+ hash_val = int(hash_val)
1412
+ except Exception:
1413
+ hash_val = 0
1414
+ try:
1415
+ sort_val = int(sort_val)
1416
+ except Exception:
1417
+ sort_val = 0
1418
+ if hash_val > 0 and sort_val > 0:
1419
+ matched = True
1420
+ break
1421
+
1422
+if not matched:
1423
+ raise SystemExit("no top-queries row had hashMatch and sorts counts for the join query")
1424
+PY
1425
+}
1426
+
1427
+verify_deadlock_info_no_deadlock() {
1428
+ local output
1429
+
1430
+ output="$(run_mssql_function_with_retry deadlock-info '__job:local' 'false')"
1431
+ validate "$output"
1432
+ assert_deadlock_info_empty_success "$output"
1433
+}
1434
+
1435
+verify_deadlock_info() {
1436
+ local attempt
1437
+ local output
1438
+ local found="false"
1439
+
1440
+ for attempt in 1 2 3 4 5; do
1441
+ induce_deadlock_once
1442
+ output="$(run_mssql_function_with_retry deadlock-info '__job:local' 'false')"
1443
+ if has_min_rows "$output" 1; then
1444
+ validate "$output" --min-rows 1
1445
+ if assert_deadlock_info_content "$output"; then
1446
+ found="true"
1447
+ break
1448
+ fi
1449
+ fi
1450
+ sleep 1
1451
+ done
1452
+
1453
+ if [ "$found" != "true" ]; then
1454
+ echo "deadlock-info did not produce valid deadlock attribution after 5 attempts" >&2
1455
+ return 1
1456
+ fi
1457
+}
1458
+
1459
+verify_deadlock_info_no_deadlock
1460
+verify_deadlock_info
1461
+
1462
+assert_top_queries_error_attribution_not_enabled "$WORKDIR/mssql-top-queries.json"
1463
+
1464
+error_output="$(run_mssql_function_with_retry error-info '__job:local' 'false')"
1465
+assert_error_info_not_enabled "$error_output"
1466
+
1467
+mssql_exec_sa "IF EXISTS (SELECT 1 FROM sys.server_event_sessions WHERE name = 'netdata_errors') DROP EVENT SESSION [netdata_errors] ON SERVER;"
1468
+mssql_exec_sa "CREATE EVENT SESSION [netdata_errors] ON SERVER ADD EVENT sqlserver.error_reported(ACTION(sqlserver.sql_text, sqlserver.query_hash)) ADD TARGET package0.ring_buffer;"
1469
+mssql_exec_sa "ALTER EVENT SESSION [netdata_errors] ON SERVER STATE = START;"
1470
+mssql_exec_sa "ALTER DATABASE netdata SET QUERY_STORE (QUERY_CAPTURE_MODE = ALL, OPERATION_MODE = READ_WRITE);"
1471
+
1472
+mssql_exec_sa "IF OBJECT_ID('dbo.netdata_error_map_e2e', 'U') IS NOT NULL DROP TABLE dbo.netdata_error_map_e2e;"
1473
+mssql_exec_sa "CREATE TABLE dbo.netdata_error_map_e2e (id int NOT NULL PRIMARY KEY);"
1474
+mssql_exec_sa "INSERT INTO dbo.netdata_error_map_e2e (id) VALUES (1), (2), (3);"
1475
+
1476
+for _ in 1 2 3 4 5 6 7 8 9 10; do
1477
+ mssql_exec_sa "SELECT COUNT(*) FROM dbo.netdata_error_map_e2e;"
1478
+done
1479
+
1480
+mssql_exec_sa "DROP TABLE dbo.netdata_error_map_e2e;"
1481
+
1482
+# Generate errors for multiple categories:
1483
+# 1. Table not found (error 208)
1484
+for _ in 1 2 3; do
1485
+ mssql_exec_sa_allow_error "SELECT COUNT(*) FROM dbo.netdata_error_map_e2e;"
1486
+done
1487
+
1488
+# 2. Syntax error (error 102)
1489
+for _ in 1 2 3; do
1490
+ mssql_exec_sa_allow_error "SELECT * FORM dbo.sample;"
1491
+done
1492
+
1493
+# 3. Duplicate key / constraint violation (error 2627)
1494
+for _ in 1 2 3; do
1495
+ mssql_exec_sa_allow_error "INSERT INTO dbo.error_test (id, unique_col, int_col) VALUES (1, 'new_value', 200);"
1496
+ mssql_exec_sa_allow_error "INSERT INTO dbo.error_test (id, unique_col, int_col) VALUES (99, 'existing_value', 300);"
1497
+done
1498
+
1499
+# 4. Data type conversion error (error 245)
1500
+for _ in 1 2 3; do
1501
+ mssql_exec_sa_allow_error "SELECT CAST('not_a_number' AS INT);"
1502
+done
1503
+
1504
+# 5. Division by zero (error 8134)
1505
+for _ in 1 2 3; do
1506
+ mssql_exec_sa_allow_error "SELECT 1/0;"
1507
+done
1508
+
1509
+for _ in 1 2 3 4 5; do
1510
+ mssql_exec_sa "SET NOCOUNT ON; SELECT a.id, b.name FROM dbo.sample a JOIN dbo.sample b ON a.id = b.id ORDER BY a.value + b.value DESC OPTION (HASH JOIN);"
1511
+done
1512
+
1513
+mssql_exec_sa "EXEC sys.sp_query_store_flush_db;"
1514
+sleep 2
1515
+
1516
+error_output="$(run_mssql_function_with_retry error-info '__job:local' 'true')"
1517
+assert_error_info_has_errors "$error_output"
1518
+
1519
+run_mssql_top_queries_with_retry
1520
+assert_top_queries_error_attribution_active "$WORKDIR/mssql-top-queries.json"
1521
+assert_top_queries_error_attribution_mapped "$WORKDIR/mssql-top-queries.json" "$WORKDIR/mssql-error-info.json"
1522
+assert_top_queries_plan_ops "$WORKDIR/mssql-top-queries.json"
1523
+
1524
+echo "E2E checks passed for ${MSSQL_VARIANT_LABEL}." >&2
src/go/tools/functions-validation/e2e/mysql-matrix.sh
new
+21
@@ -0,0 +1,21 @@
1
+#!/usr/bin/env bash
2
+set -euo pipefail
3
+
4
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5
+
6
+MYSQL_VARIANTS=(
7
+ "mysql:5.7|mysql-5.7"
8
+ "mysql:8.0|mysql-8.0"
9
+ "mysql:8.4|mysql-8.4"
10
+ "mariadb:10.3|mariadb-10.3"
11
+ "mariadb:10.6|mariadb-10.6"
12
+ "mariadb:11.4|mariadb-11.4"
13
+ "percona:5.7|percona-5.7"
14
+ "percona:8.0|percona-8.0"
15
+)
16
+
17
+for entry in "${MYSQL_VARIANTS[@]}"; do
18
+ IFS='|' read -r image label <<< "$entry"
19
+ printf '\n=== Running MySQL collector E2E for %s (%s) ===\n' "$label" "$image" >&2
20
+ MYSQL_IMAGE="$image" MYSQL_VARIANT="$label" bash "$SCRIPT_DIR/mysql.sh"
21
+done
src/go/tools/functions-validation/e2e/mysql.sh
+935
-2
@@ -12,11 +12,944 @@ MYSQL_PORT="$(reserve_port)"
12
write_env "MYSQL_PORT" "$MYSQL_PORT"
13
replace_in_file "$WORKDIR/config/go.d/mysql.conf" "127.0.0.1:3306" "127.0.0.1:${MYSQL_PORT}"
14
15
+MYSQL_VARIANT_LABEL="${MYSQL_VARIANT:-mysql}"
16
+if [ -n "${MYSQL_IMAGE:-}" ]; then
17
+ write_env "MYSQL_IMAGE" "$MYSQL_IMAGE"
18
+fi
19
+
20
compose_up mysql
16
-wait_healthy mysql 90
21
+MYSQL_HEALTH_TIMEOUT="${MYSQL_HEALTH_TIMEOUT:-180}"
22
+wait_healthy mysql "$MYSQL_HEALTH_TIMEOUT"
23
24
build_plugin
25
run_info mysql
26
run_top_queries mysql
27
22
-echo "E2E checks passed for mysql." >&2
28
+assert_top_queries_error_columns() {
29
+ local input="$1"
30
+ if command -v python3 >/dev/null 2>&1; then
31
+ python3 - "$input" <<'PY'
32
+import io
33
+import json
34
+import sys
35
+
36
+path = sys.argv[1]
37
+with io.open(path, "r", encoding="utf-8") as fh:
38
+ doc = json.load(fh)
39
+columns = doc.get("columns") or {}
40
+required = {"errorAttribution", "errorNumber", "sqlState", "errorMessage"}
41
+
42
+found = set()
43
+if isinstance(columns, dict):
44
+ for key in columns.keys():
45
+ found.add(key)
46
+else:
47
+ for col in columns:
48
+ if isinstance(col, dict):
49
+ field = col.get("field")
50
+ if field:
51
+ found.add(field)
52
+
53
+missing = sorted(required - found)
54
+if missing:
55
+ raise SystemExit("missing top-queries error columns: {}".format(missing))
56
+PY
57
+ return
58
+ fi
59
+ python - "$input" <<'PY'
60
+import io
61
+import json
62
+import sys
63
+
64
+path = sys.argv[1]
65
+with open(path, "r") as fh:
66
+ doc = json.load(fh)
67
+columns = doc.get("columns") or {}
68
+required = {"errorAttribution", "errorNumber", "sqlState", "errorMessage"}
69
+
70
+found = set()
71
+if isinstance(columns, dict):
72
+ for key in columns.keys():
73
+ found.add(key)
74
+else:
75
+ for col in columns:
76
+ if isinstance(col, dict):
77
+ field = col.get("field")
78
+ if field:
79
+ found.add(field)
80
+
81
+missing = sorted(required - found)
82
+if missing:
83
+ raise SystemExit("missing top-queries error columns: %s" % missing)
84
+PY
85
+}
86
+
87
+assert_top_queries_error_columns "$WORKDIR/mysql-top-queries.json"
88
+
89
+mysql_container_id() {
90
+ "${COMPOSE[@]}" ps -q mysql
91
+}
92
+
93
+mysql_client_path() {
94
+ local cid
95
+ cid="$(mysql_container_id)"
96
+ if [ -z "$cid" ]; then
97
+ echo "MySQL container ID not found" >&2
98
+ return 1
99
+ fi
100
+ docker exec -i "$cid" sh -lc 'command -v mysql || command -v mariadb'
101
+}
102
+
103
+MYSQL_CLIENT_PATH="$(mysql_client_path)"
104
+echo "Using mysql client: $MYSQL_CLIENT_PATH" >&2
105
+
106
+mysql_exec_root() {
107
+ local sql="$1"
108
+ local cid
109
+ cid="$(mysql_container_id)"
110
+ if [ -z "$cid" ]; then
111
+ echo "MySQL container ID not found" >&2
112
+ return 1
113
+ fi
114
+ run docker exec -i "$cid" "$MYSQL_CLIENT_PATH" -uroot -prootpw netdata -e "$sql"
115
+}
116
+
117
+mysql_query_root() {
118
+ local sql="$1"
119
+ local cid
120
+ cid="$(mysql_container_id)"
121
+ if [ -z "$cid" ]; then
122
+ echo "MySQL container ID not found" >&2
123
+ return 1
124
+ fi
125
+ run docker exec -i "$cid" "$MYSQL_CLIENT_PATH" -uroot -prootpw -N -s netdata -e "$sql"
126
+}
127
+
128
+mysql_exec_root_allow_error() {
129
+ local sql="$1"
130
+ local cid
131
+ cid="$(mysql_container_id)"
132
+ if [ -z "$cid" ]; then
133
+ echo "MySQL container ID not found" >&2
134
+ return 1
135
+ fi
136
+ set +e
137
+ docker exec -i "$cid" "$MYSQL_CLIENT_PATH" -uroot -prootpw netdata -e "$sql" >/dev/null 2>&1
138
+ set -e
139
+}
140
+
141
+induce_deadlock_once() {
142
+ local tx1
143
+ local tx2
144
+
145
+ tx1="$(cat <<'SQL'
146
+SET SESSION innodb_lock_wait_timeout = 5;
147
+START TRANSACTION;
148
+UPDATE deadlock_a SET value = value + 1 WHERE id = 1;
149
+DO SLEEP(1);
150
+UPDATE deadlock_b SET value = value + 1 WHERE id = 1;
151
+COMMIT;
152
+SQL
153
+)"
154
+
155
+ tx2="$(cat <<'SQL'
156
+SET SESSION innodb_lock_wait_timeout = 5;
157
+START TRANSACTION;
158
+UPDATE deadlock_b SET value = value + 1 WHERE id = 1;
159
+DO SLEEP(1);
160
+UPDATE deadlock_a SET value = value + 1 WHERE id = 1;
161
+COMMIT;
162
+SQL
163
+)"
164
+
165
+ mysql_exec_root "$tx1" &
166
+ local pid1=$!
167
+ mysql_exec_root "$tx2" &
168
+ local pid2=$!
169
+
170
+ wait "$pid1" || true
171
+ wait "$pid2" || true
172
+}
173
+
174
+assert_deadlock_info_content() {
175
+ local input="$1"
176
+ if command -v python3 >/dev/null 2>&1; then
177
+ python3 - "$input" <<'PY'
178
+import io
179
+import json
180
+import re
181
+import sys
182
+
183
+path = sys.argv[1]
184
+with io.open(path, "r", encoding="utf-8") as fh:
185
+ doc = json.load(fh)
186
+
187
+try:
188
+ status = int(doc.get("status"))
189
+except (TypeError, ValueError):
190
+ raise SystemExit("unexpected status value: {!r}".format(doc.get("status")))
191
+
192
+if status != 200:
193
+ raise SystemExit("expected status 200, got {}".format(status))
194
+
195
+if doc.get("errorMessage"):
196
+ raise SystemExit("unexpected errorMessage on status 200: {!r}".format(doc.get("errorMessage")))
197
+
198
+columns = doc.get("columns") or {}
199
+field_to_idx = {}
200
+if isinstance(columns, dict):
201
+ for field, col in columns.items():
202
+ if not isinstance(col, dict):
203
+ continue
204
+ try:
205
+ field_to_idx[field] = int(col.get("index"))
206
+ except (TypeError, ValueError):
207
+ continue
208
+else:
209
+ for idx, col in enumerate(columns):
210
+ if not isinstance(col, dict):
211
+ continue
212
+ field = col.get("field")
213
+ if field:
214
+ field_to_idx[field] = idx
215
+
216
+for required in ("row_id", "deadlock_id", "process_id", "is_victim", "lock_mode", "lock_status", "query_text", "wait_resource", "database"):
217
+ if required not in field_to_idx:
218
+ raise SystemExit("missing expected column: {}".format(required))
219
+
220
+data = doc.get("data") or []
221
+if not data:
222
+ raise SystemExit("deadlock-info returned no rows")
223
+
224
+def get_value(row, field):
225
+ idx = field_to_idx[field]
226
+ return row[idx] if idx < len(row) else None
227
+
228
+def norm(val):
229
+ return "" if val is None else str(val).strip()
230
+
231
+has_waiting = any(str(get_value(row, "lock_status")).upper() == "WAITING" for row in data)
232
+if not has_waiting:
233
+ raise SystemExit("no WAITING lock_status found in deadlock-info output")
234
+
235
+table_pattern = re.compile(r"deadlock_(a|b)", re.IGNORECASE)
236
+has_expected_query = any(table_pattern.search(str(get_value(row, "query_text"))) for row in data)
237
+if not has_expected_query:
238
+ raise SystemExit("query_text does not reference deadlock tables")
239
+
240
+waiting_rows = [row for row in data if norm(get_value(row, "lock_status")).upper() == "WAITING"]
241
+if any(norm(get_value(row, "wait_resource")) == "" for row in waiting_rows):
242
+ raise SystemExit("WAITING rows must include wait_resource")
243
+
244
+lock_mode_re = re.compile(r"^[A-Za-z0-9_-]+$")
245
+if any(norm(get_value(row, "lock_mode")) != "" and not lock_mode_re.match(norm(get_value(row, "lock_mode"))) for row in waiting_rows):
246
+ raise SystemExit("WAITING rows must include a valid lock_mode")
247
+
248
+victim_counts = {}
249
+expected_db = "netdata"
250
+has_database = False
251
+for row in data:
252
+ deadlock_id = norm(get_value(row, "deadlock_id"))
253
+ if deadlock_id == "":
254
+ raise SystemExit("deadlock_id missing from deadlock-info output")
255
+ process_id = norm(get_value(row, "process_id"))
256
+ if process_id == "":
257
+ raise SystemExit("process_id missing from deadlock-info output")
258
+ row_id = norm(get_value(row, "row_id"))
259
+ if row_id != "{}:{}".format(deadlock_id, process_id):
260
+ raise SystemExit("row_id {} does not match deadlock_id/process_id".format(row_id))
261
+ victim_counts.setdefault(deadlock_id, 0)
262
+ if norm(get_value(row, "is_victim")).lower() == "true":
263
+ victim_counts[deadlock_id] += 1
264
+ db_val = norm(get_value(row, "database")).lower()
265
+ if db_val:
266
+ has_database = True
267
+ if db_val != expected_db:
268
+ raise SystemExit("unexpected database value {!r}, expected {!r}".format(db_val, expected_db))
269
+
270
+for deadlock_id, count in victim_counts.items():
271
+ if count != 1:
272
+ raise SystemExit("deadlock_id {} has victim count {}, expected 1".format(deadlock_id, count))
273
+if not has_database:
274
+ raise SystemExit("expected at least one row with database populated")
275
+PY
276
+ else
277
+ python - "$input" <<'PY'
278
+import io
279
+import json
280
+import re
281
+import sys
282
+
283
+path = sys.argv[1]
284
+with io.open(path, "r", encoding="utf-8") as fh:
285
+ doc = json.load(fh)
286
+
287
+try:
288
+ status = int(doc.get("status"))
289
+except (TypeError, ValueError):
290
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
291
+
292
+if status != 200:
293
+ raise SystemExit("expected status 200, got %s" % status)
294
+
295
+if doc.get("errorMessage"):
296
+ raise SystemExit("unexpected errorMessage on status 200: %r" % (doc.get("errorMessage"),))
297
+
298
+columns = doc.get("columns") or {}
299
+field_to_idx = {}
300
+if isinstance(columns, dict):
301
+ for field, col in columns.items():
302
+ if not isinstance(col, dict):
303
+ continue
304
+ try:
305
+ field_to_idx[field] = int(col.get("index"))
306
+ except (TypeError, ValueError):
307
+ continue
308
+else:
309
+ for idx, col in enumerate(columns):
310
+ if not isinstance(col, dict):
311
+ continue
312
+ field = col.get("field")
313
+ if field:
314
+ field_to_idx[field] = idx
315
+
316
+for required in ("row_id", "deadlock_id", "process_id", "is_victim", "lock_mode", "lock_status", "query_text", "wait_resource", "database"):
317
+ if required not in field_to_idx:
318
+ raise SystemExit("missing expected column: {}".format(required))
319
+
320
+data = doc.get("data") or []
321
+if not data:
322
+ raise SystemExit("deadlock-info returned no rows")
323
+
324
+def get_value(row, field):
325
+ idx = field_to_idx[field]
326
+ return row[idx] if idx < len(row) else None
327
+
328
+def norm(val):
329
+ return "" if val is None else str(val).strip()
330
+
331
+has_waiting = any(str(get_value(row, "lock_status")).upper() == "WAITING" for row in data)
332
+if not has_waiting:
333
+ raise SystemExit("no WAITING lock_status found in deadlock-info output")
334
+
335
+table_pattern = re.compile(r"deadlock_(a|b)", re.IGNORECASE)
336
+has_expected_query = any(table_pattern.search(str(get_value(row, "query_text"))) for row in data)
337
+if not has_expected_query:
338
+ raise SystemExit("query_text does not reference deadlock tables")
339
+
340
+waiting_rows = [row for row in data if norm(get_value(row, "lock_status")).upper() == "WAITING"]
341
+if any(norm(get_value(row, "lock_mode")) == "" for row in waiting_rows):
342
+ raise SystemExit("WAITING rows must include lock_mode")
343
+if any(norm(get_value(row, "wait_resource")) == "" for row in waiting_rows):
344
+ raise SystemExit("WAITING rows must include wait_resource")
345
+
346
+lock_mode_re = re.compile(r"^[A-Za-z0-9_-]+$")
347
+if any(not lock_mode_re.match(norm(get_value(row, "lock_mode"))) for row in waiting_rows):
348
+ raise SystemExit("WAITING rows must include a valid lock_mode")
349
+
350
+victim_counts = {}
351
+expected_db = "netdata"
352
+has_database = False
353
+for row in data:
354
+ deadlock_id = norm(get_value(row, "deadlock_id"))
355
+ if deadlock_id == "":
356
+ raise SystemExit("deadlock_id missing from deadlock-info output")
357
+ process_id = norm(get_value(row, "process_id"))
358
+ if process_id == "":
359
+ raise SystemExit("process_id missing from deadlock-info output")
360
+ row_id = norm(get_value(row, "row_id"))
361
+ if row_id != "%s:%s" % (deadlock_id, process_id):
362
+ raise SystemExit("row_id %s does not match deadlock_id/process_id" % row_id)
363
+ victim_counts.setdefault(deadlock_id, 0)
364
+ if norm(get_value(row, "is_victim")).lower() == "true":
365
+ victim_counts[deadlock_id] += 1
366
+ db_val = norm(get_value(row, "database")).lower()
367
+ if db_val:
368
+ has_database = True
369
+ if db_val != expected_db:
370
+ raise SystemExit("unexpected database value %r, expected %r" % (db_val, expected_db))
371
+
372
+for deadlock_id, count in victim_counts.items():
373
+ if count != 1:
374
+ raise SystemExit("deadlock_id {} has victim count {}, expected 1".format(deadlock_id, count))
375
+if not has_database:
376
+ raise SystemExit("expected at least one row with database populated")
377
+PY
378
+ fi
379
+}
380
+
381
+assert_deadlock_info_empty_success() {
382
+ local input="$1"
383
+
384
+ if command -v python3 >/dev/null 2>&1; then
385
+ python3 - "$input" <<'PY'
386
+import io
387
+import json
388
+import sys
389
+
390
+path = sys.argv[1]
391
+with io.open(path, "r", encoding="utf-8") as fh:
392
+ doc = json.load(fh)
393
+
394
+try:
395
+ status = int(doc.get("status"))
396
+except (TypeError, ValueError):
397
+ raise SystemExit("unexpected status value: {!r}".format(doc.get("status")))
398
+
399
+if status != 200:
400
+ raise SystemExit("expected status 200, got {}".format(status))
401
+
402
+if doc.get("errorMessage"):
403
+ raise SystemExit("unexpected errorMessage on status 200: {!r}".format(doc.get("errorMessage")))
404
+
405
+data = doc.get("data") or []
406
+if len(data) != 0:
407
+ raise SystemExit("expected no rows, got {}".format(len(data)))
408
+PY
409
+ return
410
+ fi
411
+
412
+ python - "$input" <<'PY'
413
+import io
414
+import json
415
+import sys
416
+
417
+path = sys.argv[1]
418
+with open(path, "r") as fh:
419
+ doc = json.load(fh)
420
+
421
+try:
422
+ status = int(doc.get("status"))
423
+except (TypeError, ValueError):
424
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
425
+
426
+if status != 200:
427
+ raise SystemExit("expected status 200, got %s" % status)
428
+
429
+if doc.get("errorMessage"):
430
+ raise SystemExit("unexpected errorMessage on status 200: %r" % (doc.get("errorMessage"),))
431
+
432
+data = doc.get("data") or []
433
+if len(data) != 0:
434
+ raise SystemExit("expected no rows, got %s" % len(data))
435
+PY
436
+}
437
+
438
+assert_error_info_not_enabled() {
439
+ local input="$1"
440
+
441
+ if command -v python3 >/dev/null 2>&1; then
442
+ python3 - "$input" <<'PY'
443
+import io
444
+import json
445
+import sys
446
+
447
+path = sys.argv[1]
448
+with io.open(path, "r", encoding="utf-8") as fh:
449
+ doc = json.load(fh)
450
+
451
+try:
452
+ status = int(doc.get("status"))
453
+except (TypeError, ValueError):
454
+ raise SystemExit("unexpected status value: {!r}".format(doc.get("status")))
455
+
456
+if status < 400:
457
+ raise SystemExit("expected error status, got {}".format(status))
458
+
459
+err = str(doc.get("errorMessage") or "").lower()
460
+if "not enabled" not in err:
461
+ raise SystemExit("expected errorMessage to contain 'not enabled', got {!r}".format(err))
462
+PY
463
+ return
464
+ fi
465
+
466
+ python - "$input" <<'PY'
467
+import io
468
+import json
469
+import sys
470
+
471
+path = sys.argv[1]
472
+with open(path, "r") as fh:
473
+ doc = json.load(fh)
474
+
475
+try:
476
+ status = int(doc.get("status"))
477
+except (TypeError, ValueError):
478
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
479
+
480
+if status < 400:
481
+ raise SystemExit("expected error status, got %s" % status)
482
+
483
+err = str(doc.get("errorMessage") or "").lower()
484
+if "not enabled" not in err:
485
+ raise SystemExit("expected errorMessage to contain 'not enabled', got %r" % err)
486
+PY
487
+}
488
+
489
+assert_error_info_has_errors() {
490
+ local input="$1"
491
+
492
+ if command -v python3 >/dev/null 2>&1; then
493
+ python3 - "$input" <<'PY'
494
+import io
495
+import json
496
+import sys
497
+
498
+path = sys.argv[1]
499
+with io.open(path, "r", encoding="utf-8") as fh:
500
+ doc = json.load(fh)
501
+
502
+try:
503
+ status = int(doc.get("status"))
504
+except (TypeError, ValueError):
505
+ raise SystemExit("unexpected status value: {!r}".format(doc.get("status")))
506
+
507
+if status != 200:
508
+ raise SystemExit("expected status 200, got {}".format(status))
509
+
510
+if doc.get("errorMessage"):
511
+ raise SystemExit("unexpected errorMessage on status 200: {!r}".format(doc.get("errorMessage")))
512
+
513
+columns = doc.get("columns") or {}
514
+field_to_idx = {}
515
+if isinstance(columns, dict):
516
+ for field, col in columns.items():
517
+ if not isinstance(col, dict):
518
+ continue
519
+ try:
520
+ field_to_idx[field] = int(col.get("index"))
521
+ except (TypeError, ValueError):
522
+ continue
523
+else:
524
+ for idx, col in enumerate(columns):
525
+ if not isinstance(col, dict):
526
+ continue
527
+ field = col.get("field")
528
+ if field:
529
+ field_to_idx[field] = idx
530
+
531
+for required in ("errorNumber", "errorMessage"):
532
+ if required not in field_to_idx:
533
+ raise SystemExit("missing expected column: {}".format(required))
534
+
535
+data = doc.get("data") or []
536
+if not data:
537
+ raise SystemExit("error-info returned no rows")
538
+
539
+err_idx = field_to_idx["errorMessage"]
540
+num_idx = field_to_idx["errorNumber"]
541
+def normalize(val):
542
+ return "" if val is None else str(val)
543
+
544
+# Error categories to verify:
545
+# 1146 - Table doesn't exist (missing_table)
546
+# 1062 - Duplicate key (constraint violation)
547
+# Note: Syntax errors (1064) are not captured in events_statements_history
548
+# because they fail during parsing before instrumentation.
549
+error_categories = {
550
+ "table_not_found": {"patterns": ["missing_table", "doesn't exist", "does not exist"], "found": False},
551
+ "duplicate_key": {"patterns": ["duplicate", "primary", "unique"], "found": False},
552
+}
553
+
554
+for row in data:
555
+ if num_idx >= len(row) or row[num_idx] is None:
556
+ continue
557
+ msg = normalize(row[err_idx]).lower()
558
+ for cat, info in error_categories.items():
559
+ if info["found"]:
560
+ continue
561
+ for pattern in info["patterns"]:
562
+ if pattern in msg:
563
+ info["found"] = True
564
+ break
565
+
566
+missing = [cat for cat, info in error_categories.items() if not info["found"]]
567
+if missing:
568
+ raise SystemExit("error-info missing error categories: {}".format(", ".join(missing)))
569
+PY
570
+ return
571
+ fi
572
+
573
+ python - "$input" <<'PY'
574
+import io
575
+import json
576
+import sys
577
+
578
+path = sys.argv[1]
579
+with open(path, "r") as fh:
580
+ doc = json.load(fh)
581
+
582
+try:
583
+ status = int(doc.get("status"))
584
+except (TypeError, ValueError):
585
+ raise SystemExit("unexpected status value: %r" % (doc.get("status"),))
586
+
587
+if status != 200:
588
+ raise SystemExit("expected status 200, got %s" % status)
589
+
590
+if doc.get("errorMessage"):
591
+ raise SystemExit("unexpected errorMessage on status 200: %r" % (doc.get("errorMessage"),))
592
+
593
+columns = doc.get("columns") or {}
594
+field_to_idx = {}
595
+if isinstance(columns, dict):
596
+ for field, col in columns.items():
597
+ if not isinstance(col, dict):
598
+ continue
599
+ try:
600
+ field_to_idx[field] = int(col.get("index"))
601
+ except (TypeError, ValueError):
602
+ continue
603
+else:
604
+ for idx, col in enumerate(columns):
605
+ if not isinstance(col, dict):
606
+ continue
607
+ field = col.get("field")
608
+ if field:
609
+ field_to_idx[field] = idx
610
+
611
+for required in ("errorNumber", "errorMessage"):
612
+ if required not in field_to_idx:
613
+ raise SystemExit("missing expected column: %s" % required)
614
+
615
+data = doc.get("data") or []
616
+if not data:
617
+ raise SystemExit("error-info returned no rows")
618
+
619
+err_idx = field_to_idx["errorMessage"]
620
+num_idx = field_to_idx["errorNumber"]
621
+def normalize(val):
622
+ return "" if val is None else str(val)
623
+
624
+# Error categories to verify:
625
+# 1146 - Table doesn't exist (missing_table)
626
+# 1062 - Duplicate key (constraint violation)
627
+# Note: Syntax errors (1064) are not captured in events_statements_history
628
+# because they fail during parsing before instrumentation.
629
+error_categories = {
630
+ "table_not_found": {"patterns": ["missing_table", "doesn't exist", "does not exist"], "found": False},
631
+ "duplicate_key": {"patterns": ["duplicate", "primary", "unique"], "found": False},
632
+}
633
+
634
+for row in data:
635
+ if num_idx >= len(row) or row[num_idx] is None:
636
+ continue
637
+ msg = normalize(row[err_idx]).lower()
638
+ for cat, info in error_categories.items():
639
+ if info["found"]:
640
+ continue
641
+ for pattern in info["patterns"]:
642
+ if pattern in msg:
643
+ info["found"] = True
644
+ break
645
+
646
+missing = [cat for cat, info in error_categories.items() if not info["found"]]
647
+if missing:
648
+ raise SystemExit("error-info missing error categories: %s" % ", ".join(missing))
649
+PY
650
+}
651
+
652
+assert_top_queries_error_attribution_enabled() {
653
+ local input="$1"
654
+
655
+ if command -v python3 >/dev/null 2>&1; then
656
+ python3 - "$input" <<'PY'
657
+import io
658
+import json
659
+import sys
660
+
661
+path = sys.argv[1]
662
+with io.open(path, "r", encoding="utf-8") as fh:
663
+ doc = json.load(fh)
664
+
665
+columns = doc.get("columns") or {}
666
+field_to_idx = {}
667
+if isinstance(columns, dict):
668
+ for field, col in columns.items():
669
+ if not isinstance(col, dict):
670
+ continue
671
+ try:
672
+ field_to_idx[field] = int(col.get("index"))
673
+ except (TypeError, ValueError):
674
+ continue
675
+else:
676
+ for idx, col in enumerate(columns):
677
+ if not isinstance(col, dict):
678
+ continue
679
+ field = col.get("field")
680
+ if field:
681
+ field_to_idx[field] = idx
682
+
683
+for required in ("errorAttribution", "errorNumber", "errorMessage"):
684
+ if required not in field_to_idx:
685
+ raise SystemExit("missing expected column: {}".format(required))
686
+
687
+data = doc.get("data") or []
688
+status_idx = field_to_idx["errorAttribution"]
689
+num_idx = field_to_idx["errorNumber"]
690
+msg_idx = field_to_idx["errorMessage"]
691
+
692
+matched = False
693
+for row in data:
694
+ if status_idx >= len(row):
695
+ continue
696
+ if str(row[status_idx]) != "enabled":
697
+ continue
698
+ num = row[num_idx] if num_idx < len(row) else None
699
+ msg = str(row[msg_idx]).lower() if msg_idx < len(row) else ""
700
+ if num is not None and "missing_table" in msg:
701
+ matched = True
702
+ break
703
+
704
+if not matched:
705
+ raise SystemExit("no top-queries row had enabled error attribution for missing_table")
706
+PY
707
+ return
708
+ fi
709
+
710
+ python - "$input" <<'PY'
711
+import io
712
+import json
713
+import sys
714
+
715
+path = sys.argv[1]
716
+with open(path, "r") as fh:
717
+ doc = json.load(fh)
718
+
719
+columns = doc.get("columns") or {}
720
+field_to_idx = {}
721
+if isinstance(columns, dict):
722
+ for field, col in columns.items():
723
+ if not isinstance(col, dict):
724
+ continue
725
+ try:
726
+ field_to_idx[field] = int(col.get("index"))
727
+ except (TypeError, ValueError):
728
+ continue
729
+else:
730
+ for idx, col in enumerate(columns):
731
+ if not isinstance(col, dict):
732
+ continue
733
+ field = col.get("field")
734
+ if field:
735
+ field_to_idx[field] = idx
736
+
737
+for required in ("errorAttribution", "errorNumber", "errorMessage"):
738
+ if required not in field_to_idx:
739
+ raise SystemExit("missing expected column: %s" % required)
740
+
741
+data = doc.get("data") or []
742
+status_idx = field_to_idx["errorAttribution"]
743
+num_idx = field_to_idx["errorNumber"]
744
+msg_idx = field_to_idx["errorMessage"]
745
+
746
+matched = False
747
+for row in data:
748
+ if status_idx >= len(row):
749
+ continue
750
+ if str(row[status_idx]) != "enabled":
751
+ continue
752
+ num = row[num_idx] if num_idx < len(row) else None
753
+ msg = str(row[msg_idx]).lower() if msg_idx < len(row) else ""
754
+ if num is not None and "missing_table" in msg:
755
+ matched = True
756
+ break
757
+
758
+if not matched:
759
+ raise SystemExit("no top-queries row had enabled error attribution for missing_table")
760
+PY
761
+}
762
+
763
+assert_top_queries_error_attribution_not_enabled() {
764
+ local input="$1"
765
+
766
+ if command -v python3 >/dev/null 2>&1; then
767
+ python3 - "$input" <<'PY'
768
+import io
769
+import json
770
+import sys
771
+
772
+path = sys.argv[1]
773
+with io.open(path, "r", encoding="utf-8") as fh:
774
+ doc = json.load(fh)
775
+
776
+columns = doc.get("columns") or {}
777
+field_to_idx = {}
778
+if isinstance(columns, dict):
779
+ for field, col in columns.items():
780
+ if not isinstance(col, dict):
781
+ continue
782
+ try:
783
+ field_to_idx[field] = int(col.get("index"))
784
+ except (TypeError, ValueError):
785
+ continue
786
+else:
787
+ for idx, col in enumerate(columns):
788
+ if not isinstance(col, dict):
789
+ continue
790
+ field = col.get("field")
791
+ if field:
792
+ field_to_idx[field] = idx
793
+
794
+if "errorAttribution" not in field_to_idx:
795
+ raise SystemExit("missing expected column: errorAttribution")
796
+
797
+data = doc.get("data") or []
798
+idx = field_to_idx["errorAttribution"]
799
+for row in data:
800
+ if idx >= len(row):
801
+ continue
802
+ if str(row[idx]) != "not_enabled":
803
+ raise SystemExit("expected errorAttribution 'not_enabled', got {!r}".format(row[idx]))
804
+PY
805
+ return
806
+ fi
807
+
808
+ python - "$input" <<'PY'
809
+import io
810
+import json
811
+import sys
812
+
813
+path = sys.argv[1]
814
+with open(path, "r") as fh:
815
+ doc = json.load(fh)
816
+
817
+columns = doc.get("columns") or {}
818
+field_to_idx = {}
819
+if isinstance(columns, dict):
820
+ for field, col in columns.items():
821
+ if not isinstance(col, dict):
822
+ continue
823
+ try:
824
+ field_to_idx[field] = int(col.get("index"))
825
+ except (TypeError, ValueError):
826
+ continue
827
+else:
828
+ for idx, col in enumerate(columns):
829
+ if not isinstance(col, dict):
830
+ continue
831
+ field = col.get("field")
832
+ if field:
833
+ field_to_idx[field] = idx
834
+
835
+if "errorAttribution" not in field_to_idx:
836
+ raise SystemExit("missing expected column: errorAttribution")
837
+
838
+data = doc.get("data") or []
839
+idx = field_to_idx["errorAttribution"]
840
+for row in data:
841
+ if idx >= len(row):
842
+ continue
843
+ if str(row[idx]) != "not_enabled":
844
+ raise SystemExit("expected errorAttribution 'not_enabled', got %r" % row[idx])
845
+PY
846
+}
847
+
848
+capture_statement_history_states() {
849
+ local output
850
+ output="$(mysql_query_root "
851
+SELECT
852
+ COALESCE(MAX(CASE WHEN NAME = 'events_statements_history_long' THEN ENABLED END), 'NO') AS history_long,
853
+ COALESCE(MAX(CASE WHEN NAME = 'events_statements_history' THEN ENABLED END), 'NO') AS history,
854
+ COALESCE(MAX(CASE WHEN NAME = 'events_statements_current' THEN ENABLED END), 'NO') AS history_current
855
+FROM performance_schema.setup_consumers
856
+WHERE NAME IN ('events_statements_history_long','events_statements_history','events_statements_current');")"
857
+ local history_long
858
+ local history
859
+ local history_current
860
+ IFS=$'\t' read -r history_long history history_current <<<"$output"
861
+ MYSQL_HISTORY_LONG_STATE="$history_long"
862
+ MYSQL_HISTORY_STATE="$history"
863
+ MYSQL_HISTORY_CURRENT_STATE="$history_current"
864
+}
865
+
866
+disable_statement_history_consumers() {
867
+ mysql_exec_root "UPDATE performance_schema.setup_consumers SET ENABLED = 'NO' WHERE NAME IN ('events_statements_history_long','events_statements_history','events_statements_current');"
868
+}
869
+
870
+enable_statement_history_consumers() {
871
+ mysql_exec_root "UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME IN ('events_statements_history_long','events_statements_history','events_statements_current');"
872
+}
873
+
874
+restore_statement_history_consumers() {
875
+ if [ -n "${MYSQL_HISTORY_LONG_STATE:-}" ]; then
876
+ mysql_exec_root "UPDATE performance_schema.setup_consumers SET ENABLED = '${MYSQL_HISTORY_LONG_STATE}' WHERE NAME = 'events_statements_history_long';"
877
+ fi
878
+ if [ -n "${MYSQL_HISTORY_STATE:-}" ]; then
879
+ mysql_exec_root "UPDATE performance_schema.setup_consumers SET ENABLED = '${MYSQL_HISTORY_STATE}' WHERE NAME = 'events_statements_history';"
880
+ fi
881
+ if [ -n "${MYSQL_HISTORY_CURRENT_STATE:-}" ]; then
882
+ mysql_exec_root "UPDATE performance_schema.setup_consumers SET ENABLED = '${MYSQL_HISTORY_CURRENT_STATE}' WHERE NAME = 'events_statements_current';"
883
+ fi
884
+}
885
+
886
+verify_deadlock_info_no_deadlock() {
887
+ local output
888
+
889
+ output="$(run_function mysql deadlock-info '__job:local' 'false')"
890
+ validate "$output"
891
+ assert_deadlock_info_empty_success "$output"
892
+}
893
+
894
+verify_deadlock_info() {
895
+ local output=""
896
+ local found="false"
897
+
898
+ for attempt in 1 2 3 4 5; do
899
+ induce_deadlock_once
900
+ output="$(run_function mysql deadlock-info '__job:local' 'false')"
901
+ if has_min_rows "$output" 1; then
902
+ validate "$output" --min-rows 1
903
+ if assert_deadlock_info_content "$output"; then
904
+ found="true"
905
+ break
906
+ fi
907
+ fi
908
+ sleep 1
909
+ done
910
+
911
+ if [ "$found" != "true" ]; then
912
+ echo "deadlock-info did not produce valid deadlock attribution after 5 attempts" >&2
913
+ return 1
914
+ fi
915
+}
916
+
917
+verify_deadlock_info_no_deadlock
918
+verify_deadlock_info
919
+
920
+capture_statement_history_states
921
+disable_statement_history_consumers
922
+
923
+error_output="$(run_function mysql error-info '__job:local' 'false')"
924
+assert_error_info_not_enabled "$error_output"
925
+
926
+run_top_queries mysql
927
+assert_top_queries_error_attribution_not_enabled "$WORKDIR/mysql-top-queries.json"
928
+
929
+enable_statement_history_consumers
930
+
931
+# Generate errors for multiple categories:
932
+# 1. Table not found (error 1146)
933
+for _ in 1 2 3; do
934
+ mysql_exec_root_allow_error "SELECT * FROM missing_table;"
935
+done
936
+
937
+# 2. Duplicate key / constraint violation (error 1062)
938
+# Note: Syntax errors (1064) are NOT captured in events_statements_history
939
+# because they fail during parsing before instrumentation records them.
940
+for _ in 1 2 3; do
941
+ mysql_exec_root_allow_error "INSERT INTO error_test (id, unique_col, int_col) VALUES (1, 'new_value', 200);"
942
+ mysql_exec_root_allow_error "INSERT INTO error_test (id, unique_col, int_col) VALUES (99, 'existing_value', 300);"
943
+done
944
+
945
+sleep 1
946
+
947
+error_output="$(run_function mysql error-info '__job:local' 'true')"
948
+assert_error_info_has_errors "$error_output"
949
+
950
+run_top_queries mysql
951
+assert_top_queries_error_attribution_enabled "$WORKDIR/mysql-top-queries.json"
952
+
953
+restore_statement_history_consumers
954
+
955
+echo "E2E checks passed for ${MYSQL_VARIANT_LABEL}." >&2
src/go/tools/functions-validation/seed/mssql/init.sql
+82
@@ -7,9 +7,44 @@ GO
7
ALTER DATABASE netdata SET QUERY_STORE = ON;
8
GO
9
10
+IF NOT EXISTS (SELECT 1 FROM sys.server_principals WHERE name = 'netdata_limited')
11
+BEGIN
12
+ CREATE LOGIN netdata_limited WITH PASSWORD = 'Netdata123!';
13
+END
14
+GO
15
+
16
+-- Ensure the limited user can start the collector in E2E:
17
+-- A previous DENY will override GRANT, so revoke first.
18
+REVOKE VIEW SERVER STATE FROM netdata_limited;
19
+GO
20
+
21
+GRANT VIEW SERVER STATE TO netdata_limited;
22
+GO
23
+
24
+USE msdb;
25
+GO
26
+
27
+IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE name = 'netdata_limited')
28
+BEGIN
29
+ CREATE USER netdata_limited FOR LOGIN netdata_limited;
30
+END
31
+GO
32
+
33
+GRANT SELECT ON dbo.sysjobs TO netdata_limited;
34
+GO
35
+
36
USE netdata;
37
GO
38
39
+IF NOT EXISTS (SELECT 1 FROM sys.database_principals WHERE name = 'netdata_limited')
40
+BEGIN
41
+ CREATE USER netdata_limited FOR LOGIN netdata_limited;
42
+END
43
+GO
44
+
45
+GRANT CONNECT TO netdata_limited;
46
+GO
47
+
48
IF OBJECT_ID('dbo.sample', 'U') IS NULL
49
BEGIN
50
CREATE TABLE dbo.sample (
@@ -20,6 +55,53 @@ BEGIN
55
END
56
GO
57
58
+IF OBJECT_ID('dbo.deadlock_a', 'U') IS NULL
59
+BEGIN
60
+ CREATE TABLE dbo.deadlock_a (
61
+ id INT PRIMARY KEY,
62
+ value INT NOT NULL
63
+ );
64
+END
65
+GO
66
+
67
+IF OBJECT_ID('dbo.deadlock_b', 'U') IS NULL
68
+BEGIN
69
+ CREATE TABLE dbo.deadlock_b (
70
+ id INT PRIMARY KEY,
71
+ value INT NOT NULL
72
+ );
73
+END
74
+GO
75
+
76
+IF NOT EXISTS (SELECT 1 FROM dbo.deadlock_a WHERE id = 1)
77
+BEGIN
78
+ INSERT INTO dbo.deadlock_a (id, value) VALUES (1, 10);
79
+END
80
+GO
81
+
82
+IF NOT EXISTS (SELECT 1 FROM dbo.deadlock_b WHERE id = 1)
83
+BEGIN
84
+ INSERT INTO dbo.deadlock_b (id, value) VALUES (1, 20);
85
+END
86
+GO
87
+
88
+-- Table for error category testing (constraint violations, data type errors).
89
+IF OBJECT_ID('dbo.error_test', 'U') IS NULL
90
+BEGIN
91
+ CREATE TABLE dbo.error_test (
92
+ id INT PRIMARY KEY,
93
+ unique_col NVARCHAR(64) NOT NULL UNIQUE,
94
+ int_col INT NOT NULL
95
+ );
96
+END
97
+GO
98
+
99
+IF NOT EXISTS (SELECT 1 FROM dbo.error_test WHERE id = 1)
100
+BEGIN
101
+ INSERT INTO dbo.error_test (id, unique_col, int_col) VALUES (1, 'existing_value', 100);
102
+END
103
+GO
104
+
105
INSERT INTO dbo.sample (name, value)
106
VALUES ('alpha', 10), ('beta', 20), ('gamma', 30);
107
GO
src/go/tools/functions-validation/seed/mysql/init.sql
+23
@@ -4,6 +4,29 @@ CREATE TABLE IF NOT EXISTS sample (
4
value INT NOT NULL
5
);
6
7
+-- Tables for deadlock induction tests.
8
+CREATE TABLE IF NOT EXISTS deadlock_a (
9
+ id INT PRIMARY KEY,
10
+ value INT NOT NULL
11
+) ENGINE=InnoDB;
12
+
13
+CREATE TABLE IF NOT EXISTS deadlock_b (
14
+ id INT PRIMARY KEY,
15
+ value INT NOT NULL
16
+) ENGINE=InnoDB;
17
+
18
+INSERT INTO deadlock_a (id, value) VALUES (1, 10);
19
+INSERT INTO deadlock_b (id, value) VALUES (1, 20);
20
+
21
+-- Table for error category testing (constraint violations, data type errors).
22
+CREATE TABLE IF NOT EXISTS error_test (
23
+ id INT PRIMARY KEY,
24
+ unique_col VARCHAR(64) UNIQUE NOT NULL,
25
+ int_col INT NOT NULL
26
+) ENGINE=InnoDB;
27
+
28
+INSERT INTO error_test (id, unique_col, int_col) VALUES (1, 'existing_value', 100);
29
+
30
-- Ensure statement digest collection is enabled.
31
UPDATE performance_schema.setup_consumers
32
SET ENABLED = 'YES'