@cryptotaxi247 / netdata-1
Code Commits Issues Pulls Sessions CI/CD
master
master
md 300 lines 9.16 KB
Rendered Raw
1 # Metric Correlations
2
3 The **Metric Correlations** feature helps you quickly identify metrics and charts relevant to a specific time window of interest, allowing for faster root cause analysis.
4
5 :::tip
6
7 By filtering your standard Netdata dashboard to **display only the most relevant charts**, Metric Correlations make it easier for you to pinpoint anomalies and investigate issues.
8
9 :::
10
11 Since it leverages every available metric in your infrastructure with up to 1-second granularity, **Metric Correlations provides you with highly accurate insights**.
12
13 ## Using Metric Correlations
14
15 When viewing the [Metrics tab or a single-node dashboard](/docs/dashboards-and-charts/metrics-tab-and-single-node-tabs.md), you'll find the **Metric Correlations** button in the top-right corner.
16
17 <details>
18 <summary><strong>To start:</strong></summary><br/>
19
20 1. Click **Metric Correlations**.
21 2. Highlight a selection of metrics on a single chart. **The selected timeframe must be at least 15 seconds**.
22 3. The menu displays details about your selected area and reference baseline. Metric Correlations compares your highlighted window to a reference baseline, which is four times its length and precedes it immediately.
23 4. Click **Find Correlations**.
24
25 :::note
26
27 This button is only active if you've selected a valid timeframe.
28
29 :::
30
31 5. **The process evaluates all your available metrics and returns a filtered Netdata dashboard** showing only the most changed metrics between the baseline and your highlighted window.
32 6. If needed, select another window and press **Find Correlations** again to refine your analysis.
33
34 </details>
35
36 ## Integration with Anomaly Detection
37
38 You can combine Metric Correlations with Anomaly Detection for powerful troubleshooting:
39
40 :::tip
41
42 When you notice an anomaly in your system, use Metric Correlations with the **Anomaly Rate** data type to quickly identify which metrics are contributing to the anomalous behavior.
43
44 :::
45
46 ### How to Use Together
47
48 ```mermaid
49 flowchart LR
50 A("Detection<br/>Spot anomaly spike<br/>in node rate chart")
51 B("Selection<br/>Highlight specific<br/>time period")
52 C("Configuration<br/>Select Anomaly Rate<br/>as data type")
53 D("Execute<br/>Click Find<br/>Correlations")
54 E("Analysis<br/>Review metrics with<br/>highest anomaly rates")
55 F("Resolution<br/>Examine metrics to<br/>determine root cause")
56
57 A --> B --> C --> D --> E --> F
58
59 %% Style definitions
60 classDef alert fill:#ffeb3b,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
61 classDef neutral fill:#f9f9f9,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
62 classDef complete fill:#4caf50,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
63 classDef database fill:#2196F3,stroke:#000000,stroke-width:3px,color:#000000,font-size:14px
64
65 %% Apply styles
66 class A,B alert
67 class C,D neutral
68 class E complete
69 class F database
70 ```
71
72 :::tip
73
74 **This workflow helps you move from detecting** that *"something is wrong"* **to understanding** exactly which components are behaving abnormally, significantly reducing your troubleshooting time.
75
76 :::
77
78 ## API Access
79
80 You can access anomaly detection data and use it with metric correlations through Netdata's API:
81
82 <details>
83 <summary><strong>Querying Anomaly Bits</strong></summary><br/>
84
85 To get the anomaly bits for any metric, add the `options=anomaly-bit` parameter to your API query:
86
87 ```
88 https://your-netdata-node/api/v1/data?chart=system.cpu&dimensions=user&after=-60&options=anomaly-bit
89 ```
90
91 Sample response:
92
93 ```json
94 {
95 "labels": [
96 "time",
97 "user"
98 ],
99 "data": [
100 [
101 1684852570,
102 0
103 ],
104 [
105 1684852569,
106 0
107 ],
108 [
109 1684852568,
110 0
111 ],
112 [
113 1684852567,
114 0
115 ],
116 [
117 1684852566,
118 0
119 ],
120 [
121 1684852565,
122 0
123 ],
124 [
125 1684852564,
126 0
127 ],
128 [
129 1684852563,
130 0
131 ],
132 [
133 1684852562,
134 0
135 ],
136 [
137 1684852561,
138 0
139 ]
140 ]
141 }
142 ```
143
144 </details>
145
146 <details>
147 <summary><strong>Querying Anomaly Rates</strong></summary><br/>
148
149 For anomaly rates over a time window, use the same parameter but with aggregated data:
150
151 ```
152 https://your-netdata-node/api/v1/data?chart=system.cpu&dimensions=user&after=-600&before=0&points=10&options=anomaly-bit
153 ```
154
155 Sample response showing the percentage of time each metric was anomalous:
156
157 ```json
158 {
159 "labels": [
160 "time",
161 "user"
162 ],
163 "data": [
164 [
165 1684852770,
166 0
167 ],
168 [
169 1684852710,
170 20
171 ],
172 [
173 1684852650,
174 0
175 ],
176 [
177 1684852590,
178 10
179 ],
180 [
181 1684852530,
182 0
183 ],
184 [
185 1684852470,
186 0
187 ],
188 [
189 1684852410,
190 30
191 ],
192 [
193 1684852350,
194 0
195 ],
196 [
197 1684852290,
198 0
199 ],
200 [
201 1684852230,
202 0
203 ]
204 ]
205 }
206 ```
207
208 </details>
209
210 :::tip
211
212 You can programmatically access this data to build custom dashboards or alerts based on anomaly patterns in your infrastructure.
213
214 :::
215
216 ## Metric Correlations Options
217
218 Metric Correlations offer adjustable parameters for deeper data exploration. Since different data types and incidents require different approaches, **these settings allow for flexible analysis**.
219
220 <details>
221 <summary><strong>Method</strong></summary><br/>
222
223 Two algorithms are available for scoring metrics based on changes between the baseline and highlight windows:
224
225 * **`KS2` (Kolmogorov-Smirnov Test)**: A statistical method comparing distributions between the highlighted and baseline windows to detect significant changes. [Implementation details](https://github.com/netdata/netdata/blob/d917f9831c0a1638ef4a56580f321eb6c9a88037/database/metric_correlations.c#L212).
226 * **`Volume`**: A heuristic approach based on percentage change in averages, designed to handle edge cases. [Implementation details](https://github.com/netdata/netdata/blob/d917f9831c0a1638ef4a56580f321eb6c9a88037/database/metric_correlations.c#L516).
227
228 </details>
229
230 <details>
231 <summary><strong>Aggregation</strong></summary><br/>
232
233 To accommodate different window lengths, Netdata aggregates your raw data as needed. The default aggregation method is `Average`, but you can also choose `Median`, `Min`, `Max`, or `Stddev`.
234 </details>
235
236 <details>
237 <summary><strong>Data Type</strong></summary><br/>
238
239 Netdata assigns an [Anomaly Bit](https://github.com/netdata/netdata/tree/master/src/ml#anomaly-bit) to each of your metrics in real-time, flagging whether it deviates significantly from normal behavior. You can analyze either raw data or anomaly rates:
240
241 * **`Metrics`**: Runs Metric Correlations on your raw metric values.
242 * **`Anomaly Rate`**: Runs Metric Correlations on anomaly rates for each of your metrics.
243
244 </details>
245
246 ## Metric Correlations on the Agent
247
248 Metric Correlations (MC) requests to Netdata Cloud are handled in two ways:
249
250 1. **If MC is enabled** on any of your nodes, the request is routed to the highest-level node (a Parent node or the node itself).
251 2. **If MC is not enabled** on any of your nodes, Netdata Cloud processes the request by collecting data from your nodes and computing correlations on its backend.
252
253 ## Interpreting Combined Results
254
255 When you use Metric Correlations together with Anomaly Detection, you'll want to understand how to interpret the results:
256
257 :::tip
258
259 **High anomaly rates combined with significant metric changes** often indicate genuine issues rather than false positives.
260
261 :::
262
263 Here's how to interpret different scenarios:
264
265 | Anomaly Rate | Metric Correlation | Interpretation |
266 |--------------|--------------------|------------------------------------------------------|
267 | High | Strong | Likely a significant issue affecting system behavior |
268 | High | Weak | Possible edge case or intermittent issue |
269 | Low | Strong | Normal but significant change in system behavior |
270 | Low | Weak | Likely normal system operation |
271
272 :::tip
273
274 By examining both the anomaly rate and the correlation strength, you can prioritize your troubleshooting efforts more effectively.
275
276 :::
277
278 ## Usage Tips
279
280 :::tip
281
282 When running Metric Correlations from the [Metrics tab](/docs/dashboards-and-charts/metrics-tab-and-single-node-tabs.md) across multiple nodes, refine your results by grouping by node:
283
284 1. Run MC on all your nodes if you're unsure which ones are relevant.
285 2. Group the most interesting charts by node to determine whether changes affect all your nodes or just a subset.
286 3. If a subset of your nodes stands out, filter for those nodes and rerun MC to get more precise results.
287
288 Choose the **`Volume`** algorithm for sparse metrics (e.g., request latency with few requests). Otherwise, use **`KS2`**.
289
290 - **`KS2`** is ideal for detecting complex distribution changes in your metrics, such as shifts in variance.
291 - **`Volume`** is better for detecting your metrics that were inactive and then spiked (or vice versa).
292
293 **Example:**
294
295 - `Volume` can highlight network traffic suddenly turning on in your system.
296 - `KS2` can detect entropy distribution changes in your data missed by `Volume`.
297
298 Combine **`Volume`** and **`Anomaly Rate`** to identify the most anomalous metrics within your selected timeframe. Expand the anomaly rate chart to visualize results more clearly.
299
300 :::