Troubleshooting: add troubleshoot and custom investigations docs (#20759)
Co-authored-by: ilyam8 <ilya@netdata.cloud>
Kanela committed
Aug 25, 2025 at 15:22 UTC
3fd8307b1c7723bc008a1a91a58beda3927bb2a4
2 files changed
+196
docs/troubleshooting/custom-investigations.md
new
+119
@@ -0,0 +1,119 @@
1
+# Custom Investigations
2
+
3
+## Overview
4
+
5
+Custom Investigations let you ask open-ended questions about your infrastructure and receive deeply researched reports powered by AI. Unlike traditional dashboards or query languages, this conversational interface analyzes your real-time, high-fidelity data to answer complex operational questions in minutes.
6
+
7
+### When to Use Custom Investigations
8
+
9
+Create investigations for any scenario where you need deep analysis:
10
+
11
+- **Troubleshoot complex issues** - Delegate multiple parallel investigations during incidents
12
+- **Analyze deployment impact** - Understand how new releases affect performance
13
+- **Optimize costs** - Identify underutilized resources and quantify savings
14
+- **Explore trends** - Get summaries of system behavior changes over time
15
+
16
+### Creating Effective Investigations
17
+
18
+The key to powerful investigations is providing context. Think of it like briefing a teammate—the more details you share, the better the analysis.
19
+
20
+#### Example 1: Troubleshooting Service Failures
21
+
22
+**Your Request:**
23
+
24
+```
25
+Why are my checkout-service pods crashing repeatedly?
26
+```
27
+
28
+**Your Context:**
29
+
30
+```
31
+- Started after: deployment at 14:00 UTC of version 2.3.1
32
+- Impact: Customer checkout failures, lost revenue ~$X/hour
33
+- Recent changes: Updated payment gateway integration, increased worker threads from 10 to 20
34
+- Error pattern in logs: "connection refused to payment-service:8080", "Java heap space"
35
+- Environment: production / eks-prod-us-east-1
36
+- Related services: payment-service, inventory-service, redis-session-store
37
+```
38
+
39
+#### Example 2: Analyzing Deployment Changes
40
+
41
+**Your Request:**
42
+
43
+```
44
+Compare system metrics before and after the recent user-authentication-service deployment.
45
+```
46
+
47
+**Your Context:**
48
+
49
+```
50
+- Service: user-authentication-service v2.2.0
51
+- Deployed: 2025-01-24 09:00 UTC
52
+- Changes: Switched from JWT to Redis sessions, added Argon2 password hashing
53
+- Specific concerns: Users reporting intermittent logouts, suspicious increase in redis_connected_clients
54
+- Time windows: 24h before deployment vs 24h after
55
+```
56
+
57
+#### Example 3: Cost Optimization
58
+
59
+**Your Request:**
60
+
61
+```
62
+Identify underutilized nodes for cost optimization.
63
+```
64
+
65
+**Your Context:**
66
+
67
+```
68
+- Monthly AWS bill: $12K for compute
69
+- Environment: Mixed workloads (prod + staging on same cluster)
70
+- Known issues: Dev environments run 24/7, batch processing nodes idle 20h/day
71
+- Goal: Find $2-3K/month in savings without impacting reliability
72
+```
73
+
74
+### Starting a Custom Investigation
75
+
76
+You can create investigations in two ways:
77
+
78
+#### From the Insights Tab
79
+
80
+1. Navigate to the **Insights** tab
81
+2. Click **"New Investigation"**
82
+3. Enter your question and context
83
+
84
+#### From Anywhere with "Troubleshoot with AI"
85
+
86
+Click the **"Troubleshoot with AI"** button in the top right corner from any screen. This automatically captures your current context—including the specific chart, dashboard, or service you're viewing. Add your question and any extra context, then start the investigation.
87
+
88
+[SCREENSHOT FROM FIRST BLOG POST SHOULD BE PLACED HERE - showing the Troubleshoot with AI button]
89
+
90
+### Getting Your Results
91
+
92
+- Reports generate in approximately 2 minutes
93
+- View completed reports in the **Insights** tab
94
+- Receive email notifications when reports are ready
95
+
96
+[SCREENSHOT FROM FIRST BLOG POST SHOULD BE PLACED HERE - showing the Insights tab interface]
97
+
98
+### Best Practices
99
+
100
+1. **Be specific** - Include timeframes, service names, and environments
101
+2. **Add context** - Paste relevant details from tickets, Slack threads, or deployment logs
102
+3. **Set clear goals** - Specify what you're trying to achieve (reduce costs, find root cause, etc.)
103
+4. **Use parallel investigations** - Run multiple investigations simultaneously during incidents
104
+
105
+### Access and Availability
106
+
107
+This feature is available in preview mode for:
108
+
109
+- All Business and Homelab plan users
110
+- New users get 10 AI investigation sessions per month during their Business plan trial
111
+- Community users can request access by contacting product@netdata.cloud
112
+
113
+### Coming Soon
114
+
115
+We're actively developing:
116
+
117
+- Scheduled recurring investigations for regular reports
118
+- Custom SLO report templates
119
+- Weekly cost-optimization analyses
docs/troubleshooting/troubleshoot.md
new
+77
@@ -0,0 +1,77 @@
1
+# AI-Powered Alert Troubleshooting
2
+
3
+## Overview
4
+
5
+When an alert fires, you can use AI to generate a detailed troubleshooting report that analyzes whether the alert requires immediate action or is just noise. The AI examines your alert's history, correlates it with thousands of other metrics across your infrastructure, and provides actionable insights—all within minutes.
6
+
7
+### Key Benefits
8
+
9
+- **Save hours of manual investigation** - Skip the initial data collection and correlation work
10
+- **Reduce alert fatigue** - Quickly identify false positives versus legitimate issues
11
+- **Get actionable guidance** - Receive specific next steps based on the analysis
12
+- **Start from insight, not zero** - Begin troubleshooting with a comprehensive baseline analysis
13
+
14
+### How It Works
15
+
16
+The AI troubleshooting engine performs three key analyses when you trigger an investigation:
17
+
18
+1. **Alert Analysis** - Examines the alert's history and underlying metric behavior to determine if it's a transient false positive or legitimate issue
19
+2. **Correlation Discovery** - Scans thousands of metrics and log patterns across your infrastructure to identify what else was behaving abnormally at the same time
20
+3. **Root Cause Hypothesis** - Provides a summary of findings and suggests likely root causes, pointing you to the most relevant metrics or dimensions
21
+
22
+### Starting an Alert Investigation
23
+
24
+You can trigger AI troubleshooting in three ways:
25
+
26
+#### From the Alerts Tab
27
+
28
+Click the **"Ask AI"** button next to any active or recent alert.
29
+
30
+<!--
31
+
32
+
33
+
34
+-->
35
+
36
+#### From the Insights Tab
37
+
38
+1. Navigate to the **Insights** tab
39
+2. Select **"Alert Troubleshooting"** from the investigations section
40
+3. Choose any recent alert from the dropdown menu
41
+
42
+<!--
43
+
44
+
45
+
46
+-->
47
+
48
+#### From Alert Notifications
49
+
50
+When you receive an alert email, click the **"Troubleshoot with AI"** link to automatically start the investigation.
51
+
52
+<!--
53
+
54
+
55
+
56
+-->
57
+
58
+### Understanding Your Report
59
+
60
+Reports typically generate in 1-2 minutes. Once complete:
61
+
62
+- The report appears in your **Alerts** tab
63
+- A copy is saved in the **Insights** tab under "Investigations"
64
+- You receive an email notification with the analysis summary
65
+
66
+### Access and Availability
67
+
68
+This feature is available in preview mode for:
69
+
70
+- All Business and Homelab plan users
71
+- New users get 10 AI troubleshooting sessions per month during their Business plan trial
72
+
73
+:::note
74
+
75
+Community users can request access by contacting product@netdata.cloud
76
+
77
+:::