| 1 | # Custom Investigations |
| 2 | |
| 3 | ## Overview |
| 4 | |
| 5 | Custom Investigations let you ask open-ended questions about your infrastructure and receive deeply researched reports powered by AI. Unlike traditional dashboards or query languages, this conversational interface analyzes your real-time, high-fidelity data to answer complex operational questions in minutes. |
| 6 | |
| 7 | ### When to Use Custom Investigations |
| 8 | |
| 9 | Create investigations for any scenario where you need deep analysis: |
| 10 | |
| 11 | - **Troubleshoot complex issues** - Delegate multiple parallel investigations during incidents |
| 12 | - **Analyze deployment impact** - Understand how new releases affect performance |
| 13 | - **Optimize costs** - Identify underutilized resources and quantify savings |
| 14 | - **Explore trends** - Get summaries of system behavior changes over time |
| 15 | |
| 16 | ### Creating Effective Investigations |
| 17 | |
| 18 | The key to powerful investigations is providing context. Think of it like briefing a teammate—the more details you share, the better the analysis. |
| 19 | |
| 20 | #### Example 1: Troubleshooting Service Failures |
| 21 | |
| 22 | **Your Request:** |
| 23 | |
| 24 | ``` |
| 25 | Why are my checkout-service pods crashing repeatedly? |
| 26 | ``` |
| 27 | |
| 28 | **Your Context:** |
| 29 | |
| 30 | ``` |
| 31 | - Started after: deployment at 14:00 UTC of version 2.3.1 |
| 32 | - Impact: Customer checkout failures, lost revenue ~$X/hour |
| 33 | - Recent changes: Updated payment gateway integration, increased worker threads from 10 to 20 |
| 34 | - Error pattern in logs: "connection refused to payment-service:8080", "Java heap space" |
| 35 | - Environment: production / eks-prod-us-east-1 |
| 36 | - Related services: payment-service, inventory-service, redis-session-store |
| 37 | ``` |
| 38 | |
| 39 | #### Example 2: Analyzing Deployment Changes |
| 40 | |
| 41 | **Your Request:** |
| 42 | |
| 43 | ``` |
| 44 | Compare system metrics before and after the recent user-authentication-service deployment. |
| 45 | ``` |
| 46 | |
| 47 | **Your Context:** |
| 48 | |
| 49 | ``` |
| 50 | - Service: user-authentication-service v2.2.0 |
| 51 | - Deployed: 2025-01-24 09:00 UTC |
| 52 | - Changes: Switched from JWT to Redis sessions, added Argon2 password hashing |
| 53 | - Specific concerns: Users reporting intermittent logouts, suspicious increase in redis_connected_clients |
| 54 | - Time windows: 24h before deployment vs 24h after |
| 55 | ``` |
| 56 | |
| 57 | #### Example 3: Cost Optimization |
| 58 | |
| 59 | **Your Request:** |
| 60 | |
| 61 | ``` |
| 62 | Identify underutilized nodes for cost optimization. |
| 63 | ``` |
| 64 | |
| 65 | **Your Context:** |
| 66 | |
| 67 | ``` |
| 68 | - Monthly AWS bill: $12K for compute |
| 69 | - Environment: Mixed workloads (prod + staging on same cluster) |
| 70 | - Known issues: Dev environments run 24/7, batch processing nodes idle 20h/day |
| 71 | - Goal: Find $2-3K/month in savings without impacting reliability |
| 72 | ``` |
| 73 | |
| 74 | ### Starting a Custom Investigation |
| 75 | |
| 76 | You can create investigations in two ways: |
| 77 | |
| 78 | #### From the Insights Tab |
| 79 | |
| 80 | 1. Navigate to the **Insights** tab |
| 81 | 2. Click **"New Investigation"** |
| 82 | 3. Enter your question and context |
| 83 | |
| 84 | #### From Any Alert |
| 85 | |
| 86 | Click **"Ask AI"** next to any alert, or use the **"Alert Troubleshooting"** option in the Insights tab. This automatically captures your current context—including the specific alert, timeframe, and affected services. Add your question and any extra context, then start the investigation. |
| 87 | |
| 88 | ### Getting Your Results |
| 89 | |
| 90 | - Reports generate in approximately 2 minutes |
| 91 | - View completed reports in the **Insights** tab |
| 92 | - Receive email notifications when reports are ready |
| 93 | |
| 94 | [SCREENSHOT FROM FIRST BLOG POST SHOULD BE PLACED HERE - showing the Insights tab interface] |
| 95 | |
| 96 | ### Best Practices |
| 97 | |
| 98 | 1. **Be specific** - Include timeframes, service names, and environments |
| 99 | 2. **Add context** - Paste relevant details from tickets, Slack threads, or deployment logs |
| 100 | 3. **Set clear goals** - Specify what you're trying to achieve (reduce costs, find root cause, etc.) |
| 101 | 4. **Use parallel investigations** - Run multiple investigations simultaneously during incidents |
| 102 | |
| 103 | ### Access and Availability |
| 104 | |
| 105 | - Generally available in Netdata Cloud (Business and Free Trial) |
| 106 | - Eligible Spaces receive 10 free AI runs per month; additional usage via AI Credits |
| 107 | |
| 108 | :::note |
| 109 | Track AI credit usage from `Settings → Usage & Billing → AI Credits`. |
| 110 | ::: |
| 111 | |
| 112 | ### Scheduling |
| 113 | |
| 114 | You can schedule recurring investigations from the `Insights` tab (daily/weekly/monthly). Use this to automate weekly health checks, monthly optimization reviews, or SLO conformance reports. |