| 1 | --- |
| 2 | title: AI Analyst — Analyst review workflow |
| 3 | description: Review and grade every AI investigation, correct IOC verdicts, teach the agent with durable or one-off lessons, replay with a different template, and track feedback trends over time. |
| 4 | --- |
| 5 | |
| 6 | Every report Talon produces is a **draft**. The review workflow lets a SOC analyst grade it, correct what's wrong, teach the agent with a lesson, and — if needed — replay the investigation with a different template. Feedback is aggregated per customer so you can see which templates are reliable and which need tuning. |
| 7 | |
| 8 | This page is for **operators** (analysts reviewing reports). For the architecture and deployment guide, see [AI Analyst (Talon)](/power-features/ai-analyst). |
| 9 | |
| 10 | --- |
| 11 | |
| 12 | ## Why review matters |
| 13 | |
| 14 | AI reports are fast, consistent, and cheap — but they're not infallible. Without a feedback loop you can't tell: |
| 15 | |
| 16 | - Whether the agent picked the right investigation template |
| 17 | - Whether IOC verdicts match reality (was that hash really malicious?) |
| 18 | - Whether the severity call was appropriate for your environment |
| 19 | - What recurring patterns the agent should treat as benign (and stop paging you about) |
| 20 | |
| 21 | The review workflow turns every investigation into a training signal. Lessons you capture land in **MemPalace** — the agent's persistent memory — and are surfaced on the next investigation for that customer. |
| 22 | |
| 23 | --- |
| 24 | |
| 25 | ## Where it lives in the UI |
| 26 | |
| 27 | Reviews and feedback live across two places: |
| 28 | |
| 29 | | Location | Purpose | |
| 30 | |----------|---------| |
| 31 | | **Incident Management → Alert → AI Analyst tab → Review** | Grade a specific report, correct IOCs, queue a lesson, replay | |
| 32 | | **AI Analyst page → Reports** | Browse all reports, jump into any to review | |
| 33 | | **AI Analyst page → Feedback** | Per-customer rollup: thumbs, ratings, template accuracy, IOC accuracy, recent reviews | |
| 34 | |
| 35 | --- |
| 36 | |
| 37 | ## Reviewing a report |
| 38 | |
| 39 | ### Open the report |
| 40 | |
| 41 | 1. **Incident Management → Alerts** → open any alert that has an AI investigation |
| 42 | 2. Click the **AI Analyst** tab (pulses if a report exists) |
| 43 | 3. Inside that tab, click **Review** |
| 44 | |
| 45 | You'll see a rubric if this is the first review, or your previous grades pre-filled if you've reviewed this report before — submitting again updates the existing review (one review per analyst per report). |
| 46 | |
| 47 | ### The rubric |
| 48 | |
| 49 | | Field | What it captures | |
| 50 | |-------|------------------| |
| 51 | | **Overall verdict** | Thumbs up / down — fast signal, shows up in the dashboard | |
| 52 | | **Template choice** | `correct` / `partial` / `wrong` — was the right investigation template picked? | |
| 53 | | **Rating: instructions** | 1–5 — did the agent follow the template's instructions? | |
| 54 | | **Rating: artifacts** | 1–5 — did it find and cite the right evidence from SIEM? | |
| 55 | | **Rating: severity** | 1–5 — did the severity assessment match reality? | |
| 56 | | **Missing steps** | Free-text — what should the agent have done but didn't? | |
| 57 | | **Suggested edits** | Free-text — specific rewrites or additions for the report | |
| 58 | |
| 59 | Leave any axis blank if you don't have a confident opinion — averages ignore nulls. |
| 60 | |
| 61 | ### IOC corrections |
| 62 | |
| 63 | Below the rubric you'll see the IOCs the agent extracted, each with its VirusTotal verdict. For every IOC you can mark: |
| 64 | |
| 65 | - **Verdict correct** ✓ — agent's verdict matches reality |
| 66 | - **Verdict wrong** ✗ — explain in the note field (e.g. "this IP is our jumphost, not malicious") |
| 67 | |
| 68 | IOC-level accuracy rolls up into the feedback dashboard separately from the overall rubric — useful for spotting when the agent trusts VirusTotal too much or too little for your environment. |
| 69 | |
| 70 | ### Submit |
| 71 | |
| 72 | Click **Submit review** (or **Update review** if you're editing). The review persists immediately — no pending state. |
| 73 | |
| 74 | --- |
| 75 | |
| 76 | ## Teach the palace |
| 77 | |
| 78 | The **Teach the palace** section under the rubric lets you add a lesson to the agent's persistent memory. Lessons get retrieved automatically at the start of every investigation for that customer, so the next time the agent sees a similar pattern it already has your context. |
| 79 | |
| 80 | ### When to add a lesson |
| 81 | |
| 82 | - **After a false positive** — "Host X runs nightly backups at 02:00 UTC; Sysmon 1 on robocopy during that window is benign" |
| 83 | - **After confirming threat intel** — "APT group Y targets customer; any outbound to IP range Z should be escalated" |
| 84 | - **Asset context** — "DC-01 is the primary domain controller; any unsigned binary execution there is critical" |
| 85 | - **Environment specifics** — "This customer uses piHole at 192.168.1.53; DNS traffic to that IP is expected" |
| 86 | |
| 87 | ### Lesson types (rooms) |
| 88 | |
| 89 | Lessons are filed into one of four "rooms" so the agent can retrieve them by context: |
| 90 | |
| 91 | | Room | Use for | |
| 92 | |------|---------| |
| 93 | | `environment` | Customer infra, network layout, scheduled jobs, expected traffic patterns | |
| 94 | | `false_positives` | Confirmed benign patterns that should stop paging the on-call | |
| 95 | | `assets` | Per-host context — role, owner, criticality, known-good processes | |
| 96 | | `threat_intel` | Campaigns, IOC blocklists, TTP notes specific to this customer | |
| 97 | |
| 98 | Pick the room that matches how you'd want to retrieve the lesson later. |
| 99 | |
| 100 | ### Durable vs one-off |
| 101 | |
| 102 | | Durability | TTL | Use for | |
| 103 | |------------|-----|---------| |
| 104 | | **Durable** | Never expires | Long-term truths — "DC-01 is the PDC", "customer uses Cloudflare" | |
| 105 | | **One-off** | 7 days | Temporary context — "maintenance window April 15–17", "incident IR-2025-0042 in progress" | |
| 106 | |
| 107 | One-off lessons are swept automatically after their TTL — CoPilot tracks the expiry and tells MemPalace to forget them. Keep the palace clean so retrieval stays relevant. |
| 108 | |
| 109 | ### Similar-lessons preview |
| 110 | |
| 111 | As you type a lesson, a debounced search runs against the palace and shows up to 5 already-stored lessons that overlap your draft. Use it to: |
| 112 | |
| 113 | - Avoid duplicating an existing lesson |
| 114 | - See what the agent already "knows" about this pattern |
| 115 | - Phrase the new lesson consistently with prior ones |
| 116 | |
| 117 | ### Submit the lesson |
| 118 | |
| 119 | Click **Queue lesson**. The lesson is persisted to CoPilot's database with `status=pending`. A background drainer (APScheduler) picks it up within ~30 seconds, POSTs to Talon, and flips the row to `status=ingested` with a `drawer_id` handle. After that, the agent will retrieve it on the next investigation for that customer. |
| 120 | |
| 121 | --- |
| 122 | |
| 123 | ## Replay with a different template |
| 124 | |
| 125 | If the agent picked the wrong template — or you want to try a different one — click **Replay** on the Review tab. |
| 126 | |
| 127 | 1. The modal lists all templates currently deployed in Talon's `groups/copilot/prompts/` directory |
| 128 | 2. Pick a template (e.g. `sysmon_event_1.txt`, `windows_defender.txt`) |
| 129 | 3. Click **Replay** |
| 130 | |
| 131 | Talon spins up a **brand-new investigation job** for the same alert with your chosen template forced. The original report is untouched — CoPilot now has two (or more) reports for the alert, and the **Compare** tab lets you view them side-by-side. |
| 132 | |
| 133 | Good use cases: |
| 134 | |
| 135 | - Agent ran the generic template when a specific one would've been better |
| 136 | - You want to see how a different template frames the same raw evidence |
| 137 | - A/B test a newly tuned template against the previous one |
| 138 | |
| 139 | --- |
| 140 | |
| 141 | ## Palace consolidation |
| 142 | |
| 143 | Over time the palace accumulates lessons. Some expire, some duplicate each other, some get stale. The **Consolidate lessons** button (Feedback tab → top right) opens a point-in-time digest for the selected customer: |
| 144 | |
| 145 | | Panel | What it shows | |
| 146 | |-------|---------------| |
| 147 | | **Summary tiles** | Total active, durable, one-off, near-duplicate pair count | |
| 148 | | **Expiring soon** | One-off lessons within 2 days of expiry — act now or let them lapse | |
| 149 | | **Near-duplicate candidates** | Lesson pairs above 70% similarity — merge or delete | |
| 150 | | **By room** | Full lesson list grouped by room, with durability + status badges | |
| 151 | |
| 152 | Click **Copy markdown** to paste the digest into a ticket, a team channel, or your own knowledge base — useful for monthly palace reviews. |
| 153 | |
| 154 | This is read-only — you can't edit lessons from the drawer. To remove a lesson, either wait for the one-off TTL or mark the row manually via the database / API. |
| 155 | |
| 156 | --- |
| 157 | |
| 158 | ## Feedback dashboard |
| 159 | |
| 160 | **AI Analyst page → Feedback tab.** Pick a customer and see: |
| 161 | |
| 162 | ### Tiles |
| 163 | |
| 164 | - **Total reviews** — how much feedback you have for this customer |
| 165 | - **Thumbs up %** — overall sentiment |
| 166 | - **IOC verdict accuracy %** — of all IOC corrections submitted, how often did the agent agree with the analyst |
| 167 | - **Avg rating (overall)** — composite of instructions / artifacts / severity (nulls excluded) |
| 168 | |
| 169 | ### Template choice distribution |
| 170 | |
| 171 | Stacked bar — `correct` / `partial` / `wrong`. If the "wrong" bar is non-trivial, your agent is mis-selecting templates. Candidates for fixing: |
| 172 | |
| 173 | - The template detection logic (`rule.groups` matching in Talon) |
| 174 | - Adding a more specific template for the miss case |
| 175 | |
| 176 | ### Per-template performance |
| 177 | |
| 178 | Table showing per-template counts and averages. Use it to spot: |
| 179 | |
| 180 | - Templates with consistently low `instructions` ratings → the template itself may be wrong |
| 181 | - Templates with high `template_choice=wrong` for a rule type → detection rule is mis-classified |
| 182 | - Templates with low IOC accuracy → the template's enrichment steps may be flawed |
| 183 | |
| 184 | ### Recent reviews |
| 185 | |
| 186 | Last 10 reviews with drill-in. Click any one to open a drawer with the full rubric, IOC corrections, and free-text fields. |
| 187 | |
| 188 | --- |
| 189 | |
| 190 | ## Typical workflows |
| 191 | |
| 192 | ### Fast triage (10 seconds) |
| 193 | |
| 194 | 1. Open alert → AI Analyst tab → skim report |
| 195 | 2. If report matches reality → thumbs up, submit |
| 196 | 3. If obviously wrong → thumbs down, one-line in "Missing steps", submit |
| 197 | |
| 198 | ### False-positive capture (30 seconds) |
| 199 | |
| 200 | 1. Confirm the alert is benign (e.g. scheduled job, known-good process) |
| 201 | 2. Review tab → thumbs down → template choice `correct` (template was right, signal was noise) |
| 202 | 3. **Teach the palace** → room `false_positives`, durable, describe the benign pattern with enough detail that the agent would recognize it next time |
| 203 | 4. Queue lesson → submit review |
| 204 | |
| 205 | ### Template tuning (2 minutes) |
| 206 | |
| 207 | 1. Report picked a bad template → review it, template choice `wrong` |
| 208 | 2. Note in "Suggested edits" what template *should* have been used |
| 209 | 3. Click **Replay** → select the correct template → submit |
| 210 | 4. Compare tab → confirm the new report is better |
| 211 | 5. Report the mis-selection pattern to whoever maintains Talon's template detection |
| 212 | |
| 213 | ### Monthly palace review (10 minutes) |
| 214 | |
| 215 | 1. Feedback tab → pick customer → **Consolidate lessons** |
| 216 | 2. Review **Expiring soon** — promote anything still valid from one-off to a fresh durable lesson |
| 217 | 3. Review **Near-duplicates** — pick the better-worded lesson, manually delete the other |
| 218 | 4. **Copy markdown** → paste into your team's wiki for the customer |
| 219 | |
| 220 | --- |
| 221 | |
| 222 | ## Safety & guardrails |
| 223 | |
| 224 | - **Reviews are analyst-scoped** — one review per analyst per report, updates overwrite. Multiple analysts can each leave their own review. |
| 225 | - **Lessons are customer-scoped** — a lesson queued for customer `00001` is only retrieved on investigations for that customer. |
| 226 | - **One-off lessons auto-expire** — use them for temporary context so the palace stays clean. |
| 227 | - **Replays don't mutate the original** — every replay is a new job/report; the original stays for comparison. |
| 228 | - **Palace consolidation is read-only** — you can't accidentally delete the palace from the UI. |
| 229 | - **Don't put secrets in lessons** — lesson text is sent to Talon and embedded by MemPalace / ChromaDB. Treat it as you would a SIEM comment. |
| 230 | |
| 231 | --- |
| 232 | |
| 233 | ## Troubleshooting |
| 234 | |
| 235 | | Symptom | Likely cause | Fix | |
| 236 | |---------|--------------|-----| |
| 237 | | "Review" tab missing | Report doesn't exist yet | Click **Investigate with AI Analyst** on the alert Overview tab first | |
| 238 | | Lesson stays `pending` forever | Drainer job not running | Check CoPilot scheduler logs for `invoke_palace_lesson_drainer` | |
| 239 | | Lesson ingested but not retrieved on next investigation | Customer code mismatch, or wrong room | Verify lesson's `customer_code` matches the alert's; check palace search with the expected query | |
| 240 | | Replay modal shows no templates | Talon unreachable | Check `GET /api/talon/templates` — should list `.txt` files from `groups/copilot/prompts/` | |
| 241 | | Feedback dashboard shows zero reviews | No reviews submitted yet, or wrong customer picked | Submit at least one review, confirm the customer dropdown matches the alert's code | |
| 242 | | IOC accuracy shows `0/0` | No IOC corrections submitted | Review individual IOCs on the Review tab, not just the overall rubric | |
| 243 | |
| 244 | --- |
| 245 | |
| 246 | ## Video context |
| 247 | |
| 248 | - AI analyst (alert-context + exclusion-rule assistance): https://www.youtube.com/watch?v=-2srPC-Dw-0 |