investigation-playbooks.md · cryptotaxi247/netdata

Code Commits Issues Pulls Sessions CI/CD

master

md 187 lines 10 KB

Rendered Raw

1	<!--startmeta
2	custom_edit_url: "https://github.com/netdata/netdata/edit/master/docs/network-flows/investigation-playbooks.md"
3	sidebar_label: "Investigation Playbooks"
4	learn_status: "Published"
5	learn_rel_path: "Network Flows"
6	keywords: ['playbooks', 'investigation', 'workflows', 'troubleshooting traffic']
7	endmeta-->
8
9	<!-- markdownlint-disable-file -->
10
11	# Investigation playbooks
12
13	Step-by-step recipes for common questions, all using the Netdata Network Flows view (open the Live tab and select Network Flows). Each playbook fits in a 5-15 minute investigation window.
14
15	## Playbook 1 — "The link is saturated, who's responsible?"
16
17	The situation. SNMP shows your Internet link at 95% utilisation. Users complain about slowness.
18
19	The goal. Identify the talker(s) consuming the bandwidth.
20
21	Steps.
22
23	1. Open Network Flows with the default view (Sankey + Table). Set the time range to the last 15 minutes — recent enough to be live, wide enough to smooth bursts.
24
25	2. Filter to the saturated interface. In the filter ribbon, set:
26	- `Exporter Name` = the router with the saturated link
27	- `Egress Interface Name` = the interface name (or `Ingress Interface Name` if you want incoming traffic)
28
29	This eliminates the doubling effect and shows only one direction.
30
31	3. Change the aggregation to "who's responsible". Click the group-by selector and change the fields to:
32	- `Source AS Name` → `Destination AS Name` (for an Internet-edge link)
33
34	Or for an internal link:
35	- `Source IP` → `Destination IP`
36
37	4. Read the Sankey. The widest band is your top talker pair. Click on the wide band to drill in.
38
39	5. If a single ASN/IP dominates — that's your answer. Click the value to add it as a filter and look at the table for the specific 5-tuple details.
40
41	6. If traffic is evenly distributed across many sources — the problem is aggregate demand, not a single offender. The link genuinely needs more capacity, or you need traffic shaping. Move to Playbook 3.
42
43	What to record.
44
45	- Timestamp range of the investigation
46	- Top 3 talker pairs and their byte volumes
47	- Whether this is a one-time spike or sustained
48	- The URL of the dashboard view (preserves all filters and aggregation)
49
50	Common findings.
51
52	- Backup software running during business hours.
53	- A SaaS sync or cloud upload.
54	- Misconfigured automation (e.g., logs being shipped to the wrong place).
55	- New user / new application doing something unexpected.
56
57	## Playbook 2 — "Investigating a specific IP"
58
59	The situation. A security alert references an IP address. You need to know what it talked to, when, and how much.
60
61	The goal. Construct a timeline and traffic profile for that IP.
62
63	Steps.
64
65	1. Open Network Flows with the last 24 hours as the time range.
66
67	2. Filter by the IP. In the filter ribbon:
68	- For inbound investigation: `Destination IP` = the IP
69	- For outbound investigation: `Source IP` = the IP
70	- For both directions: filter both, separately, in two browser tabs
71
72	IP filtering forces raw tier (raw retention) — the time depth is bounded by your raw-tier retention. If you need to look further back than that, the data isn't there.
73
74	3. Switch to Time-Series view. This shows when the IP was active. Look for:
75	- When did activity start? End?
76	- Is it constant, periodic, or bursty?
77	- Does it correlate with a known event (deployment, business hours, maintenance window)?
78
79	4. Switch back to Sankey + Table. Change the group-by to surface the relevant context:
80	- `Destination IP` → `Destination Port` → `Destination Country` (if the suspect is a source)
81	- `Source IP` → `Source ASN` (if the suspect is a destination)
82
83	5. Read the table. The top rows show the IP's most-talked-to peers, ranked by bytes. Look for:
84	- Unknown external IPs in unexpected geographies.
85	- Connections on unusual ports (anything not in your normal protocol mix).
86	- Sustained outbound transfers (potential exfiltration) vs short bursts (likely normal).
87
88	6. For each suspicious peer, drill in. Add the peer IP to the filter ribbon, switch back to Time-Series. Confirm the timeline aligns with the original alert.
89
90	What to record.
91
92	- Time range of all activity by the IP
93	- Top destinations and their byte/packet counts
94	- Whether the activity is consistent with a legitimate use (backup, SaaS sync) or anomalous
95	- The URL of each dashboard view used in the investigation
96
97	Caveats.
98
99	- IP filter forces raw tier; older data may not be available.
100	- If the IP is internal and you haven't declared it under `enrichment.networks`, GeoIP may misrepresent its country.
101	- If the IP is a NAT public address, multiple internal hosts may be hidden behind it. Cross-check with NAT translation logs.
102
103	## Playbook 3 — "Justifying a link upgrade"
104
105	The situation. A WAN circuit is at 80% utilisation during peak hours. Finance wants justification before approving an upgrade.
106
107	The goal. Produce a defensible trend showing growth and projecting the date of saturation.
108
109	Steps.
110
111	1. Open Network Flows with the last 30 days as the time range. (Adjust based on tier-1/5/60 retention. If your retention is shorter, use whatever you have.)
112
113	2. Filter to the WAN interface. Set `Exporter Name` and `Egress Interface Name` (or `Ingress Interface Name` — pick one direction). This removes the doubling effect.
114
115	3. Switch to Time-Series view. The chart now shows ~30 days of bandwidth on the link. The bucket size auto-adjusts to roughly 1 hour at this range.
116
117	4. Identify the trend. Look at the daily peaks (one curve cycle = one day). The peak should be growing month-over-month. Eyeball the slope.
118
119	5. Identify the growth driver. Switch back to Sankey + Table, group by `Destination ASN` or `Destination Port` (service). Compare top consumers from the start of the period to the end. New entries that weren't there 30 days ago are growth drivers.
120
121	6. Compute the upgrade need. Take the current peak (e.g., 80% of 100 Mbps = 80 Mbps), project forward at the observed monthly growth rate (e.g., 10%/month = ~30%/quarter), and find when it crosses 100% (or 70% if you want headroom).
122
123	Example: if peak grows from 70 Mbps to 80 Mbps over 30 days, that's roughly 14% monthly growth. At that rate it crosses 100 Mbps in ~2 months and 200 Mbps would buy you ~1 year.
124
125	What to record.
126
127	- Trend chart (screenshot or shareable URL)
128	- Growth driver: the specific applications / services consuming the new bandwidth
129	- Projected saturation date and recommended upgrade timeline
130	Caveats.
131
132	- A large spike one day shouldn't drive the projection. Use weekly peaks (averaged across same-day-of-week) for stability.
133	- If your retention is shorter than 30 days, use what you have but caveat the projection.
134
135	## Playbook 4 — "Scoping a security alert"
136
137	The situation. Your IDS / EDR / SIEM fired an alert: an internal host communicated with a known-malicious external IP. You have the internal IP, the external IP, and a rough time window.
138
139	The goal. Determine the scope and timeline. Did other internal hosts talk to the same external IP? When did it start? How much data was exchanged?
140
141	Steps.
142
143	1. Open Network Flows with the time range covering 24 hours before the alert through now.
144
145	2. Filter by the external IP. In the filter ribbon: `Destination IP` = the external IP.
146
147	This forces raw tier. Time depth is your raw-tier retention.
148
149	3. Switch to Time-Series view. When did communication start? Is it ongoing? Did it correlate with the alert time?
150
151	4. Switch to Sankey + Table. Group by `Source IP`. The result is "every internal IP that talked to this external IP, ranked by bytes".
152
153	- If only one internal host appears, scope is contained.
154	- If multiple appear, you have a broader scope. Investigate each.
155
156	5. For the alerted internal host, swap the filter: `Source IP` = the internal host (remove the external filter). Group by `Destination IP` → `Destination Country` → `Destination ASN`. Look for other suspicious peers.
157
158	6. Reverse-direction check. Switch the filter to the external IP as `Source IP` (now you're looking at incoming traffic from it). Internal hosts that received connections from the external IP show up — useful for inbound C2 / probe analysis.
159
160	7. Geographic check. Switch to Country Map. The location of the external IP gives a quick "where" — useful to compare against what your threat intelligence said.
161
162	What to record.
163
164	- All internal IPs that communicated with the external IP, time ranges, byte counts
165	- Other suspicious destinations the alerted host talked to in the same window
166	- Whether traffic is ongoing or stopped
167	- The dashboard URL of each view (for the incident report)
168
169	Caveats.
170
171	- Sampled flows can miss small connections. Beaconing at low rates may not be visible at 1-in-1000 sampling. If you sample, your security investigation has a floor.
172	- An external IP behind a CDN may be one of many destinations served by that infrastructure. ASN-level analysis (`Destination ASN`) is often more informative than IP.
173	- The malicious IP being public doesn't mean the internal host was compromised — false positives in threat intel are common. Cross-check with the host's logs.
174
175	## A note on the dashboard
176
177	All the playbooks above use the same controls: time range, filters, group-by fields, view switcher. Once you're comfortable with these four, every investigation becomes a permutation. Mastering the tool means knowing which permutation fits the question.
178
179	The URL preserves all your selections — copy it and paste into your incident-management ticket so anyone reviewing has the exact same view you saw.
180
181	## What's next
182
183	- [Sankey and Table](/docs/network-flows/visualization/summary-sankey.md) — Full reference for the default view.
184	- [Filters and Facets](/docs/network-flows/visualization/filters-facets.md) — How to narrow effectively.
185	- [Time-Series](/docs/network-flows/visualization/time-series.md) — Trends over the time range.
186	- [Anti-patterns](/docs/network-flows/anti-patterns.md) — What "wrong" looks like and why.
187	- [Validation and Data Quality](/docs/network-flows/validation.md) — Confirming your numbers before acting on them.